home-network-k3s-impl/decisions/2026-09-30-jackett-oauth2-proxy-session.md

---
date: 2026-09-30
branch: fix/jackett-oauth2-proxy-session
title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
---

## What changed

Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
never fixes it, and I am never re-prompted for login if it is a login issue."*

**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.

Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
three; do not rest the case on the first.

**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
pods' own logs, interleaved to the second:

    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
    15:27:59  radarr   "cookie signature not valid, removing session"
    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie

`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
so plain expiry does not explain it"): it was never expiry, and it was never Jackett.

It also explains the half of the report that looked like user error. **Reloading genuinely does not
help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.

**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
bounded, Crafty is evidence **for** the fix.

**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
(disabled), so the proxy holds a refresh token it has no mandate to spend.

**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
nothing logged as a failure anywhere.

### The fix has one home, and it is shared — this is the blocker

The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
apps to a sibling branch. **That split is not implementable, and I found out by trying it.**

`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
gate rejected it exactly as designed:

    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)

So I made the change at its real source instead and **measured the blast radius** before committing
to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
apps, and `render-golden` requires all five be regenerated in the same commit:

    CHANGED: apps/wave-0/jackett.yaml
    CHANGED: apps/wave-0/pihole.yaml
    CHANGED: apps/wave-0/qbittorrent.yaml
    CHANGED: apps/wave-1/radarr.yaml
    CHANGED: apps/wave-1/sonarr.yaml

Four of those five are the files I was explicitly told not to touch, and all four are being edited
right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
branch owns, and two branches rewriting the same generated block across five files is precisely the
cross-branch conflict this repo's §7 exists to prevent.

I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
as a single renderer commit on one branch. The intended content, validated below:

    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
    --cookie-domain={host}                    # host-scoped, derived
    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
    (drop --pass-access-token, --set-xauthrequest)

`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
either a per-app hook or four more measurements.

**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
change. It belongs in the same single renderer commit — which is an argument for cutting that commit
soon, not for me to hand-edit a golden a second time after the gate already refused the first.
(Crafty is separately hand-written and could take the metrics flag on its own.)

**Crafty is unaffected by all of this** — its sidecar is hand-written in
`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
2026-09-29.

### What this branch does land

`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
the app's own login apart (including the upstream `-` field that means the proxy answered a request
the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
than the file merely existing.

## What I verified

- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
  operator's whole report.
- **Six instances / seven pods are in the collision set** — the brief said five; the shared
  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
  can read each other's cookie — the precondition for the eviction.
- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
  `API route - Path: ^/api/`.
- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
  the proxy would have refused to start. Read off `--help`.
- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
  **least privilege and inertness**, not size.
- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
  `/oauth2/auth` is never called.
- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
  renderer-only change listed all five apps as changed (transcript above).
- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
  distinct user-visible failures.
- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
  `['http']` on all six per-app sidecars; only the shared proxy has
  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
  reported, and **only one of them ran**:

      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
      gate / conformance (pull_request)  state=success   desc='Has been skipped'

  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
  Anyone reading the PR's check mark alone would conclude otherwise.
- **`main` is protected**, re-measured this session rather than assumed:
  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.

## What I did NOT verify

- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
  change ships. **This is the single most important line in this entry.**
- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
  is proven to start, but nobody has added an indexer through the proposed configuration. The human
  test is below, for whoever lands the renderer change.
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
  Sonarr's direct `:9117` access.
- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
  `--session-cookie-minimal` is ruled out (above).
- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
  and the realm is where to look next.
- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
  **unverified by me this session** — I did not re-measure them against the Gitea API.
- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
  per-app ratio over a window longer than the current pod's life is not something I measured at all.
- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
  sufficient** — the coordinator's framing, and I agree with it on this evidence.
- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
  per-namespace `for` loops are edited from the scripted forms I actually used.

### The human test — for whoever lands the renderer fix

After it merges and the pod rolls (watch for a **new** pod name at 2/2):

1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
   today** — it is the 15:31:29 line. Return to the Jackett tab.
3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
   test; the session currently dies at five.
5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
   did not take; check the running pod's args before assuming anything else.
6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
   A 401 page on reload means the regex is too broad — that is a regression.