home-network-k3s-impl/decisions/2026-09-30-jackett-oauth2-proxy-session.md

date, branch, title

date	branch	title
2026-09-30	fix/jackett-oauth2-proxy-session	Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app

What changed

Reported as "GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page never fixes it, and I am never re-prompted for login if it is a login issue."

This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a deliberate stop, not an omission — see The fix has one home and it is shared, below.

Three mechanisms, all measured. Weighting matters here, so it is stated up front: mechanisms 2 and 3 are demonstrated from Jackett's own access log and between them fully explain the report as written — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1 is a real, firing defect that Jackett genuinely violates, and it explains the intermittency on top; I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all three; do not rest the case on the first.

1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator. Six oauth2-proxy instances (seven pods) emit oauth2-proxy's default cookie name _oauth2_proxy scoped to .home.keithjackson.dev while holding seven different cookie-secrets. The ping-pong is in two pods' own logs, interleaved to the second:

15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
15:27:59  radarr   "cookie signature not valid, removing session"
15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie

removing session is oauth2-proxy calling ClearSession, which expires the cookie on its own --cookie-domain — the whole zone. So jackett's session is not merely overwritten, it is deleted by a process in another namespace, 26 s after a clean login whose own expires was five minutes out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window, so plain expiry does not explain it"): it was never expiry, and it was never Jackett.

It also explains the half of the report that looked like user error. Reloading genuinely does not help, because the operator's Radarr tab re-clears the cookie seconds after each new login — and Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against 1 AuthSuccess, the top paths are /api/v3/command (62), /api/v3/queue/details (46) and /signalr/messages/negotiate (43). Every one of those polls, arriving with a cookie Radarr cannot decrypt, calls ClearSession again. A left-open Radarr tab is a cookie-deletion engine firing every few seconds, which is why Jackett's sessions die at irregular, short, apparently random intervals.

A counter-hypothesis was put to me and it does not survive bounding. The observation was that Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from Loki), implying the fix does not work. But Crafty's pod started 2026-09-30T03:57:19Z with 0 restarts, so a 24 h window is ~10 h of the pre-fix pod. Rather than bound it in Loki I read the ratio off the pod's own log, whose boundary is the process start and therefore exact: inside the fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 No valid authentication events, not 81 — from one client IP, two of them WebSocket /ws reconnects, in two isolated minutes (04:02 and 15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty) manufacture failure lines at their retry rate — so it ranks chattiness, not breakage. Correctly bounded, Crafty is evidence for the fix.

2. Session lifetime pinned to a 5-minute access token. Measured from the sidecar's own [AuthSuccess]: created:15:31:06.391860254 / expires: + 300.000 s exactly, with refresh_token:true on the same line. --cookie-refresh is unset and its default is 0 (disabled), so the proxy holds a refresh token it has no mandate to spend.

3. XHRs answered with 302 instead of 401 — why none of it was visible. oauth2-proxy's built-in AJAX guard fires only on an exact Accept: application/json, which Jackett's jQuery does not consistently send. Measured on /api/ paths in this pod's log: 33 × 302 against 2 × 401, and the split runs straight through Jackett's own POSTs (/api/v2.0/server/adminpassword → 401, /api/v2.0/server/update → 302). A 302 on an unauthenticated XHR is followed invisibly by the browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302 re-issues a POST as a GET and drops the body, so the indexer configuration is discarded with nothing logged as a failure anywhere.

The fix has one home, and it is shared — this is the blocker

The brief assigned me apps/wave-0/jackett.yaml as exclusively mine, and assigned the other four apps to a sibling branch. That split is not implementable, and I found out by trying it.

apps/wave-0/jackett.yaml is a generated golden. The sidecar's argument list is built by one shared function, _oauth2_sidecar_container() in catalog/render.py. catalog/jackett.yaml declares only auth.mode and auth.groups, and the auth schema is additionalProperties: false — there is no per-app override hook. I wrote the fix into the rendered file first; the merge gate rejected it exactly as designed:

[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
[fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)

So I made the change at its real source instead and measured the blast radius before committing to it. A change confined to that one function rewrites the sidecar block in all five rendered apps, and render-golden requires all five be regenerated in the same commit:

CHANGED: apps/wave-0/jackett.yaml
CHANGED: apps/wave-0/pihole.yaml
CHANGED: apps/wave-0/qbittorrent.yaml
CHANGED: apps/wave-1/radarr.yaml
CHANGED: apps/wave-1/sonarr.yaml

Four of those five are the files I was explicitly told not to touch, and all four are being edited right now on fix/oauth2-proxy-cookie-collision-fleet. The gate and the ownership boundary are in direct contradiction: I cannot fix Jackett's sidecar without regenerating four files another branch owns, and two branches rewriting the same generated block across five files is precisely the cross-branch conflict this repo's §7 exists to prevent.

I stopped rather than force it. The cookie fix is one change, not five, and it should be re-cut as a single renderer commit on one branch. The intended content, validated below:

--cookie-name=_oauth2_proxy_{name}        # per-app, derived
--cookie-domain={host}                    # host-scoped, derived
--whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
--cookie-refresh=2m30s                    # half the measured 300 s token lifetime
(drop --pass-access-token, --set-xauthrequest)

--api-route=^/api/ is the one flag I would not apply renderer-wide on this evidence: I measured the 302/401 split on Jackett only, and PiHole's UI does not serve /api/ at all. It needs either a per-app hook or four more measurements.

Metrics — asked for, and declined here for the same structural reason, not on merit. It is true and I re-measured it: no per-app sidecar exposes metrics. Only the shared proxy carries --metrics-address=0.0.0.0:44180 and a metrics container port; jackett, pihole, qbittorrent, radarr, sonarr and crafty all have (ABSENT) and ports ['http'] alone, so the listener never starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is already proven in-tree by the shared proxy. But it is not "inside my file": the container's args and its ports both come from _oauth2_sidecar_container(), so adding --metrics-address plus a metrics port hits the identical five-file blast radius as the cookie change. It belongs in the same single renderer commit — which is an argument for cutting that commit soon, not for me to hand-edit a golden a second time after the gate already refused the first. (Crafty is separately hand-written and could take the metrics flag on its own.)

Crafty is unaffected by all of this — its sidecar is hand-written in platform/crafty/deployment.yaml, not rendered, which is why it could be and was fixed alone on 2026-09-29.

What this branch does land

runbooks/oauth2-proxy-session.md — the four sidecar log lines that tell Keycloak, the sidecar and the app's own login apart (including the upstream - field that means the proxy answered a request the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the 302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the runbooks configMapGenerator and listed in runbooks/README.md, so the URL resolves rather than the file merely existing.

What I verified

• The collision is the live failure. [stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session — 1 occurrence in jackett's log (15:23:44), 8 in radarr's (15:22:49, 15:23:58, 15:27:59, 15:31:29, 15:32:29 ×4), and zero in pihole, qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the operator's whole report.

• Six instances / seven pods are in the collision set — the brief said five; the shared forward-auth proxy in namespace oauth2-proxy is the sixth and runs 2 replicas. Read from live pod specs: jackett, pihole, qbittorrent, radarr, sonarr and oauth2-proxy all have no --cookie-name argument and --cookie-domain=.home.keithjackson.dev. Only crafty is fixed.

• All seven cookie-secrets differ. Compared in-cluster by sha256, first 12 hex only, values never printed: d9c5…/0b3b…/b025…/bb6a…/25f4…/e679…/911a…. So no two of these proxies can read each other's cookie — the precondition for the eviction.

• The 5-minute lifetime, from four [AuthSuccess] lines: expires − created = 300.000 s, with refresh_token:true and groups:[platform-admins]. Live args confirm --cookie-refresh absent.

• The 302/401 split on /api/ paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.

• 4 of the 6 POSTs never reached Jackett — the access log's upstream field is -, not /: adminpassword → 401, adminpassword → 401, server/update → 302, server/update → 302. The two that did reach it (adminpassword, indexers/1337x/config) carry the authenticated email.

• The proposed argument list starts, against the pinned image and the real issuer. Ran quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 locally with the full proposed args; OIDC discovery against keycloak.home.keithjackson.dev succeeded and it printed back its own reading: Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s and API route - Path: ^/api/.

• --api-route is singular. The brief specified --api-routes, which does not exist in v7.8.1; the proxy would have refused to start. Read off --help.

• --session-cookie-minimal is mutually exclusive with --cookie-refresh — provoked, not assumed: invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions. session_cookie_minimal cannot be set. This kills the brief's premise that dropping --pass-access-token shrinks the cookie: only --session-cookie-minimal strips tokens from the session, and it is unavailable while refresh is on. Dropping those two flags is justified on least privilege and inertness, not size.

• --set-xauthrequest is inert here: HTTPRoute jackett-9117 backendRef is Service/jackett port 4180, there is no ext-authz attachment on the shared Gateway, so /oauth2/auth is never called.

• apps/** is generated and the fix cannot be scoped to one app — both halves measured, not inferred: the gate's render-golden stage rejected the hand-edit, and re-rendering after a renderer-only change listed all five apps as changed (transcript above).

• Crafty's post-fix failure rate, bounded exactly. Pod crafty-647c7d8cc5-5spv9, startTime=2026-09-30T03:57:19Z, restarts=0, live args --cookie-name=_oauth2_proxy_crafty and --cookie-domain=crafty.home.keithjackson.dev. Over that pod's whole lifetime: 3 No valid authentication, 0 AuthSuccess, one client IP, 2 of the 3 on /ws. The 81:1 figure is a 24 h window over a 13.9 h pod and is dominated by the pre-fix process.

• The ratio metric ranks chattiness, not breakage. Radarr 156:1 with the top paths being /api/v3/command, /api/v3/queue/details and /signalr/messages/negotiate — UI polling, not 156 distinct user-visible failures.

• No per-app sidecar exposes metrics. --metrics-address is (ABSENT) and ports is ['http'] on all six per-app sidecars; only the shared proxy has --metrics-address=0.0.0.0:44180 and a metrics port.

• Gate: ./ci/gate.sh in the pinned toolbox, rc 3 — UNDETERMINED, not green. Green: spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1. Skipped: gitea-settings (no SETTINGS_API_TOKEN) and L2/L3 conformance (CLUSTER_AVAILABLE unset), leaving touched requirement R-OBS-04 uncertified. Note spec-immutability is green, not skipped — mounting the sibling spec repo at /home-network-k3s (the path ci/spec-immutability.sh actually defaults to) in the ENGINE shim fixes that skip; the commonly circulated shim mounts it at its host path, where SPEC_REPO never finds it.

• Gitea CI on this PR, read from the commit's /statuses rather than from a tick. Two contexts reported, and only one of them ran:

  gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
  gate / conformance (pull_request)  state=success   desc='Has been skipped'
  

  That second line is §2b's trap in the wild: a skipped Gitea job reports state=success. The conformance job did not execute, so the green tick beside it certifies nothing. Read together with the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran. Anyone reading the PR's check mark alone would conclude otherwise.

• main is protected, re-measured this session rather than assumed: GET /repos/.../branches/main → protected=true, user_can_push=false.

What I did NOT verify

• The operator's problem is NOT FIXED by this branch. Only the diagnosis and the runbook land. Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer change ships. This is the single most important line in this entry.
• No browser round-trip. Every mechanism is measured from logs and specs and the argument list is proven to start, but nobody has added an indexer through the proposed configuration. The human test is below, for whoever lands the renderer change.
• Nothing was applied. No kubectl apply/edit/patch, no flux reconcile. The running pod is still jackett-c5bfcbfb8-c75f9 (2/2 Running, 17 d, 0 restarts — unchanged by this session's read-only probes). Note that when the fix does land, merging is what deploys, and it will restart the pod (strategy: Recreate on an RWO Longhorn volume), briefly interrupting Radarr's and Sonarr's direct :9117 access.
• Cookie size and chunking: UNMEASURED, and I will not claim either way. I could not read a Set-Cookie header without a browser. I found no _oauth2_proxy_0/_1 chunk evidence and no 4 kB warning in 36,287 log lines — but I did not establish that v7.8.1 emits such a warning at this log level, so that absence is not evidence. If sessions still misbehave after the fix lands, this is the next thing to measure, with --session-store-type=redis as the escape hatch, since --session-cookie-minimal is ruled out (above).
• --cookie-refresh=2m30s has never executed a refresh here. It depends on Keycloak accepting the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin credentials). If Keycloak's refresh token is also short-lived, the five-minute logout persists and the realm is where to look next.
• --api-route is measured for Jackett only. PiHole and qBittorrent were not measured, and PiHole's UI does not serve /api/. Do not apply it renderer-wide on my evidence.
• The four sibling apps still collide with each other regardless of what happens to Jackett. The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
• L2/L3 conformance did not run (CLUSTER_AVAILABLE unset), so nothing about runtime behaviour is certified by the gate. gitea-settings also skipped, so branch protection and runner state are unverified by me this session — I did not re-measure them against the Gitea API.
• I did not query Loki. The Crafty bounding above comes from the pod log, which I argue is the better instrument (an exact process boundary rather than a wall-clock window against a 72 h retention). But it means I cannot speak to the 24 h or 12 h platform-wide ratios, and a per-app ratio over a window longer than the current pod's life is not something I measured at all.
• Collision causation is one adjacency, not a population. The 15:31:29 → 15:31:32 pair is three seconds and the mechanism is documented and present, but I did not enumerate every jackett failure and match it to a radarr ClearSession. Treat mechanism 1 as necessary and firing, not proven sufficient — the coordinator's framing, and I agree with it on this evidence.
• The metrics change is NOT in this branch (reason above). Jackett therefore remains inside the oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
• The runbook's commands were not all executed verbatim. The log-reading ones I ran; the per-namespace for loops are edited from the scripted forms I actually used.

The human test — for whoever lands the renderer fix

After it merges and the pod rolls (watch for a new pod name at 2/2):

1. Sign in at https://jackett.home.keithjackson.dev/UI/Dashboard.
2. Open Radarr in a second tab and sign in there too. This is the step that breaks Jackett today — it is the 15:31:29 line. Return to the Jackett tab.
3. Add an indexer — + Add indexer, pick one, Configure, Okay. It must save.
4. Wait six minutes on the open dashboard, then save another. That is the --cookie-refresh test; the session currently dies at five.
5. If something does still fail, it must now fail LOUDLY — DevTools → Network, a broken /api/ call shows 401, not a 302 followed by HTML. A silent success-looking no-op means the change did not take; check the running pod's args before assuming anything else.
6. A plain page reload must still sign you in silently (/UI/Dashboard is outside ^/api/). A 401 page on reload means the regex is too broad — that is a regression.