fix(jackett): root-cause the oauth2-proxy session loss; the fix cannot be scoped to one app #30
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#30
Reference in new issue
Block a user
The report
This PR lands the diagnosis and a runbook. It deliberately does NOT land the manifest fix — the fix cannot be scoped to one app, and I found that out by trying it. See Why no manifest fix below.
What actually happens, measured
Three mechanisms. Weighting stated up front: 2 and 3 are demonstrated from Jackett's own access log and between them fully explain the report as written. 1 is a real, firing defect that Jackett genuinely violates and explains the intermittency on top — necessary and firing, not proven sufficient.
1. Cookie collision. Six oauth2-proxy instances (seven pods) emit the default cookie name
_oauth2_proxyon.home.keithjackson.devwhile holding seven different cookie-secrets (compared in-cluster by sha256; all distinct).removing sessionis oauth2-proxy calling ClearSession, which expires the cookie on its own zone-wide--cookie-domain— so Jackett's session is deleted by a process in another namespace:8 such lines in radarr's log, 1 in jackett's, 0 in pihole/qbittorrent/sonarr/crafty/both shared replicas. And Radarr is not a passive tab — 156 unauthenticated requests against 1 AuthSuccess, topped by
/api/v3/command(62),/api/v3/queue/details(46),/signalr/messages/negotiate(43). Every poll arriving with an undecryptable cookie calls ClearSession again. A left-open Radarr tab is a cookie-deletion engine firing every few seconds — which is why reloading never helps.2. Session pinned to a 5-minute access token. From the sidecar's own
[AuthSuccess]:expires − created = 300.000 sexactly, withrefresh_token:trueon the same line.--cookie-refreshis unset; its default is0(disabled). The proxy holds a refresh token it has no mandate to spend.3. XHRs answered 302, not 401 — why none of it was visible. oauth2-proxy's AJAX guard fires only on an exact
Accept: application/json, which Jackett's jQuery does not consistently send. On/api/paths: 33 × 302 against 2 × 401, and the split runs through Jackett's own POSTs (adminpassword→ 401,server/update→ 302). A browser follows a 302 invisibly, SSO satisfies it with no prompt, and the XHR resolves with HTML where JSON was expected. On a write it is worse: a POST following a 302 is re-issued as a GET with the body dropped. 4 of the 6 POSTs in this pod's log were answered by the proxy and never reached Jackett.Counter-evidence tested, and it inverted
It was put to me that Crafty already carries the cookie fix yet showed the platform's worst ratio (81:1 over 24 h), implying the fix does not work. Crafty's pod started
2026-09-30T03:57:19Zwithrestarts=0, so a 24 h window is ~10 h of the pre-fix process. Bounded to the pod's own log — an exact process boundary rather than a wall-clock guess — Crafty logged 3No valid authenticationin its fixed pod's entire 13 h 52 m life, from one client IP, two of them/wsWebSocket reconnects, in two isolated minutes. Post-fix Crafty is idle, not broken.The ratio metric ranks chattiness, not breakage. Added to the runbook, because ranking apps this way is how a deployed fix gets written off.
Why no manifest fix
apps/wave-0/jackett.yamlis a generated golden. The args come from one shared function,_oauth2_sidecar_container()incatalog/render.py;catalog/jackett.yamldeclares onlyauth.mode/auth.groupsand theauthschema isadditionalProperties: false— no per-app override hook. I wrote the fix into the rendered file first and the gate rejected it exactly as designed (GOLDEN DRIFT … render-golden: FAIL, rc 1). Changing the renderer instead rewrites all five rendered apps, andrender-goldenrequires all five be regenerated in the same commit — measured:Four of those are owned by
fix/oauth2-proxy-cookie-collision-fleetright now. The cookie fix is one change, not five. It needs re-cutting as a single renderer commit rather than two branches rewriting the same generated block.Validated the intended argument list against the pinned image and the real issuer before stopping — it printed back
name:_oauth2_proxy_jackett … domains:jackett.home.keithjackson.dev … refresh:after 2m30sandAPI route - Path: ^/api/. Two corrections fell out:--api-routeis singular.--api-routesdoes not exist in v7.8.1; the proxy refuses to start.--session-cookie-minimalis mutually exclusive with--cookie-refresh(provoked, not assumed). So dropping--pass-access-tokendoes not shrink the cookie — that is a least-privilege change, not a size one.Metrics: confirmed and declined here for the same structural reason. No per-app sidecar exposes metrics (
--metrics-addressABSENT, ports['http']on all six); only the shared proxy has the flag and ametricsport. Worth adding — butargsandportsboth come from the same shared function, so it hits the identical five-file blast radius and belongs in that same renderer commit.Gate
./ci/gate.shin the pinned toolbox: rc 3 — UNDETERMINED, not green.Green:
spec-immutability,approval-policy,decisions-log,lint,l0,render-golden,l1.Skipped:
gitea-settings(noSETTINGS_API_TOKEN), L2/L3 conformance (CLUSTER_AVAILABLEunset) — touched requirementR-OBS-04uncertified.spec-immutabilityis green, not skipped: mount the sibling spec repo at/home-network-k3s(the pathci/spec-immutability.shdefaults to) rather than at its host path.Not verified
kubectl apply/edit/patch, noflux reconcile). Running pod is stilljackett-c5bfcbfb8-c75f9, 2/2, 17 d, 0 restarts.Set-Cookiewithout a browser, and I did not establish that v7.8.1 logs a chunking warning at all, so the absence of one is not evidence.Full detail and the six-step human test:
decisions/2026-09-30-jackett-oauth2-proxy-session.md.🤖 Generated with Claude Code
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps, and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT, deliberately -- see the blocker below. ROOT CAUSE, measured from two pods' logs interleaved to the second: 15:31:06 jackett [AuthSuccess] 15:31:29 radarr "cookie signature not valid, removing session" 15:31:32 jackett "No valid authentication in request" <- 3s later Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name `_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT cookie-secrets (compared in-cluster by sha256; all distinct). `removing session` is ClearSession, which expires the cookie on radarr's own zone-wide --cookie-domain, so jackett's session is DELETED by a process in another namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it was never Jackett -- and it is why reloading does not help: the operator's Radarr tab re-clears the cookie seconds after each new login. Two contributing mechanisms, also measured: * session lifetime == Keycloak access-token lifetime, 300.000s exactly, with refresh_token:true on the same line and --cookie-refresh unset (default 0). * XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact `Accept: application/json`, which Jackett's jQuery does not send. On /api/ paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all. WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The sidecar args come from one shared function, _oauth2_sidecar_container() in catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and the auth schema is additionalProperties:false, so there is NO per-app override hook. Hand-editing the rendered file is rejected by the gate exactly as designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer instead rewrites all five rendered apps -- measured: CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml CHANGED: apps/wave-1/{radarr,sonarr}.yaml and render-golden requires all five be regenerated in the same commit. Four of them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie fix is ONE change, not five; it needs re-cutting as a single renderer commit rather than two branches rewriting the same generated block. Validated the intended argument list against the pinned image and the real issuer before stopping: it printed back `name:_oauth2_proxy_jackett ... domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and `API route - Path: ^/api/`. Two corrections fell out of that: --api-route is SINGULAR (--api-routes does not exist and the proxy refuses to start), and --session-cookie-minimal is mutually exclusive with --cookie-refresh ("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping --pass-access-token does NOT shrink the cookie -- it is a least-privilege change, not a size one. Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the auth layers apart (including the upstream `-` that means the proxy answered a request the app never saw), the cross-log collision diagnosis, and the generated-golden warning with its blast radius. Registered in the configMapGenerator AND runbooks/README.md so the URL resolves. Jackett is NOT fixed by this commit and will keep losing its session to Radarr until the renderer change ships. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>Coordinator raised crafty as counter-evidence: it carries the cookie fix and showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix does not work. Bounded it exactly instead of in Loki -- crafty's pod started 2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix process. Read off the pod's own log, whose boundary is the process start: crafty 3 'No valid authentication', 0 AuthSuccess, in 13h52m <- not 81 one client IP, 2 of the 3 on /ws (WebSocket reconnects), two isolated minutes (04:02, 15:22) Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is /api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) -- UI polling. Added that trap to the runbook, because ranking apps this way is how a deployed fix gets written off. That polling also sharpens the jackett mechanism: every poll arriving with a cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr tab deletes jackett's cookie every few seconds. Explains the intermittency. Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from jackett's own log and between them fully explain the report. Mechanism 1 (collision) is firing and jackett genuinely violates it, but causation is one 3-second adjacency, not a population -- necessary and firing, not proven sufficient. Stated that way in the entry. Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT, ports ['http'] on all six); only the shared proxy has the flag and a metrics port. Worth adding and DECLINED HERE for the structural reason only -- the container's args AND ports both come from _oauth2_sidecar_container(), so it hits the identical five-file blast radius. It belongs in the same single renderer commit. Recorded, not silently skipped. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>