home-network-k3s-impl/decisions/2026-09-30-jackett-oauth2-proxy-session.md

date, branch, title

date	branch	title
2026-09-30	fix/jackett-oauth2-proxy-session	Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app

What changed

Reported as "GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page never fixes it, and I am never re-prompted for login if it is a login issue."

This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a deliberate stop, not an omission — see The fix has one home and it is shared, below.

Three mechanisms, all measured.

1. Cookie collision — and on Jackett it is the live failure, not a latent one. Six oauth2-proxy instances (seven pods) emit oauth2-proxy's default cookie name _oauth2_proxy scoped to .home.keithjackson.dev while holding seven different cookie-secrets. The ping-pong is in two pods' own logs, interleaved to the second:

15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
15:27:59  radarr   "cookie signature not valid, removing session"
15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie

removing session is oauth2-proxy calling ClearSession, which expires the cookie on its own --cookie-domain — the whole zone. So jackett's session is not merely overwritten, it is deleted by a process in another namespace, 26 s after a clean login whose own expires was five minutes out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window, so plain expiry does not explain it"): it was never expiry, and it was never Jackett.

It also explains the half of the report that looked like user error. Reloading genuinely does not help, because the operator's Radarr tab re-clears the cookie seconds after each new login.

2. Session lifetime pinned to a 5-minute access token. Measured from the sidecar's own [AuthSuccess]: created:15:31:06.391860254 / expires: + 300.000 s exactly, with refresh_token:true on the same line. --cookie-refresh is unset and its default is 0 (disabled), so the proxy holds a refresh token it has no mandate to spend.

3. XHRs answered with 302 instead of 401 — why none of it was visible. oauth2-proxy's built-in AJAX guard fires only on an exact Accept: application/json, which Jackett's jQuery does not consistently send. Measured on /api/ paths in this pod's log: 33 × 302 against 2 × 401, and the split runs straight through Jackett's own POSTs (/api/v2.0/server/adminpassword → 401, /api/v2.0/server/update → 302). A 302 on an unauthenticated XHR is followed invisibly by the browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302 re-issues a POST as a GET and drops the body, so the indexer configuration is discarded with nothing logged as a failure anywhere.

The fix has one home, and it is shared — this is the blocker

The brief assigned me apps/wave-0/jackett.yaml as exclusively mine, and assigned the other four apps to a sibling branch. That split is not implementable, and I found out by trying it.

apps/wave-0/jackett.yaml is a generated golden. The sidecar's argument list is built by one shared function, _oauth2_sidecar_container() in catalog/render.py. catalog/jackett.yaml declares only auth.mode and auth.groups, and the auth schema is additionalProperties: false — there is no per-app override hook. I wrote the fix into the rendered file first; the merge gate rejected it exactly as designed:

[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
[fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)

So I made the change at its real source instead and measured the blast radius before committing to it. A change confined to that one function rewrites the sidecar block in all five rendered apps, and render-golden requires all five be regenerated in the same commit:

CHANGED: apps/wave-0/jackett.yaml
CHANGED: apps/wave-0/pihole.yaml
CHANGED: apps/wave-0/qbittorrent.yaml
CHANGED: apps/wave-1/radarr.yaml
CHANGED: apps/wave-1/sonarr.yaml

Four of those five are the files I was explicitly told not to touch, and all four are being edited right now on fix/oauth2-proxy-cookie-collision-fleet. The gate and the ownership boundary are in direct contradiction: I cannot fix Jackett's sidecar without regenerating four files another branch owns, and two branches rewriting the same generated block across five files is precisely the cross-branch conflict this repo's §7 exists to prevent.

I stopped rather than force it. The cookie fix is one change, not five, and it should be re-cut as a single renderer commit on one branch. The intended content, validated below:

--cookie-name=_oauth2_proxy_{name}        # per-app, derived
--cookie-domain={host}                    # host-scoped, derived
--whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
--cookie-refresh=2m30s                    # half the measured 300 s token lifetime
(drop --pass-access-token, --set-xauthrequest)

--api-route=^/api/ is the one flag I would not apply renderer-wide on this evidence: I measured the 302/401 split on Jackett only, and PiHole's UI does not serve /api/ at all. It needs either a per-app hook or four more measurements.

Crafty is unaffected by all of this — its sidecar is hand-written in platform/crafty/deployment.yaml, not rendered, which is why it could be and was fixed alone on 2026-09-29.

What this branch does land

runbooks/oauth2-proxy-session.md — the four sidecar log lines that tell Keycloak, the sidecar and the app's own login apart (including the upstream - field that means the proxy answered a request the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the 302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the runbooks configMapGenerator and listed in runbooks/README.md, so the URL resolves rather than the file merely existing.

What I verified

• The collision is the live failure. [stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session — 1 occurrence in jackett's log (15:23:44), 8 in radarr's (15:22:49, 15:23:58, 15:27:59, 15:31:29, 15:32:29 ×4), and zero in pihole, qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the operator's whole report.
• Six instances / seven pods are in the collision set — the brief said five; the shared forward-auth proxy in namespace oauth2-proxy is the sixth and runs 2 replicas. Read from live pod specs: jackett, pihole, qbittorrent, radarr, sonarr and oauth2-proxy all have no --cookie-name argument and --cookie-domain=.home.keithjackson.dev. Only crafty is fixed.
• All seven cookie-secrets differ. Compared in-cluster by sha256, first 12 hex only, values never printed: d9c5…/0b3b…/b025…/bb6a…/25f4…/e679…/911a…. So no two of these proxies can read each other's cookie — the precondition for the eviction.
• The 5-minute lifetime, from four [AuthSuccess] lines: expires − created = 300.000 s, with refresh_token:true and groups:[platform-admins]. Live args confirm --cookie-refresh absent.
• The 302/401 split on /api/ paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
• 4 of the 6 POSTs never reached Jackett — the access log's upstream field is -, not /: adminpassword → 401, adminpassword → 401, server/update → 302, server/update → 302. The two that did reach it (adminpassword, indexers/1337x/config) carry the authenticated email.
• The proposed argument list starts, against the pinned image and the real issuer. Ran quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 locally with the full proposed args; OIDC discovery against keycloak.home.keithjackson.dev succeeded and it printed back its own reading: Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s and API route - Path: ^/api/.
• --api-route is singular. The brief specified --api-routes, which does not exist in v7.8.1; the proxy would have refused to start. Read off --help.
• --session-cookie-minimal is mutually exclusive with --cookie-refresh — provoked, not assumed: invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions. session_cookie_minimal cannot be set. This kills the brief's premise that dropping --pass-access-token shrinks the cookie: only --session-cookie-minimal strips tokens from the session, and it is unavailable while refresh is on. Dropping those two flags is justified on least privilege and inertness, not size.
• --set-xauthrequest is inert here: HTTPRoute jackett-9117 backendRef is Service/jackett port 4180, there is no ext-authz attachment on the shared Gateway, so /oauth2/auth is never called.
• apps/** is generated and the fix cannot be scoped to one app — both halves measured, not inferred: the gate's render-golden stage rejected the hand-edit, and re-rendering after a renderer-only change listed all five apps as changed (transcript above).
• Gate: exit code and stage list in the next section.

What I did NOT verify

• The operator's problem is NOT FIXED by this branch. Only the diagnosis and the runbook land. Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer change ships. This is the single most important line in this entry.
• No browser round-trip. Every mechanism is measured from logs and specs and the argument list is proven to start, but nobody has added an indexer through the proposed configuration. The human test is below, for whoever lands the renderer change.
• Nothing was applied. No kubectl apply/edit/patch, no flux reconcile. The running pod is still jackett-c5bfcbfb8-c75f9 (2/2 Running, 17 d, 0 restarts — unchanged by this session's read-only probes). Note that when the fix does land, merging is what deploys, and it will restart the pod (strategy: Recreate on an RWO Longhorn volume), briefly interrupting Radarr's and Sonarr's direct :9117 access.
• Cookie size and chunking: UNMEASURED, and I will not claim either way. I could not read a Set-Cookie header without a browser. I found no _oauth2_proxy_0/_1 chunk evidence and no 4 kB warning in 36,287 log lines — but I did not establish that v7.8.1 emits such a warning at this log level, so that absence is not evidence. If sessions still misbehave after the fix lands, this is the next thing to measure, with --session-store-type=redis as the escape hatch, since --session-cookie-minimal is ruled out (above).
• --cookie-refresh=2m30s has never executed a refresh here. It depends on Keycloak accepting the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin credentials). If Keycloak's refresh token is also short-lived, the five-minute logout persists and the realm is where to look next.
• --api-route is measured for Jackett only. PiHole and qBittorrent were not measured, and PiHole's UI does not serve /api/. Do not apply it renderer-wide on my evidence.
• The four sibling apps still collide with each other regardless of what happens to Jackett. The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
• L2/L3 conformance did not run (CLUSTER_AVAILABLE unset), so nothing about runtime behaviour is certified by the gate.
• The runbook's commands were not all executed verbatim. The log-reading ones I ran; the per-namespace for loops are edited from the scripted forms I actually used.

The human test — for whoever lands the renderer fix

After it merges and the pod rolls (watch for a new pod name at 2/2):

1. Sign in at https://jackett.home.keithjackson.dev/UI/Dashboard.
2. Open Radarr in a second tab and sign in there too. This is the step that breaks Jackett today — it is the 15:31:29 line. Return to the Jackett tab.
3. Add an indexer — + Add indexer, pick one, Configure, Okay. It must save.
4. Wait six minutes on the open dashboard, then save another. That is the --cookie-refresh test; the session currently dies at five.
5. If something does still fail, it must now fail LOUDLY — DevTools → Network, a broken /api/ call shows 401, not a 302 followed by HTML. A silent success-looking no-op means the change did not take; check the running pod's args before assuming anything else.
6. A plain page reload must still sign you in silently (/UI/Dashboard is outside ^/api/). A 401 page on reload means the regex is too broad — that is a regression.