From 6a6fea81515adeae62ea00b747152f136a3b389f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 12:47:42 -0500 Subject: [PATCH 1/3] docs(auth): root-cause jackett's session loss; add the oauth2-proxy-session runbook Reported as "GET works, PUT/POST to add indexers fail, reloading never helps, and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT, deliberately -- see the blocker below. ROOT CAUSE, measured from two pods' logs interleaved to the second: 15:31:06 jackett [AuthSuccess] 15:31:29 radarr "cookie signature not valid, removing session" 15:31:32 jackett "No valid authentication in request" <- 3s later Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name `_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT cookie-secrets (compared in-cluster by sha256; all distinct). `removing session` is ClearSession, which expires the cookie on radarr's own zone-wide --cookie-domain, so jackett's session is DELETED by a process in another namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it was never Jackett -- and it is why reloading does not help: the operator's Radarr tab re-clears the cookie seconds after each new login. Two contributing mechanisms, also measured: * session lifetime == Keycloak access-token lifetime, 300.000s exactly, with refresh_token:true on the same line and --cookie-refresh unset (default 0). * XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact `Accept: application/json`, which Jackett's jQuery does not send. On /api/ paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all. WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The sidecar args come from one shared function, _oauth2_sidecar_container() in catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and the auth schema is additionalProperties:false, so there is NO per-app override hook. Hand-editing the rendered file is rejected by the gate exactly as designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer instead rewrites all five rendered apps -- measured: CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml CHANGED: apps/wave-1/{radarr,sonarr}.yaml and render-golden requires all five be regenerated in the same commit. Four of them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie fix is ONE change, not five; it needs re-cutting as a single renderer commit rather than two branches rewriting the same generated block. Validated the intended argument list against the pinned image and the real issuer before stopping: it printed back `name:_oauth2_proxy_jackett ... domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and `API route - Path: ^/api/`. Two corrections fell out of that: --api-route is SINGULAR (--api-routes does not exist and the proxy refuses to start), and --session-cookie-minimal is mutually exclusive with --cookie-refresh ("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping --pass-access-token does NOT shrink the cookie -- it is a least-privilege change, not a size one. Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the auth layers apart (including the upstream `-` that means the proxy answered a request the app never saw), the cross-log collision diagnosis, and the generated-golden warning with its blast radius. Registered in the configMapGenerator AND runbooks/README.md so the URL resolves. Jackett is NOT fixed by this commit and will keep losing its session to Radarr until the renderer change ships. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-09-30-jackett-oauth2-proxy-session.md | 199 ++++++++++++++++++ runbooks/README.md | 8 + runbooks/kustomization.yaml | 5 + runbooks/oauth2-proxy-session.md | 168 +++++++++++++++ 4 files changed, 380 insertions(+) create mode 100644 decisions/2026-09-30-jackett-oauth2-proxy-session.md create mode 100644 runbooks/oauth2-proxy-session.md diff --git a/decisions/2026-09-30-jackett-oauth2-proxy-session.md b/decisions/2026-09-30-jackett-oauth2-proxy-session.md new file mode 100644 index 0000000..cb68515 --- /dev/null +++ b/decisions/2026-09-30-jackett-oauth2-proxy-session.md @@ -0,0 +1,199 @@ +--- +date: 2026-09-30 +branch: fix/jackett-oauth2-proxy-session +title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app +--- + +## What changed + +Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page +never fixes it, and I am never re-prompted for login if it is a login issue."* + +**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a +deliberate stop, not an omission** — see *The fix has one home and it is shared*, below. + +Three mechanisms, all measured. + +**1. Cookie collision — and on Jackett it is the live failure, not a latent one.** Six oauth2-proxy +instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to +`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two +pods' own logs, interleaved to the second: + + 15:23:17 radarr [AuthSuccess] -> radarr writes _oauth2_proxy + 15:23:44 jackett "cookie signature not valid, removing session" -> re-login #1 + 15:23:58 radarr "cookie signature not valid, removing session" -> radarr DELETES jackett's + 15:27:17 jackett [AuthSuccess] -> re-login #2 + 15:27:59 radarr "cookie signature not valid, removing session" + 15:31:06 jackett [AuthSuccess] -> re-login #3 + 15:31:29 radarr "cookie signature not valid, removing session" <- deletes it again + 15:31:32 jackett "No valid authentication in request" <- 3 SECONDS later, no cookie + +`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own** +`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted +by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes +out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window, +so plain expiry does not explain it"): it was never expiry, and it was never Jackett. + +It also explains the half of the report that looked like user error. **Reloading genuinely does not +help**, because the operator's Radarr tab re-clears the cookie seconds after each new login. + +**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own +`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with +`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0` +(disabled), so the proxy holds a refresh token it has no mandate to spend. + +**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in +AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not +consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the +split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401, +`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the +browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the +JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302 +re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with +nothing logged as a failure anywhere. + +### The fix has one home, and it is shared — this is the blocker + +The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four +apps to a sibling branch. **That split is not implementable, and I found out by trying it.** + +`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one +shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml` +declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false` +— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge +gate rejected it exactly as designed: + + [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render + [fail] stage render-golden: FAIL -> MERGE GATE: BLOCKED (rc 1) + +So I made the change at its real source instead and **measured the blast radius** before committing +to it. A change confined to that one function rewrites the sidecar block in **all five** rendered +apps, and `render-golden` requires all five be regenerated in the same commit: + + CHANGED: apps/wave-0/jackett.yaml + CHANGED: apps/wave-0/pihole.yaml + CHANGED: apps/wave-0/qbittorrent.yaml + CHANGED: apps/wave-1/radarr.yaml + CHANGED: apps/wave-1/sonarr.yaml + +Four of those five are the files I was explicitly told not to touch, and all four are being edited +right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in +direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another +branch owns, and two branches rewriting the same generated block across five files is precisely the +cross-branch conflict this repo's §7 exists to prevent. + +I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut +as a single renderer commit on one branch. The intended content, validated below: + + --cookie-name=_oauth2_proxy_{name} # per-app, derived + --cookie-domain={host} # host-scoped, derived + --whitelist-domain=.{ZONE} # unchanged: redirect targets, not cookie scope + --cookie-refresh=2m30s # half the measured 300 s token lifetime + (drop --pass-access-token, --set-xauthrequest) + +`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I +measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs +either a per-app hook or four more measurements. + +**Crafty is unaffected by all of this** — its sidecar is hand-written in +`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on +2026-09-29. + +### What this branch does land + +`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and +the app's own login apart (including the upstream `-` field that means the proxy answered a request +the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the +302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the +runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather +than the file merely existing. + +## What I verified + +- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session: + cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in + radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole, + qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the + operator's whole report. +- **Six instances / seven pods are in the collision set** — the brief said five; the shared + forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live + pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no** + `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed. +- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values + never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies + can read each other's cookie — the precondition for the eviction. +- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with + `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent. +- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204. +- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`: + `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The + two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email. +- **The proposed argument list starts, against the pinned image and the real issuer.** Ran + `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery + against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading: + `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s + domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and + `API route - Path: ^/api/`. +- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1; + the proxy would have refused to start. Read off `--help`. +- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not + assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions. + session_cookie_minimal cannot be set`. This kills the brief's premise that dropping + `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the + session, and it is unavailable while refresh is on. Dropping those two flags is justified on + **least privilege and inertness**, not size. +- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is + `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so + `/oauth2/auth` is never called. +- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not + inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a + renderer-only change listed all five apps as changed (transcript above). +- **Gate:** exit code and stage list in the next section. + +## What I did NOT verify + +- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land. + Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer + change ships. **This is the single most important line in this entry.** +- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list + is proven to start, but nobody has added an indexer through the proposed configuration. The human + test is below, for whoever lands the renderer change. +- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is + still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's + read-only probes). Note that when the fix does land, merging is what deploys, and it will restart + the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and + Sonarr's direct `:9117` access. +- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a + `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no + 4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at + this log level, so that absence is not evidence. If sessions still misbehave after the fix lands, + this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since + `--session-cookie-minimal` is ruled out (above). +- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting + the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin + credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists + and the realm is where to look next. +- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and + PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence. +- **The four sibling apps still collide with each other** regardless of what happens to Jackett. + The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr. +- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour + is certified by the gate. +- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the + per-namespace `for` loops are edited from the scripted forms I actually used. + +### The human test — for whoever lands the renderer fix + +After it merges and the pod rolls (watch for a **new** pod name at 2/2): + +1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`. +2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett + today** — it is the 15:31:29 line. Return to the Jackett tab. +3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save. +4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh` + test; the session currently dies at five. +5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/` + call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change + did not take; check the running pod's args before assuming anything else. +6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`). + A 401 page on reload means the regex is too broad — that is a regression. diff --git a/runbooks/README.md b/runbooks/README.md index 9c859fd..2368296 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. +- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never + re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log + (the four lines that matter, including the upstream `-` that means the proxy answered a request the + app never saw). Covers the cookie collision between per-app sidecars sharing the default + `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in + *other* namespaces and is only legible with two logs side by side — the 5-minute access-token + session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a + POST body silently. - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password. Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 08793d7..4e341bf 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -85,6 +85,11 @@ configMapGenerator: - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md + # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell + # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars + # presents as random session loss in ONE app's log and is only legible with both side by side, + # so the page has to be reachable at 3am without going looking for it in git. + - oauth2-proxy-session=oauth2-proxy-session.md # Operator procedure, not an alert page (no alert links to it — the # alert-runbooks-resolvable check treats that as advisory). It is served # because the plex.tv claim window is short and is not a thing to go looking diff --git a/runbooks/oauth2-proxy-session.md b/runbooks/oauth2-proxy-session.md new file mode 100644 index 0000000..85cd017 --- /dev/null +++ b/runbooks/oauth2-proxy-session.md @@ -0,0 +1,168 @@ +# oauth2-proxy-session — telling the auth layers apart from the sidecar log + +**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars +(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth +proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and +signed out at the same time: the page loads, but its buttons and saves quietly do nothing. + +> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an +> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.) + +## What this means here + +Every gated app on `home.keithjackson.dev` has **at least two** independent authentications +stacked on it, and the symptom rarely names the one that refused: + +1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login + form, or an error page from Keycloak's own hostname. +2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim + (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of + this page is about reading it. +3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login + *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer. + +The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \ + -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1' + +Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`: +`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has +already changed once. Do not run two `dev.sh` invocations at the same time. + +### The four lines, and what each one means + +| Log line | What actually happened | Which layer | +|---|---|---| +| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ | +| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar | +| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision | +| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar | + +That last column is the one people miss. The upstream field sits between the method and the quoted +path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means +it was proxied to the app. Count them: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \ + -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \ + | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140' + +If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's +log had `-`. + +## First commands + +**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem — +go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n logs \ + -l app.kubernetes.io/name= -c oauth2-proxy --tail=-1 | grep -c AuthSuccess' + +**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line. +If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy +is not refreshing it: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n logs \ + -l app.kubernetes.io/name= -c oauth2-proxy --tail=-1 \ + | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"' + +The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only +spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled. + +**3. Is this a cookie collision?** Two checks, and you want both. + +*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared +`.home.keithjackson.dev` domain is in the collision set with every other one: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \ + | grep -E "cookie-name|cookie-domain"' + +An instance with **no** `--cookie-name` argument is using the default. That is the failure — the +flag being absent, not wrong. + +*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal +line, and put the timestamps next to each other: + + infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \ + echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \ + -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done' + +**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between +instances that genuinely cannot read each other. Compare **hashes**, never values: + + infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \ + printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \ + -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done' + +Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here +before. + +## Likely causes, most likely first + +1. **Cookie collision between two per-app proxies.** Several instances emit the default + `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever + authenticates last overwrites the rest; the losers log `cookie signature not valid, removing + session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which + is the whole zone — **delete a session belonging to an app in another namespace**. The victim + then logs `No valid authentication` with no cookie at all, seconds after a clean login, while + its own `expires` is still minutes away. **Read both logs side by side; one app's log alone + makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped + `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide + scope — cross-host auth is its job. +2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means + the proxy never spends the refresh token it holds, so the session expires with the access token. + Set it to about half the measured token lifetime. +3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the + live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where + it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this + is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a + browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is + discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact + `Accept: application/json`, which most app JavaScript does not send; make it deterministic with + `--api-route=` (**singular** — `--api-routes` does not exist and the proxy refuses to + start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in. +4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a + session problem. Check the `groups:[...]` field on the `AuthSuccess` line. +5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed + by the app's own `/login` page is layer 3, not layer 2. + +## When to escalate or stop + +- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a + hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes + go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an + agent action. +- **Do not hand-edit `apps/**/.yaml` — those files are GENERATED.** The five per-app sidecars + (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function, + `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/.yaml` declares only + `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there + is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's + `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be + erased by the next render anyway. Change the renderer, then regenerate: + + python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation + + **Budget for the blast radius**: because the function is shared, any change to it rewrites the + sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in + the same commit. There is no way to change one app's sidecar arguments alone. Measured + 2026-09-30. Coordinate before starting — this is a single change, not five. +- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in + `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently. +- **Validate an argument change before you open the PR**, not after. The pinned image will tell + you what it understood, without touching the cluster: + + podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help + + Running it with your full proposed argument list prints a `Cookie settings:` line and an + `API route - Path:` line back at you, and rejects mutually exclusive flags outright + (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop + on a service you have just told someone is fixed. +- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of + that app and does nothing about the collision, which is caused by the *names* matching, not the + secrets differing. +- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if + there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no + cookie flag will help. +- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already + has a unique `--cookie-name`. That combination should not occur; it means something is writing + that name which is not in this inventory. Record what you measured. -- 2.54.0 From 0a0adce4d829ca147dc9be78376d2634f8d5fed2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 12:52:12 -0500 Subject: [PATCH 2/3] docs(auth): bound crafty's post-fix ratio to its own pod; record the metrics gap Coordinator raised crafty as counter-evidence: it carries the cookie fix and showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix does not work. Bounded it exactly instead of in Loki -- crafty's pod started 2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix process. Read off the pod's own log, whose boundary is the process start: crafty 3 'No valid authentication', 0 AuthSuccess, in 13h52m <- not 81 one client IP, 2 of the 3 on /ws (WebSocket reconnects), two isolated minutes (04:02, 15:22) Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is /api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) -- UI polling. Added that trap to the runbook, because ranking apps this way is how a deployed fix gets written off. That polling also sharpens the jackett mechanism: every poll arriving with a cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr tab deletes jackett's cookie every few seconds. Explains the intermittency. Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from jackett's own log and between them fully explain the report. Mechanism 1 (collision) is firing and jackett genuinely violates it, but causation is one 3-second adjacency, not a population -- necessary and firing, not proven sufficient. Stated that way in the entry. Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT, ports ['http'] on all six); only the shared proxy has the flag and a metrics port. Worth adding and DECLINED HERE for the structural reason only -- the container's args AND ports both come from _oauth2_sidecar_container(), so it hits the identical five-file blast radius. It belongs in the same single renderer commit. Recorded, not silently skipped. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-09-30-jackett-oauth2-proxy-session.md | 72 +++++++++++++++++-- runbooks/oauth2-proxy-session.md | 27 +++++++ 2 files changed, 94 insertions(+), 5 deletions(-) diff --git a/decisions/2026-09-30-jackett-oauth2-proxy-session.md b/decisions/2026-09-30-jackett-oauth2-proxy-session.md index cb68515..0bf3dc8 100644 --- a/decisions/2026-09-30-jackett-oauth2-proxy-session.md +++ b/decisions/2026-09-30-jackett-oauth2-proxy-session.md @@ -12,9 +12,14 @@ never fixes it, and I am never re-prompted for login if it is a login issue."* **This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a deliberate stop, not an omission** — see *The fix has one home and it is shared*, below. -Three mechanisms, all measured. +Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2 +and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as +written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1 +is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top; +I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all +three; do not rest the case on the first. -**1. Cookie collision — and on Jackett it is the live failure, not a latent one.** Six oauth2-proxy +**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to `.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two pods' own logs, interleaved to the second: @@ -35,7 +40,24 @@ out. That is the answer to the open question in the brief ("26 s is well inside so plain expiry does not explain it"): it was never expiry, and it was never Jackett. It also explains the half of the report that looked like user error. **Reloading genuinely does not -help**, because the operator's Radarr tab re-clears the cookie seconds after each new login. +help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and +Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against +**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and +`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot +decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every +few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals. + +**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that +Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from +Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0 +restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the +ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the +fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** — +from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and +15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The +ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty) +manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly +bounded, Crafty is evidence **for** the fix. **2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own `[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with @@ -95,6 +117,18 @@ as a single renderer commit on one branch. The intended content, validated below measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs either a per-app hook or four more measurements. +**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true +and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries +`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent, +radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never +starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is +already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's +`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding +`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie +change. It belongs in the same single renderer commit — which is an argument for cutting that commit +soon, not for me to hand-edit a golden a second time after the gate already refused the first. +(Crafty is separately hand-written and could take the metrics flag on its own.) + **Crafty is unaffected by all of this** — its sidecar is hand-written in `platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on 2026-09-29. @@ -148,7 +182,24 @@ than the file merely existing. - **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a renderer-only change listed all five apps as changed (transcript above). -- **Gate:** exit code and stage list in the next section. +- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`, + `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and + `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3** + `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a + 24 h window over a 13.9 h pod and is dominated by the pre-fix process. +- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being + `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156 + distinct user-visible failures. +- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is + `['http']` on all six per-app sidecars; only the shared proxy has + `--metrics-address=0.0.0.0:44180` and a `metrics` port. +- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green: + `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. + Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE` + unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**, + not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path + `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly + circulated shim mounts it at its host path, where `SPEC_REPO` never finds it. ## What I did NOT verify @@ -178,7 +229,18 @@ than the file merely existing. - **The four sibling apps still collide with each other** regardless of what happens to Jackett. The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr. - **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour - is certified by the gate. + is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are + **unverified by me this session** — I did not re-measure them against the Gitea API. +- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the + better instrument (an exact process boundary rather than a wall-clock window against a 72 h + retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a + per-app ratio over a window longer than the current pod's life is not something I measured at all. +- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three + seconds and the mechanism is documented and present, but I did not enumerate every jackett failure + and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven + sufficient** — the coordinator's framing, and I agree with it on this evidence. +- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the + oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it. - **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the per-namespace `for` loops are edited from the scripted forms I actually used. diff --git a/runbooks/oauth2-proxy-session.md b/runbooks/oauth2-proxy-session.md index 85cd017..da09678 100644 --- a/runbooks/oauth2-proxy-session.md +++ b/runbooks/oauth2-proxy-session.md @@ -97,6 +97,27 @@ instances that genuinely cannot read each other. Compare **hashes**, never value Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here before. +### Do not rank apps by their failure ratio + +Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is +not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket +manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes +scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured +2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top +paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate` +(43) — polling, not 156 user-visible failures. + +**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod +that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a +deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process +start — so read it there: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n get pod \ + -l app.kubernetes.io/name= -o jsonpath="{.items[*].status.startTime}{\"\n\"}"' + +then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h +to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects. + ## Likely causes, most likely first 1. **Cookie collision between two per-app proxies.** Several instances emit the default @@ -148,6 +169,12 @@ before. 2026-09-30. Coordinate before starting — this is a single change, not five. - **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently. +- **You cannot currently alert on any of this**, and that is worth knowing before you promise + monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a + `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy + series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both. + Until the renderer emits them for the per-app sidecars, every diagnosis on this page is + log-reading only. - **Validate an argument change before you open the PR**, not after. The pinned image will tell you what it understood, without touching the cluster: -- 2.54.0 From c89537c4654664a9e9fd0457bc51a9678f8cbec7 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 16:13:06 -0500 Subject: [PATCH 3/3] docs(auth): record what PR #30's CI actually ran, not what its tick said Read the commit's /statuses rather than the check mark: gate / gate (pull_request) state=success desc='Successful in 1m41s' gate / conformance (pull_request) state=success desc='Has been skipped' The second is CLAUDE.md 2b's trap in the wild -- a SKIPPED Gitea job reports state=success, so that green tick certifies nothing. It agrees exactly with the local rc 3: the offline L0/L1 subset passed, L2/L3 never ran. Anyone reading the PR's check mark alone would conclude the opposite. Also re-measured branch protection instead of trusting the table in CLAUDE.md: GET /repos/.../branches/main -> protected=true, user_can_push=false. Co-Authored-By: Claude Opus 5 (1M context) --- decisions/2026-09-30-jackett-oauth2-proxy-session.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/decisions/2026-09-30-jackett-oauth2-proxy-session.md b/decisions/2026-09-30-jackett-oauth2-proxy-session.md index 0bf3dc8..d8c0259 100644 --- a/decisions/2026-09-30-jackett-oauth2-proxy-session.md +++ b/decisions/2026-09-30-jackett-oauth2-proxy-session.md @@ -200,6 +200,18 @@ than the file merely existing. not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly circulated shim mounts it at its host path, where `SPEC_REPO` never finds it. +- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts + reported, and **only one of them ran**: + + gate / gate (pull_request) state=success desc='Successful in 1m41s' + gate / conformance (pull_request) state=success desc='Has been skipped' + + That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The + conformance job did not execute, so the green tick beside it certifies nothing. Read together with + the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran. + Anyone reading the PR's check mark alone would conclude otherwise. +- **`main` is protected**, re-measured this session rather than assumed: + `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`. ## What I did NOT verify -- 2.54.0