Merge pull request 'fix(jackett): root-cause the oauth2-proxy session loss; the fix cannot be scoped to one app' (#30) from fix/jackett-oauth2-proxy-session into main
Reviewed-on: #30 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #30.
This commit is contained in:
jakman85 committed
commit
08deb208a1
4 files changed
+481
No files matched your search
@@ -0,0 +1,273 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: fix/jackett-oauth2-proxy-session
|
||||
title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
|
||||
never fixes it, and I am never re-prompted for login if it is a login issue."*
|
||||
|
||||
**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
|
||||
deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
|
||||
|
||||
Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
|
||||
and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
|
||||
written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
|
||||
is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
|
||||
I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
|
||||
three; do not rest the case on the first.
|
||||
|
||||
**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
|
||||
instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
|
||||
`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
|
||||
pods' own logs, interleaved to the second:
|
||||
|
||||
15:23:17 radarr [AuthSuccess] -> radarr writes _oauth2_proxy
|
||||
15:23:44 jackett "cookie signature not valid, removing session" -> re-login #1
|
||||
15:23:58 radarr "cookie signature not valid, removing session" -> radarr DELETES jackett's
|
||||
15:27:17 jackett [AuthSuccess] -> re-login #2
|
||||
15:27:59 radarr "cookie signature not valid, removing session"
|
||||
15:31:06 jackett [AuthSuccess] -> re-login #3
|
||||
15:31:29 radarr "cookie signature not valid, removing session" <- deletes it again
|
||||
15:31:32 jackett "No valid authentication in request" <- 3 SECONDS later, no cookie
|
||||
|
||||
`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
|
||||
`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
|
||||
by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
|
||||
out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
|
||||
so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
|
||||
|
||||
It also explains the half of the report that looked like user error. **Reloading genuinely does not
|
||||
help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
|
||||
Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
|
||||
**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
|
||||
`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
|
||||
decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
|
||||
few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
|
||||
|
||||
**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
|
||||
Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
|
||||
Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
|
||||
restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
|
||||
ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
|
||||
fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
|
||||
from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
|
||||
15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
|
||||
ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
|
||||
manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
|
||||
bounded, Crafty is evidence **for** the fix.
|
||||
|
||||
**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
|
||||
`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
|
||||
`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
|
||||
(disabled), so the proxy holds a refresh token it has no mandate to spend.
|
||||
|
||||
**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
|
||||
AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
|
||||
consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
|
||||
split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
|
||||
`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
|
||||
browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
|
||||
JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
|
||||
re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
|
||||
nothing logged as a failure anywhere.
|
||||
|
||||
### The fix has one home, and it is shared — this is the blocker
|
||||
|
||||
The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
|
||||
apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
|
||||
|
||||
`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
|
||||
shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
|
||||
declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
|
||||
— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
|
||||
gate rejected it exactly as designed:
|
||||
|
||||
[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
|
||||
[fail] stage render-golden: FAIL -> MERGE GATE: BLOCKED (rc 1)
|
||||
|
||||
So I made the change at its real source instead and **measured the blast radius** before committing
|
||||
to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
|
||||
apps, and `render-golden` requires all five be regenerated in the same commit:
|
||||
|
||||
CHANGED: apps/wave-0/jackett.yaml
|
||||
CHANGED: apps/wave-0/pihole.yaml
|
||||
CHANGED: apps/wave-0/qbittorrent.yaml
|
||||
CHANGED: apps/wave-1/radarr.yaml
|
||||
CHANGED: apps/wave-1/sonarr.yaml
|
||||
|
||||
Four of those five are the files I was explicitly told not to touch, and all four are being edited
|
||||
right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
|
||||
direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
|
||||
branch owns, and two branches rewriting the same generated block across five files is precisely the
|
||||
cross-branch conflict this repo's §7 exists to prevent.
|
||||
|
||||
I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
|
||||
as a single renderer commit on one branch. The intended content, validated below:
|
||||
|
||||
--cookie-name=_oauth2_proxy_{name} # per-app, derived
|
||||
--cookie-domain={host} # host-scoped, derived
|
||||
--whitelist-domain=.{ZONE} # unchanged: redirect targets, not cookie scope
|
||||
--cookie-refresh=2m30s # half the measured 300 s token lifetime
|
||||
(drop --pass-access-token, --set-xauthrequest)
|
||||
|
||||
`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
|
||||
measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
|
||||
either a per-app hook or four more measurements.
|
||||
|
||||
**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
|
||||
and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
|
||||
`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
|
||||
radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
|
||||
starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
|
||||
already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
|
||||
`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
|
||||
`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
|
||||
change. It belongs in the same single renderer commit — which is an argument for cutting that commit
|
||||
soon, not for me to hand-edit a golden a second time after the gate already refused the first.
|
||||
(Crafty is separately hand-written and could take the metrics flag on its own.)
|
||||
|
||||
**Crafty is unaffected by all of this** — its sidecar is hand-written in
|
||||
`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
|
||||
2026-09-29.
|
||||
|
||||
### What this branch does land
|
||||
|
||||
`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
|
||||
the app's own login apart (including the upstream `-` field that means the proxy answered a request
|
||||
the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
|
||||
302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
|
||||
runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
|
||||
than the file merely existing.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
|
||||
cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
|
||||
radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
|
||||
qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
|
||||
operator's whole report.
|
||||
- **Six instances / seven pods are in the collision set** — the brief said five; the shared
|
||||
forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
|
||||
pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
|
||||
`--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
|
||||
- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
|
||||
never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
|
||||
can read each other's cookie — the precondition for the eviction.
|
||||
- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
|
||||
`refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
|
||||
- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
|
||||
- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
|
||||
`adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
|
||||
two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
|
||||
- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
|
||||
`quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
|
||||
against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
|
||||
`Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
|
||||
domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
|
||||
`API route - Path: ^/api/`.
|
||||
- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
|
||||
the proxy would have refused to start. Read off `--help`.
|
||||
- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
|
||||
assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
|
||||
session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
|
||||
`--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
|
||||
session, and it is unavailable while refresh is on. Dropping those two flags is justified on
|
||||
**least privilege and inertness**, not size.
|
||||
- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
|
||||
`Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
|
||||
`/oauth2/auth` is never called.
|
||||
- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
|
||||
inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
|
||||
renderer-only change listed all five apps as changed (transcript above).
|
||||
- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
|
||||
`startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
|
||||
`--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
|
||||
`No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
|
||||
24 h window over a 13.9 h pod and is dominated by the pre-fix process.
|
||||
- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
|
||||
`/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
|
||||
distinct user-visible failures.
|
||||
- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
|
||||
`['http']` on all six per-app sidecars; only the shared proxy has
|
||||
`--metrics-address=0.0.0.0:44180` and a `metrics` port.
|
||||
- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
|
||||
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
|
||||
Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
|
||||
unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
|
||||
not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
|
||||
`ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
|
||||
circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
|
||||
- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
|
||||
reported, and **only one of them ran**:
|
||||
|
||||
gate / gate (pull_request) state=success desc='Successful in 1m41s'
|
||||
gate / conformance (pull_request) state=success desc='Has been skipped'
|
||||
|
||||
That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
|
||||
conformance job did not execute, so the green tick beside it certifies nothing. Read together with
|
||||
the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
|
||||
Anyone reading the PR's check mark alone would conclude otherwise.
|
||||
- **`main` is protected**, re-measured this session rather than assumed:
|
||||
`GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
|
||||
Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
|
||||
change ships. **This is the single most important line in this entry.**
|
||||
- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
|
||||
is proven to start, but nobody has added an indexer through the proposed configuration. The human
|
||||
test is below, for whoever lands the renderer change.
|
||||
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
|
||||
still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
|
||||
read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
|
||||
the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
|
||||
Sonarr's direct `:9117` access.
|
||||
- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
|
||||
`Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
|
||||
4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
|
||||
this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
|
||||
this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
|
||||
`--session-cookie-minimal` is ruled out (above).
|
||||
- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
|
||||
the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
|
||||
credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
|
||||
and the realm is where to look next.
|
||||
- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
|
||||
PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
|
||||
- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
|
||||
The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
|
||||
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
|
||||
is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
|
||||
**unverified by me this session** — I did not re-measure them against the Gitea API.
|
||||
- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
|
||||
better instrument (an exact process boundary rather than a wall-clock window against a 72 h
|
||||
retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
|
||||
per-app ratio over a window longer than the current pod's life is not something I measured at all.
|
||||
- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
|
||||
seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
|
||||
and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
|
||||
sufficient** — the coordinator's framing, and I agree with it on this evidence.
|
||||
- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
|
||||
oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
|
||||
- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
|
||||
per-namespace `for` loops are edited from the scripted forms I actually used.
|
||||
|
||||
### The human test — for whoever lands the renderer fix
|
||||
|
||||
After it merges and the pod rolls (watch for a **new** pod name at 2/2):
|
||||
|
||||
1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
|
||||
2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
|
||||
today** — it is the 15:31:29 line. Return to the Jackett tab.
|
||||
3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
|
||||
4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
|
||||
test; the session currently dies at five.
|
||||
5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
|
||||
call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
|
||||
did not take; check the running pod's args before assuming anything else.
|
||||
6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
|
||||
A 401 page on reload means the regex is too broad — that is a regression.
|
||||
@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
|
||||
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
|
||||
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
|
||||
app never saw). Covers the cookie collision between per-app sidecars sharing the default
|
||||
`_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
|
||||
*other* namespaces and is only legible with two logs side by side — the 5-minute access-token
|
||||
session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
|
||||
POST body silently.
|
||||
- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
|
||||
Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
|
||||
factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
|
||||
|
||||
@@ -85,6 +85,11 @@ configMapGenerator:
|
||||
- node-io=node-io.md
|
||||
- node-memory=node-memory.md
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
||||
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
||||
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
||||
# so the page has to be reachable at 3am without going looking for it in git.
|
||||
- oauth2-proxy-session=oauth2-proxy-session.md
|
||||
# Operator procedure, not an alert page (no alert links to it — the
|
||||
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
# oauth2-proxy-session — telling the auth layers apart from the sidecar log
|
||||
|
||||
**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
|
||||
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
|
||||
proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
|
||||
signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
|
||||
|
||||
> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
|
||||
> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
|
||||
|
||||
## What this means here
|
||||
|
||||
Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
|
||||
stacked on it, and the symptom rarely names the one that refused:
|
||||
|
||||
1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
|
||||
form, or an error page from Keycloak's own hostname.
|
||||
2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
|
||||
(`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
|
||||
this page is about reading it.
|
||||
3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
|
||||
*behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
|
||||
|
||||
The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
||||
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
|
||||
|
||||
Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
|
||||
already changed once. Do not run two `dev.sh` invocations at the same time.
|
||||
|
||||
### The four lines, and what each one means
|
||||
|
||||
| Log line | What actually happened | Which layer |
|
||||
|---|---|---|
|
||||
| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
|
||||
| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
|
||||
| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
|
||||
| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
|
||||
|
||||
That last column is the one people miss. The upstream field sits between the method and the quoted
|
||||
path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
|
||||
it was proxied to the app. Count them:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
||||
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
|
||||
| grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
|
||||
|
||||
If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
|
||||
log had `-`.
|
||||
|
||||
## First commands
|
||||
|
||||
**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
|
||||
go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
||||
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
|
||||
|
||||
**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
|
||||
If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
|
||||
is not refreshing it:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
||||
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
|
||||
| grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
|
||||
|
||||
The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
|
||||
spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
|
||||
|
||||
**3. Is this a cookie collision?** Two checks, and you want both.
|
||||
|
||||
*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
|
||||
`.home.keithjackson.dev` domain is in the collision set with every other one:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
|
||||
| grep -E "cookie-name|cookie-domain"'
|
||||
|
||||
An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
|
||||
flag being absent, not wrong.
|
||||
|
||||
*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
|
||||
line, and put the timestamps next to each other:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
||||
echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
|
||||
-c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
|
||||
|
||||
**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
|
||||
instances that genuinely cannot read each other. Compare **hashes**, never values:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
||||
printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
|
||||
-o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
|
||||
|
||||
Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
|
||||
before.
|
||||
|
||||
### Do not rank apps by their failure ratio
|
||||
|
||||
Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
|
||||
not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
|
||||
manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
|
||||
scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
|
||||
2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
|
||||
paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
|
||||
(43) — polling, not 156 user-visible failures.
|
||||
|
||||
**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
|
||||
that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
|
||||
deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
|
||||
start — so read it there:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
|
||||
-l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
|
||||
|
||||
then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
|
||||
to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **Cookie collision between two per-app proxies.** Several instances emit the default
|
||||
`_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
|
||||
authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
|
||||
session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
|
||||
is the whole zone — **delete a session belonging to an app in another namespace**. The victim
|
||||
then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
|
||||
its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
|
||||
makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
|
||||
`--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
|
||||
scope — cross-host auth is its job.
|
||||
2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
|
||||
the proxy never spends the refresh token it holds, so the session expires with the access token.
|
||||
Set it to about half the measured token lifetime.
|
||||
3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
|
||||
live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
|
||||
it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
|
||||
is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
|
||||
browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
|
||||
discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
|
||||
`Accept: application/json`, which most app JavaScript does not send; make it deterministic with
|
||||
`--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
|
||||
start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
|
||||
4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
|
||||
session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
|
||||
5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
|
||||
by the app's own `/login` page is layer 3, not layer 2.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
|
||||
hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
|
||||
go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
|
||||
agent action.
|
||||
- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
|
||||
(jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
|
||||
`_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
|
||||
`auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
|
||||
is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
|
||||
`render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
|
||||
erased by the next render anyway. Change the renderer, then regenerate:
|
||||
|
||||
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
|
||||
|
||||
**Budget for the blast radius**: because the function is shared, any change to it rewrites the
|
||||
sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
|
||||
the same commit. There is no way to change one app's sidecar arguments alone. Measured
|
||||
2026-09-30. Coordinate before starting — this is a single change, not five.
|
||||
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
|
||||
`platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
|
||||
- **You cannot currently alert on any of this**, and that is worth knowing before you promise
|
||||
monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
|
||||
`metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
|
||||
series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
|
||||
Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
|
||||
log-reading only.
|
||||
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
|
||||
you what it understood, without touching the cluster:
|
||||
|
||||
podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
|
||||
|
||||
Running it with your full proposed argument list prints a `Cookie settings:` line and an
|
||||
`API route - Path:` line back at you, and rejects mutually exclusive flags outright
|
||||
(`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
|
||||
on a service you have just told someone is fixed.
|
||||
- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
|
||||
that app and does nothing about the collision, which is caused by the *names* matching, not the
|
||||
secrets differing.
|
||||
- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
|
||||
there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
|
||||
cookie flag will help.
|
||||
- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
|
||||
has a unique `--cookie-name`. That combination should not occur; it means something is writing
|
||||
that name which is not in this inventory. Record what you measured.
|
||||
Reference in new issue
Block a user