docs(auth): root-cause jackett's session loss; add the oauth2-proxy-session runbook
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
0fbf544d5e
commit
6a6fea8151
4 files changed
+380
No files matched your search
@@ -0,0 +1,199 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: fix/jackett-oauth2-proxy-session
|
||||
title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
|
||||
never fixes it, and I am never re-prompted for login if it is a login issue."*
|
||||
|
||||
**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
|
||||
deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
|
||||
|
||||
Three mechanisms, all measured.
|
||||
|
||||
**1. Cookie collision — and on Jackett it is the live failure, not a latent one.** Six oauth2-proxy
|
||||
instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
|
||||
`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
|
||||
pods' own logs, interleaved to the second:
|
||||
|
||||
15:23:17 radarr [AuthSuccess] -> radarr writes _oauth2_proxy
|
||||
15:23:44 jackett "cookie signature not valid, removing session" -> re-login #1
|
||||
15:23:58 radarr "cookie signature not valid, removing session" -> radarr DELETES jackett's
|
||||
15:27:17 jackett [AuthSuccess] -> re-login #2
|
||||
15:27:59 radarr "cookie signature not valid, removing session"
|
||||
15:31:06 jackett [AuthSuccess] -> re-login #3
|
||||
15:31:29 radarr "cookie signature not valid, removing session" <- deletes it again
|
||||
15:31:32 jackett "No valid authentication in request" <- 3 SECONDS later, no cookie
|
||||
|
||||
`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
|
||||
`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
|
||||
by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
|
||||
out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
|
||||
so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
|
||||
|
||||
It also explains the half of the report that looked like user error. **Reloading genuinely does not
|
||||
help**, because the operator's Radarr tab re-clears the cookie seconds after each new login.
|
||||
|
||||
**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
|
||||
`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
|
||||
`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
|
||||
(disabled), so the proxy holds a refresh token it has no mandate to spend.
|
||||
|
||||
**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
|
||||
AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
|
||||
consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
|
||||
split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
|
||||
`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
|
||||
browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
|
||||
JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
|
||||
re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
|
||||
nothing logged as a failure anywhere.
|
||||
|
||||
### The fix has one home, and it is shared — this is the blocker
|
||||
|
||||
The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
|
||||
apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
|
||||
|
||||
`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
|
||||
shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
|
||||
declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
|
||||
— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
|
||||
gate rejected it exactly as designed:
|
||||
|
||||
[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
|
||||
[fail] stage render-golden: FAIL -> MERGE GATE: BLOCKED (rc 1)
|
||||
|
||||
So I made the change at its real source instead and **measured the blast radius** before committing
|
||||
to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
|
||||
apps, and `render-golden` requires all five be regenerated in the same commit:
|
||||
|
||||
CHANGED: apps/wave-0/jackett.yaml
|
||||
CHANGED: apps/wave-0/pihole.yaml
|
||||
CHANGED: apps/wave-0/qbittorrent.yaml
|
||||
CHANGED: apps/wave-1/radarr.yaml
|
||||
CHANGED: apps/wave-1/sonarr.yaml
|
||||
|
||||
Four of those five are the files I was explicitly told not to touch, and all four are being edited
|
||||
right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
|
||||
direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
|
||||
branch owns, and two branches rewriting the same generated block across five files is precisely the
|
||||
cross-branch conflict this repo's §7 exists to prevent.
|
||||
|
||||
I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
|
||||
as a single renderer commit on one branch. The intended content, validated below:
|
||||
|
||||
--cookie-name=_oauth2_proxy_{name} # per-app, derived
|
||||
--cookie-domain={host} # host-scoped, derived
|
||||
--whitelist-domain=.{ZONE} # unchanged: redirect targets, not cookie scope
|
||||
--cookie-refresh=2m30s # half the measured 300 s token lifetime
|
||||
(drop --pass-access-token, --set-xauthrequest)
|
||||
|
||||
`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
|
||||
measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
|
||||
either a per-app hook or four more measurements.
|
||||
|
||||
**Crafty is unaffected by all of this** — its sidecar is hand-written in
|
||||
`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
|
||||
2026-09-29.
|
||||
|
||||
### What this branch does land
|
||||
|
||||
`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
|
||||
the app's own login apart (including the upstream `-` field that means the proxy answered a request
|
||||
the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
|
||||
302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
|
||||
runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
|
||||
than the file merely existing.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
|
||||
cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
|
||||
radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
|
||||
qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
|
||||
operator's whole report.
|
||||
- **Six instances / seven pods are in the collision set** — the brief said five; the shared
|
||||
forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
|
||||
pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
|
||||
`--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
|
||||
- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
|
||||
never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
|
||||
can read each other's cookie — the precondition for the eviction.
|
||||
- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
|
||||
`refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
|
||||
- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
|
||||
- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
|
||||
`adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
|
||||
two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
|
||||
- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
|
||||
`quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
|
||||
against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
|
||||
`Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
|
||||
domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
|
||||
`API route - Path: ^/api/`.
|
||||
- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
|
||||
the proxy would have refused to start. Read off `--help`.
|
||||
- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
|
||||
assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
|
||||
session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
|
||||
`--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
|
||||
session, and it is unavailable while refresh is on. Dropping those two flags is justified on
|
||||
**least privilege and inertness**, not size.
|
||||
- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
|
||||
`Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
|
||||
`/oauth2/auth` is never called.
|
||||
- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
|
||||
inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
|
||||
renderer-only change listed all five apps as changed (transcript above).
|
||||
- **Gate:** exit code and stage list in the next section.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
|
||||
Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
|
||||
change ships. **This is the single most important line in this entry.**
|
||||
- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
|
||||
is proven to start, but nobody has added an indexer through the proposed configuration. The human
|
||||
test is below, for whoever lands the renderer change.
|
||||
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
|
||||
still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
|
||||
read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
|
||||
the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
|
||||
Sonarr's direct `:9117` access.
|
||||
- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
|
||||
`Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
|
||||
4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
|
||||
this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
|
||||
this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
|
||||
`--session-cookie-minimal` is ruled out (above).
|
||||
- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
|
||||
the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
|
||||
credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
|
||||
and the realm is where to look next.
|
||||
- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
|
||||
PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
|
||||
- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
|
||||
The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
|
||||
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
|
||||
is certified by the gate.
|
||||
- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
|
||||
per-namespace `for` loops are edited from the scripted forms I actually used.
|
||||
|
||||
### The human test — for whoever lands the renderer fix
|
||||
|
||||
After it merges and the pod rolls (watch for a **new** pod name at 2/2):
|
||||
|
||||
1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
|
||||
2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
|
||||
today** — it is the 15:31:29 line. Return to the Jackett tab.
|
||||
3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
|
||||
4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
|
||||
test; the session currently dies at five.
|
||||
5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
|
||||
call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
|
||||
did not take; check the running pod's args before assuming anything else.
|
||||
6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
|
||||
A 401 page on reload means the regex is too broad — that is a regression.
|
||||
@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
|
||||
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
|
||||
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
|
||||
app never saw). Covers the cookie collision between per-app sidecars sharing the default
|
||||
`_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
|
||||
*other* namespaces and is only legible with two logs side by side — the 5-minute access-token
|
||||
session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
|
||||
POST body silently.
|
||||
- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
|
||||
Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
|
||||
factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
|
||||
|
||||
@@ -85,6 +85,11 @@ configMapGenerator:
|
||||
- node-io=node-io.md
|
||||
- node-memory=node-memory.md
|
||||
- node-memory-swap=node-memory-swap.md
|
||||
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
||||
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
||||
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
||||
# so the page has to be reachable at 3am without going looking for it in git.
|
||||
- oauth2-proxy-session=oauth2-proxy-session.md
|
||||
# Operator procedure, not an alert page (no alert links to it — the
|
||||
# alert-runbooks-resolvable check treats that as advisory). It is served
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
# oauth2-proxy-session — telling the auth layers apart from the sidecar log
|
||||
|
||||
**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
|
||||
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
|
||||
proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
|
||||
signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
|
||||
|
||||
> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
|
||||
> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
|
||||
|
||||
## What this means here
|
||||
|
||||
Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
|
||||
stacked on it, and the symptom rarely names the one that refused:
|
||||
|
||||
1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
|
||||
form, or an error page from Keycloak's own hostname.
|
||||
2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
|
||||
(`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
|
||||
this page is about reading it.
|
||||
3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
|
||||
*behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
|
||||
|
||||
The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
||||
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
|
||||
|
||||
Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
|
||||
already changed once. Do not run two `dev.sh` invocations at the same time.
|
||||
|
||||
### The four lines, and what each one means
|
||||
|
||||
| Log line | What actually happened | Which layer |
|
||||
|---|---|---|
|
||||
| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
|
||||
| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
|
||||
| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
|
||||
| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
|
||||
|
||||
That last column is the one people miss. The upstream field sits between the method and the quoted
|
||||
path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
|
||||
it was proxied to the app. Count them:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
||||
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
|
||||
| grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
|
||||
|
||||
If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
|
||||
log had `-`.
|
||||
|
||||
## First commands
|
||||
|
||||
**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
|
||||
go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
||||
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
|
||||
|
||||
**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
|
||||
If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
|
||||
is not refreshing it:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
||||
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
|
||||
| grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
|
||||
|
||||
The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
|
||||
spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
|
||||
|
||||
**3. Is this a cookie collision?** Two checks, and you want both.
|
||||
|
||||
*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
|
||||
`.home.keithjackson.dev` domain is in the collision set with every other one:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
|
||||
| grep -E "cookie-name|cookie-domain"'
|
||||
|
||||
An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
|
||||
flag being absent, not wrong.
|
||||
|
||||
*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
|
||||
line, and put the timestamps next to each other:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
||||
echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
|
||||
-c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
|
||||
|
||||
**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
|
||||
instances that genuinely cannot read each other. Compare **hashes**, never values:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
||||
printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
|
||||
-o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
|
||||
|
||||
Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
|
||||
before.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **Cookie collision between two per-app proxies.** Several instances emit the default
|
||||
`_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
|
||||
authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
|
||||
session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
|
||||
is the whole zone — **delete a session belonging to an app in another namespace**. The victim
|
||||
then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
|
||||
its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
|
||||
makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
|
||||
`--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
|
||||
scope — cross-host auth is its job.
|
||||
2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
|
||||
the proxy never spends the refresh token it holds, so the session expires with the access token.
|
||||
Set it to about half the measured token lifetime.
|
||||
3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
|
||||
live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
|
||||
it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
|
||||
is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
|
||||
browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
|
||||
discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
|
||||
`Accept: application/json`, which most app JavaScript does not send; make it deterministic with
|
||||
`--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
|
||||
start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
|
||||
4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
|
||||
session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
|
||||
5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
|
||||
by the app's own `/login` page is layer 3, not layer 2.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
|
||||
hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
|
||||
go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
|
||||
agent action.
|
||||
- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
|
||||
(jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
|
||||
`_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
|
||||
`auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
|
||||
is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
|
||||
`render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
|
||||
erased by the next render anyway. Change the renderer, then regenerate:
|
||||
|
||||
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
|
||||
|
||||
**Budget for the blast radius**: because the function is shared, any change to it rewrites the
|
||||
sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
|
||||
the same commit. There is no way to change one app's sidecar arguments alone. Measured
|
||||
2026-09-30. Coordinate before starting — this is a single change, not five.
|
||||
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
|
||||
`platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
|
||||
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
|
||||
you what it understood, without touching the cluster:
|
||||
|
||||
podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
|
||||
|
||||
Running it with your full proposed argument list prints a `Cookie settings:` line and an
|
||||
`API route - Path:` line back at you, and rejects mutually exclusive flags outright
|
||||
(`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
|
||||
on a service you have just told someone is fixed.
|
||||
- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
|
||||
that app and does nothing about the collision, which is caused by the *names* matching, not the
|
||||
secrets differing.
|
||||
- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
|
||||
there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
|
||||
cookie flag will help.
|
||||
- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
|
||||
has a unique `--cookie-name`. That combination should not occur; it means something is writing
|
||||
that name which is not in this inventory. Record what you measured.
|
||||
Reference in new issue
Block a user