home-network-k3s-impl/runbooks/oauth2-proxy-session.md

oauth2-proxy-session — telling the auth layers apart from the sidecar log

Not alert-driven. This is an operator procedure for the per-app oauth2-proxy sidecars (jackett, pihole, qbittorrent, radarr, sonarr, crafty) and the shared forward-auth proxy in namespace oauth2-proxy. Use it when a gated UI behaves as though you are signed in and signed out at the same time: the page loads, but its buttons and saves quietly do nothing.

The sidecar log contains email addresses, session UUIDs and full callback URLs including an OAuth code. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)

What this means here

Every gated app on home.keithjackson.dev has at least two independent authentications stacked on it, and the symptom rarely names the one that refused:

1. Keycloak — the homelab realm issues the token. Failures here are visible: you see a login form, or an error page from Keycloak's own hostname.
2. The oauth2-proxy sidecar — validates the session cookie and the group claim (media-users / platform-admins). This is the layer that fails silently, and the rest of this page is about reading it.
3. The app's own login, where it has one — Crafty and Radarr/Sonarr each present a local login behind the Keycloak gate. See runbooks/crafty-admin-credential.md for that layer.

The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
  -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'

Substitute the namespace and label for the app you are chasing. Always pass --context k3s-prod: dev.sh merges both clusters and picks a current-context per invocation, and that default has already changed once. Do not run two dev.sh invocations at the same time.

The four lines, and what each one means

Log line	What actually happened	Which layer
[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}	Keycloak issued a token and the proxy accepted it. The IdP is fine.	Keycloak ✔
[oauthproxy.go:1026] No valid authentication in request. Initiating login.	No session cookie was presented at all. Either you never had one, or something deleted it — see below.	sidecar
[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session	A cookie of the right name arrived that this proxy could not decrypt. Almost always a different proxy's cookie. Note removing session: this proxy then deletes it.	sidecar, cookie collision
an upstream path of - instead of / in an access-log line	the proxy answered the request itself; it never reached the app. Compare POST - "/api/..." against POST / "/api/...".	sidecar

That last column is the one people miss. The upstream field sits between the method and the quoted path. - means the proxy handled it (a redirect, a 401, its own /oauth2/* endpoints); / means it was proxied to the app. Count them:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
  -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
  | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'

If your writes show -, the app never saw them. On 2026-09-30, 4 of the 6 POSTs in jackett's log had -.

First commands

1. Did Keycloak ever succeed? No AuthSuccess at all means this is not a cookie problem — go to runbooks/keycloak-down.md or runbooks/keycloak-client-secrets.md.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
  -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'

2. How long does a session last? Subtract created from expires on an AuthSuccess line. If it is exactly 5m, the session's lifetime is Keycloak's access-token lifetime and the proxy is not refreshing it:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
  -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
  | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'

The same line says refresh_token:true — the proxy is holding a refresh token. It will only spend it if --cookie-refresh is set; the default is 0, meaning disabled.

3. Is this a cookie collision? Two checks, and you want both.

Names and scopes — every instance that emits the default name _oauth2_proxy on the shared .home.keithjackson.dev domain is in the collision set with every other one:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
  | grep -E "cookie-name|cookie-domain"'

An instance with no --cookie-name argument is using the default. That is the failure — the flag being absent, not wrong.

The smoking gun — search every proxy, not just the one you are debugging, for the removal line, and put the timestamps next to each other:

infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
  echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
  -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'

4. Do the cookie-secrets actually differ? They should — but confirm the collision is between instances that genuinely cannot read each other. Compare hashes, never values:

infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
  printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
  -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'

Never print .data or .stringData itself. A password has been leaked into a transcript here before.

Do not rank apps by their failure ratio

Counting No valid authentication against AuthSuccess per app looks like a health metric and is not one. It ranks chattiness, not breakage. An app whose UI polls or holds a WebSocket manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured 2026-09-30: Radarr showed 156 unauthenticated requests against 1 AuthSuccess, but the top paths were /api/v3/command (62), /api/v3/queue/details (46) and /signalr/messages/negotiate (43) — polling, not 156 user-visible failures.

Bound the window to the current pod, not to wall-clock. A ratio measured over 24 h across a pod that restarted 13 h ago is mostly about the previous configuration, and that is exactly how a deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process start — so read it there:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
  -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'

then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h to 3 failures in its fixed pod's entire 13 h 52 m life, two of them /ws reconnects.

Likely causes, most likely first

1. Cookie collision between two per-app proxies. Several instances emit the default _oauth2_proxy on .home.keithjackson.dev with different cookie-secrets. Whoever authenticates last overwrites the rest; the losers log cookie signature not valid, removing session and — because removing session clears the cookie on their --cookie-domain, which is the whole zone — delete a session belonging to an app in another namespace. The victim then logs No valid authentication with no cookie at all, seconds after a clean login, while its own expires is still minutes away. Read both logs side by side; one app's log alone makes this look like random session loss. Fix: a unique --cookie-name and a host-scoped --cookie-domain on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide scope — cross-host auth is its job.
2. Session pinned to a 5-minute access token with no refresh. --cookie-refresh unset means the proxy never spends the refresh token it holds, so the session expires with the access token. Set it to about half the measured token lifetime.
3. An XHR got a 302 instead of a 401. The app's JavaScript follows the redirect invisibly, the live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with HTML where it expected JSON. Nothing is logged as an error and the operator is never re-prompted — this is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a browser following a 302 re-issues a POST as a GET and drops the body, so the change is discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact Accept: application/json, which most app JavaScript does not send; make it deterministic with --api-route=<regex> (singular — --api-routes does not exist and the proxy refuses to start). Keep browser-navigable paths out of the regex so a plain page load still signs you in.
4. Group membership. An AuthSuccess followed by 403 is the --allowed-group gate, not a session problem. Check the groups:[...] field on the AuthSuccess line.
5. The app's own second login. Crafty, Radarr and Sonarr each have one. AuthSuccess followed by the app's own /login page is layer 3, not layer 2.

When to escalate or stop

• Stop before you kubectl edit/patch/apply anything. Flux reconciles main; a hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an agent action.

• Do not hand-edit apps/**/<app>.yaml — those files are GENERATED. The five per-app sidecars (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function, _oauth2_sidecar_container() in catalog/render.py; catalog/<app>.yaml declares only auth.mode and auth.groups, and the auth schema is additionalProperties: false, so there is no per-app override hook. A hand-edit to the rendered file is caught by the merge gate's render-golden stage ("GOLDEN DRIFT ... does not match the canonical render") and would be erased by the next render anyway. Change the renderer, then regenerate:

  python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
  

  Budget for the blast radius: because the function is shared, any change to it rewrites the sidecar block in all five rendered apps, and the gate requires all five be regenerated in the same commit. There is no way to change one app's sidecar arguments alone. Measured 2026-09-30. Coordinate before starting — this is a single change, not five.

• Crafty is the exception and is not rendered from the catalog: its sidecar is hand-written in platform/crafty/deployment.yaml, so it can be, and was, fixed independently.

• You cannot currently alert on any of this, and that is worth knowing before you promise monitoring. Measured 2026-09-30: no per-app sidecar sets --metrics-address and none declares a metrics container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy series in Prometheus come from the shared proxy in namespace oauth2-proxy, which does set both. Until the renderer emits them for the per-app sidecars, every diagnosis on this page is log-reading only.

• Validate an argument change before you open the PR, not after. The pinned image will tell you what it understood, without touching the cluster:

  podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
  

  Running it with your full proposed argument list prints a Cookie settings: line and an API route - Path: line back at you, and rejects mutually exclusive flags outright (--session-cookie-minimal with --cookie-refresh, for one). A flag typo here is a CrashLoop on a service you have just told someone is fixed.

• Do not rotate a cookie-secret to "clear" a bad session. It signs every live session out of that app and does nothing about the collision, which is caused by the names matching, not the secrets differing.

• Escalate to Keycloak (runbooks/keycloak-down.md, runbooks/keycloak-client-secrets.md) if there is no AuthSuccess in any proxy's log — that is an IdP or client-secret problem and no cookie flag will help.

• Hand off rather than guess if you see cookie signature not valid on a proxy that already has a unique --cookie-name. That combination should not occur; it means something is writing that name which is not in this inventory. Record what you measured.