home-network-k3s-impl/runbooks/oauth2-proxy-session.md

# oauth2-proxy-session — telling the auth layers apart from the sidecar log

**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
signed out at the same time: the page loads, but its buttons and saves quietly do nothing.

> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)

## What this means here

Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
stacked on it, and the symptom rarely names the one that refused:

1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
   form, or an error page from Keycloak's own hostname.
2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
   this page is about reading it.
3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.

The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'

Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
already changed once. Do not run two `dev.sh` invocations at the same time.

### The four lines, and what each one means

| Log line | What actually happened | Which layer |
|---|---|---|
| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |

That last column is the one people miss. The upstream field sits between the method and the quoted
path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
it was proxied to the app. Count them:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'

If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
log had `-`.

## First commands

**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'

**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
is not refreshing it:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'

The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.

**3. Is this a cookie collision?** Two checks, and you want both.

*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
`.home.keithjackson.dev` domain is in the collision set with every other one:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
      | grep -E "cookie-name|cookie-domain"'

An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
flag being absent, not wrong.

*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
line, and put the timestamps next to each other:

    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'

**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
instances that genuinely cannot read each other. Compare **hashes**, never values:

    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'

Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
before.

### Do not rank apps by their failure ratio

Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
(43) — polling, not 156 user-visible failures.

**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
start — so read it there:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'

then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.

## Likely causes, most likely first

1. **Cookie collision between two per-app proxies.** Several instances emit the default
   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
   scope — cross-host auth is its job.
2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
   the proxy never spends the refresh token it holds, so the session expires with the access token.
   Set it to about half the measured token lifetime.
3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
   by the app's own `/login` page is layer 3, not layer 2.

## When to escalate or stop

- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
  agent action.
- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
  erased by the next render anyway. Change the renderer, then regenerate:

      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation

  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
  the same commit. There is no way to change one app's sidecar arguments alone. Measured
  2026-09-30. Coordinate before starting — this is a single change, not five.
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
- **You cannot currently alert on any of this**, and that is worth knowing before you promise
  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
  log-reading only.
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
  you what it understood, without touching the cluster:

      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help

  Running it with your full proposed argument list prints a `Cookie settings:` line and an
  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
  on a service you have just told someone is fixed.
- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
  that app and does nothing about the collision, which is caused by the *names* matching, not the
  secrets differing.
- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
  cookie flag will help.
- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
  has a unique `--cookie-name`. That combination should not occur; it means something is writing
  that name which is not in this inventory. Record what you measured.