jakman85andClaude Opus 5
6a6fea8151
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
169 lines
11 KiB
Markdown
169 lines
11 KiB
Markdown
# oauth2-proxy-session — telling the auth layers apart from the sidecar log
|
|
|
|
**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
|
|
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
|
|
proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
|
|
signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
|
|
|
|
> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
|
|
> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
|
|
|
|
## What this means here
|
|
|
|
Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
|
|
stacked on it, and the symptom rarely names the one that refused:
|
|
|
|
1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
|
|
form, or an error page from Keycloak's own hostname.
|
|
2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
|
|
(`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
|
|
this page is about reading it.
|
|
3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
|
|
*behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
|
|
|
|
The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
|
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
|
|
|
|
Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
|
|
`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
|
|
already changed once. Do not run two `dev.sh` invocations at the same time.
|
|
|
|
### The four lines, and what each one means
|
|
|
|
| Log line | What actually happened | Which layer |
|
|
|---|---|---|
|
|
| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
|
|
| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
|
|
| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
|
|
| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
|
|
|
|
That last column is the one people miss. The upstream field sits between the method and the quoted
|
|
path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
|
|
it was proxied to the app. Count them:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
|
|
-l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
|
|
| grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
|
|
|
|
If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
|
|
log had `-`.
|
|
|
|
## First commands
|
|
|
|
**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
|
|
go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
|
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
|
|
|
|
**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
|
|
If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
|
|
is not refreshing it:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
|
|
-l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
|
|
| grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
|
|
|
|
The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
|
|
spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
|
|
|
|
**3. Is this a cookie collision?** Two checks, and you want both.
|
|
|
|
*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
|
|
`.home.keithjackson.dev` domain is in the collision set with every other one:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
|
|
| grep -E "cookie-name|cookie-domain"'
|
|
|
|
An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
|
|
flag being absent, not wrong.
|
|
|
|
*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
|
|
line, and put the timestamps next to each other:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
|
echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
|
|
-c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
|
|
|
|
**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
|
|
instances that genuinely cannot read each other. Compare **hashes**, never values:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
|
|
printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
|
|
-o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
|
|
|
|
Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
|
|
before.
|
|
|
|
## Likely causes, most likely first
|
|
|
|
1. **Cookie collision between two per-app proxies.** Several instances emit the default
|
|
`_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
|
|
authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
|
|
session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
|
|
is the whole zone — **delete a session belonging to an app in another namespace**. The victim
|
|
then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
|
|
its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
|
|
makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
|
|
`--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
|
|
scope — cross-host auth is its job.
|
|
2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
|
|
the proxy never spends the refresh token it holds, so the session expires with the access token.
|
|
Set it to about half the measured token lifetime.
|
|
3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
|
|
live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
|
|
it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
|
|
is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
|
|
browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
|
|
discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
|
|
`Accept: application/json`, which most app JavaScript does not send; make it deterministic with
|
|
`--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
|
|
start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
|
|
4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
|
|
session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
|
|
5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
|
|
by the app's own `/login` page is layer 3, not layer 2.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
|
|
hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
|
|
go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
|
|
agent action.
|
|
- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
|
|
(jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
|
|
`_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
|
|
`auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
|
|
is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
|
|
`render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
|
|
erased by the next render anyway. Change the renderer, then regenerate:
|
|
|
|
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
|
|
|
|
**Budget for the blast radius**: because the function is shared, any change to it rewrites the
|
|
sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
|
|
the same commit. There is no way to change one app's sidecar arguments alone. Measured
|
|
2026-09-30. Coordinate before starting — this is a single change, not five.
|
|
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
|
|
`platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
|
|
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
|
|
you what it understood, without touching the cluster:
|
|
|
|
podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
|
|
|
|
Running it with your full proposed argument list prints a `Cookie settings:` line and an
|
|
`API route - Path:` line back at you, and rejects mutually exclusive flags outright
|
|
(`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
|
|
on a service you have just told someone is fixed.
|
|
- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
|
|
that app and does nothing about the collision, which is caused by the *names* matching, not the
|
|
secrets differing.
|
|
- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
|
|
there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
|
|
cookie flag will help.
|
|
- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
|
|
has a unique `--cookie-name`. That combination should not occur; it means something is writing
|
|
that name which is not in this inventory. Record what you measured.
|