diff --git a/decisions/2026-09-30-jackett-oauth2-proxy-session.md b/decisions/2026-09-30-jackett-oauth2-proxy-session.md new file mode 100644 index 0000000..cb68515 --- /dev/null +++ b/decisions/2026-09-30-jackett-oauth2-proxy-session.md @@ -0,0 +1,199 @@ +--- +date: 2026-09-30 +branch: fix/jackett-oauth2-proxy-session +title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app +--- + +## What changed + +Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page +never fixes it, and I am never re-prompted for login if it is a login issue."* + +**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a +deliberate stop, not an omission** — see *The fix has one home and it is shared*, below. + +Three mechanisms, all measured. + +**1. Cookie collision — and on Jackett it is the live failure, not a latent one.** Six oauth2-proxy +instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to +`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two +pods' own logs, interleaved to the second: + + 15:23:17 radarr [AuthSuccess] -> radarr writes _oauth2_proxy + 15:23:44 jackett "cookie signature not valid, removing session" -> re-login #1 + 15:23:58 radarr "cookie signature not valid, removing session" -> radarr DELETES jackett's + 15:27:17 jackett [AuthSuccess] -> re-login #2 + 15:27:59 radarr "cookie signature not valid, removing session" + 15:31:06 jackett [AuthSuccess] -> re-login #3 + 15:31:29 radarr "cookie signature not valid, removing session" <- deletes it again + 15:31:32 jackett "No valid authentication in request" <- 3 SECONDS later, no cookie + +`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own** +`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted +by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes +out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window, +so plain expiry does not explain it"): it was never expiry, and it was never Jackett. + +It also explains the half of the report that looked like user error. **Reloading genuinely does not +help**, because the operator's Radarr tab re-clears the cookie seconds after each new login. + +**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own +`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with +`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0` +(disabled), so the proxy holds a refresh token it has no mandate to spend. + +**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in +AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not +consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the +split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401, +`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the +browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the +JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302 +re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with +nothing logged as a failure anywhere. + +### The fix has one home, and it is shared — this is the blocker + +The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four +apps to a sibling branch. **That split is not implementable, and I found out by trying it.** + +`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one +shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml` +declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false` +— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge +gate rejected it exactly as designed: + + [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render + [fail] stage render-golden: FAIL -> MERGE GATE: BLOCKED (rc 1) + +So I made the change at its real source instead and **measured the blast radius** before committing +to it. A change confined to that one function rewrites the sidecar block in **all five** rendered +apps, and `render-golden` requires all five be regenerated in the same commit: + + CHANGED: apps/wave-0/jackett.yaml + CHANGED: apps/wave-0/pihole.yaml + CHANGED: apps/wave-0/qbittorrent.yaml + CHANGED: apps/wave-1/radarr.yaml + CHANGED: apps/wave-1/sonarr.yaml + +Four of those five are the files I was explicitly told not to touch, and all four are being edited +right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in +direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another +branch owns, and two branches rewriting the same generated block across five files is precisely the +cross-branch conflict this repo's §7 exists to prevent. + +I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut +as a single renderer commit on one branch. The intended content, validated below: + + --cookie-name=_oauth2_proxy_{name} # per-app, derived + --cookie-domain={host} # host-scoped, derived + --whitelist-domain=.{ZONE} # unchanged: redirect targets, not cookie scope + --cookie-refresh=2m30s # half the measured 300 s token lifetime + (drop --pass-access-token, --set-xauthrequest) + +`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I +measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs +either a per-app hook or four more measurements. + +**Crafty is unaffected by all of this** — its sidecar is hand-written in +`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on +2026-09-29. + +### What this branch does land + +`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and +the app's own login apart (including the upstream `-` field that means the proxy answered a request +the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the +302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the +runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather +than the file merely existing. + +## What I verified + +- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session: + cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in + radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole, + qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the + operator's whole report. +- **Six instances / seven pods are in the collision set** — the brief said five; the shared + forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live + pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no** + `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed. +- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values + never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies + can read each other's cookie — the precondition for the eviction. +- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with + `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent. +- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204. +- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`: + `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The + two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email. +- **The proposed argument list starts, against the pinned image and the real issuer.** Ran + `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery + against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading: + `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s + domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and + `API route - Path: ^/api/`. +- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1; + the proxy would have refused to start. Read off `--help`. +- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not + assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions. + session_cookie_minimal cannot be set`. This kills the brief's premise that dropping + `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the + session, and it is unavailable while refresh is on. Dropping those two flags is justified on + **least privilege and inertness**, not size. +- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is + `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so + `/oauth2/auth` is never called. +- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not + inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a + renderer-only change listed all five apps as changed (transcript above). +- **Gate:** exit code and stage list in the next section. + +## What I did NOT verify + +- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land. + Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer + change ships. **This is the single most important line in this entry.** +- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list + is proven to start, but nobody has added an indexer through the proposed configuration. The human + test is below, for whoever lands the renderer change. +- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is + still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's + read-only probes). Note that when the fix does land, merging is what deploys, and it will restart + the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and + Sonarr's direct `:9117` access. +- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a + `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no + 4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at + this log level, so that absence is not evidence. If sessions still misbehave after the fix lands, + this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since + `--session-cookie-minimal` is ruled out (above). +- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting + the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin + credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists + and the realm is where to look next. +- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and + PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence. +- **The four sibling apps still collide with each other** regardless of what happens to Jackett. + The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr. +- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour + is certified by the gate. +- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the + per-namespace `for` loops are edited from the scripted forms I actually used. + +### The human test — for whoever lands the renderer fix + +After it merges and the pod rolls (watch for a **new** pod name at 2/2): + +1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`. +2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett + today** — it is the 15:31:29 line. Return to the Jackett tab. +3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save. +4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh` + test; the session currently dies at five. +5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/` + call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change + did not take; check the running pod's args before assuming anything else. +6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`). + A 401 page on reload means the regex is too broad — that is a regression. diff --git a/runbooks/README.md b/runbooks/README.md index 9c859fd..2368296 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. +- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never + re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log + (the four lines that matter, including the upstream `-` that means the proxy answered a request the + app never saw). Covers the cookie collision between per-app sidecars sharing the default + `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in + *other* namespaces and is only legible with two logs side by side — the 5-minute access-token + session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a + POST body silently. - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password. Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 08793d7..4e341bf 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -85,6 +85,11 @@ configMapGenerator: - node-io=node-io.md - node-memory=node-memory.md - node-memory-swap=node-memory-swap.md + # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell + # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars + # presents as random session loss in ONE app's log and is only legible with both side by side, + # so the page has to be reachable at 3am without going looking for it in git. + - oauth2-proxy-session=oauth2-proxy-session.md # Operator procedure, not an alert page (no alert links to it — the # alert-runbooks-resolvable check treats that as advisory). It is served # because the plex.tv claim window is short and is not a thing to go looking diff --git a/runbooks/oauth2-proxy-session.md b/runbooks/oauth2-proxy-session.md new file mode 100644 index 0000000..85cd017 --- /dev/null +++ b/runbooks/oauth2-proxy-session.md @@ -0,0 +1,168 @@ +# oauth2-proxy-session — telling the auth layers apart from the sidecar log + +**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars +(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth +proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and +signed out at the same time: the page loads, but its buttons and saves quietly do nothing. + +> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an +> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.) + +## What this means here + +Every gated app on `home.keithjackson.dev` has **at least two** independent authentications +stacked on it, and the symptom rarely names the one that refused: + +1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login + form, or an error page from Keycloak's own hostname. +2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim + (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of + this page is about reading it. +3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login + *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer. + +The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \ + -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1' + +Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`: +`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has +already changed once. Do not run two `dev.sh` invocations at the same time. + +### The four lines, and what each one means + +| Log line | What actually happened | Which layer | +|---|---|---| +| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ | +| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar | +| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision | +| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar | + +That last column is the one people miss. The upstream field sits between the method and the quoted +path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means +it was proxied to the app. Count them: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \ + -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \ + | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140' + +If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's +log had `-`. + +## First commands + +**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem — +go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`. + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n logs \ + -l app.kubernetes.io/name= -c oauth2-proxy --tail=-1 | grep -c AuthSuccess' + +**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line. +If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy +is not refreshing it: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n logs \ + -l app.kubernetes.io/name= -c oauth2-proxy --tail=-1 \ + | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"' + +The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only +spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled. + +**3. Is this a cookie collision?** Two checks, and you want both. + +*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared +`.home.keithjackson.dev` domain is in the collision set with every other one: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \ + | grep -E "cookie-name|cookie-domain"' + +An instance with **no** `--cookie-name` argument is using the default. That is the failure — the +flag being absent, not wrong. + +*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal +line, and put the timestamps next to each other: + + infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \ + echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \ + -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done' + +**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between +instances that genuinely cannot read each other. Compare **hashes**, never values: + + infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \ + printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \ + -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done' + +Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here +before. + +## Likely causes, most likely first + +1. **Cookie collision between two per-app proxies.** Several instances emit the default + `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever + authenticates last overwrites the rest; the losers log `cookie signature not valid, removing + session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which + is the whole zone — **delete a session belonging to an app in another namespace**. The victim + then logs `No valid authentication` with no cookie at all, seconds after a clean login, while + its own `expires` is still minutes away. **Read both logs side by side; one app's log alone + makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped + `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide + scope — cross-host auth is its job. +2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means + the proxy never spends the refresh token it holds, so the session expires with the access token. + Set it to about half the measured token lifetime. +3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the + live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where + it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this + is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a + browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is + discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact + `Accept: application/json`, which most app JavaScript does not send; make it deterministic with + `--api-route=` (**singular** — `--api-routes` does not exist and the proxy refuses to + start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in. +4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a + session problem. Check the `groups:[...]` field on the `AuthSuccess` line. +5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed + by the app's own `/login` page is layer 3, not layer 2. + +## When to escalate or stop + +- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a + hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes + go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an + agent action. +- **Do not hand-edit `apps/**/.yaml` — those files are GENERATED.** The five per-app sidecars + (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function, + `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/.yaml` declares only + `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there + is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's + `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be + erased by the next render anyway. Change the renderer, then regenerate: + + python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation + + **Budget for the blast radius**: because the function is shared, any change to it rewrites the + sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in + the same commit. There is no way to change one app's sidecar arguments alone. Measured + 2026-09-30. Coordinate before starting — this is a single change, not five. +- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in + `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently. +- **Validate an argument change before you open the PR**, not after. The pinned image will tell + you what it understood, without touching the cluster: + + podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help + + Running it with your full proposed argument list prints a `Cookie settings:` line and an + `API route - Path:` line back at you, and rejects mutually exclusive flags outright + (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop + on a service you have just told someone is fixed. +- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of + that app and does nothing about the collision, which is caused by the *names* matching, not the + secrets differing. +- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if + there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no + cookie flag will help. +- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already + has a unique `--cookie-name`. That combination should not occur; it means something is writing + that name which is not in this inventory. Record what you measured.