feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy.*"}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 10:55:19 -05:00
10 changed files with 640 additions and 481 deletions
@@ -1,273 +0,0 @@
 ---
 date: 2026-09-30
 branch: fix/jackett-oauth2-proxy-session
 title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
 ---
 ## What changed
 Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
 never fixes it, and I am never re-prompted for login if it is a login issue."*
 **This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
 deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
 Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
 and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
 written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
 is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
 I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
 three; do not rest the case on the first.
 **1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
 instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
 `.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
 pods' own logs, interleaved to the second:
    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
    15:27:59  radarr   "cookie signature not valid, removing session"
    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
 `removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
 `--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
 by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
 out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
 so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
 It also explains the half of the report that looked like user error. **Reloading genuinely does not
 help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
 Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
 **1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
 `/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
 decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
 few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
 **A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
 Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
 Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
 restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
 ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
 fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
 from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
 15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
 ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
 manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
 bounded, Crafty is evidence **for** the fix.
 **2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
 `[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
 `refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
 (disabled), so the proxy holds a refresh token it has no mandate to spend.
 **3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
 AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
 consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
 split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
 `/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
 browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
 JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
 re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
 nothing logged as a failure anywhere.
 ### The fix has one home, and it is shared — this is the blocker
 The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
 apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
 `apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
 shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
 declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
 — **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
 gate rejected it exactly as designed:
    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
 So I made the change at its real source instead and **measured the blast radius** before committing
 to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
 apps, and `render-golden` requires all five be regenerated in the same commit:
    CHANGED: apps/wave-0/jackett.yaml
    CHANGED: apps/wave-0/pihole.yaml
    CHANGED: apps/wave-0/qbittorrent.yaml
    CHANGED: apps/wave-1/radarr.yaml
    CHANGED: apps/wave-1/sonarr.yaml
 Four of those five are the files I was explicitly told not to touch, and all four are being edited
 right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
 direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
 branch owns, and two branches rewriting the same generated block across five files is precisely the
 cross-branch conflict this repo's §7 exists to prevent.
 I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
 as a single renderer commit on one branch. The intended content, validated below:
    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
    --cookie-domain={host}                    # host-scoped, derived
    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
    (drop --pass-access-token, --set-xauthrequest)
 `--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
 measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
 either a per-app hook or four more measurements.
 **Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
 and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
 `--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
 radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
 starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
 already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
 `args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
 `--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
 change. It belongs in the same single renderer commit — which is an argument for cutting that commit
 soon, not for me to hand-edit a golden a second time after the gate already refused the first.
 (Crafty is separately hand-written and could take the metrics flag on its own.)
 **Crafty is unaffected by all of this** — its sidecar is hand-written in
 `platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
 2026-09-29.
 ### What this branch does land
 `runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
 the app's own login apart (including the upstream `-` field that means the proxy answered a request
 the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
 302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
 runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
 than the file merely existing.
 ## What I verified
 - **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
  operator's whole report.
 - **Six instances / seven pods are in the collision set** — the brief said five; the shared
  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
 - **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
  can read each other's cookie — the precondition for the eviction.
 - **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
 - **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
 - **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
 - **The proposed argument list starts, against the pinned image and the real issuer.** Ran
  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
  `API route - Path: ^/api/`.
 - **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
  the proxy would have refused to start. Read off `--help`.
 - **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
  **least privilege and inertness**, not size.
 - **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
  `/oauth2/auth` is never called.
 - **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
  renderer-only change listed all five apps as changed (transcript above).
 - **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
 - **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
  distinct user-visible failures.
 - **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
  `['http']` on all six per-app sidecars; only the shared proxy has
  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
 - **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
 - **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
  reported, and **only one of them ran**:
      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
      gate / conformance (pull_request)  state=success   desc='Has been skipped'
  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
  Anyone reading the PR's check mark alone would conclude otherwise.
 - **`main` is protected**, re-measured this session rather than assumed:
  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
 ## What I did NOT verify
 - **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
  change ships. **This is the single most important line in this entry.**
 - **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
  is proven to start, but nobody has added an indexer through the proposed configuration. The human
  test is below, for whoever lands the renderer change.
 - **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
  Sonarr's direct `:9117` access.
 - **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
  `--session-cookie-minimal` is ruled out (above).
 - **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
  and the realm is where to look next.
 - **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
 - **The four sibling apps still collide with each other** regardless of what happens to Jackett.
  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
 - **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
  **unverified by me this session** — I did not re-measure them against the Gitea API.
 - **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
  per-app ratio over a window longer than the current pod's life is not something I measured at all.
 - **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
  sufficient** — the coordinator's framing, and I agree with it on this evidence.
 - **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
 - **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
  per-namespace `for` loops are edited from the scripted forms I actually used.
 ### The human test — for whoever lands the renderer fix
 After it merges and the pod rolls (watch for a **new** pod name at 2/2):
 1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
 2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
   today** — it is the 15:31:29 line. Return to the Jackett tab.
 3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
 4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
   test; the session currently dies at five.
 5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
   did not take; check the running pod's args before assuming anything else.
 6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
   A 401 page on reload means the regex is too broad — that is a regression.
@@ -0,0 +1,175 @@
 ---
 date: 2026-09-30
 branch: feat/oauth2-proxy-auth-failure-visible
 title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
 ---
 ## What changed
 Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
 operator's report named the property that matters — *"reloading the page never fixes the issue and
 the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
 about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
 the alert rules. It is an absence of data.
 **The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
 That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
 jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
 `--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
 starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
 cluster selects them.
 So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
 a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
 exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
 permanently silent, and indistinguishable from health. It was not written.
 What was written instead alerts on the condition that made the incident invisible, from families
 that **are** scraped:
 - **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
        unless
      count by (namespace) (up{container="oauth2-proxy"})
  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
  clears per-namespace as each gate gains a real scrape.
 - **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
  prod, not invented.
 - **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
  `configMapGenerator` and listed in `runbooks/README.md`.
 - **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
  per-app sidecars, plus the measured reason.
 - **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
  three-part change that would close the gap.
 **Why the obvious alternative was rejected.** An alert on the *shared* proxy's
 `oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
 need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
 and — decisively — **it would not have fired on this incident**: the shared instance is not in the
 data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
 at all, having proxied nothing since they started (the counter exists in the TSDB only from an
 older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
 Shipping it would have looked like coverage of the reported failure and been none.
 ## What I verified
 Everything below was measured against live prod through the pinned container, read-only. No
 `apply`, no `edit`, no `patch`, no `flux reconcile`.
 - **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
  `1`, `namespace="oauth2-proxy"`.
 - **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
  `oauth2-proxy` container, with its declared ports: the two shared replicas have
  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
 - **No existing alert is inert *for this reason*, because no alert references these metrics at
  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
 - **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
  observable from the IdP side.
 - **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
 - **The alert expression fires against live data.** Run verbatim on prod:
  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
  radarr`, value 1 each; `oauth2-proxy` correctly absent.
 - **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
  runbook as the thing to compare against.
 - **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
 - **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
  `runbook_url` points at exactly that name.
 - **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
  per-stage outcomes in the next section — read them there, not here.
 ## What I did NOT verify
 - **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
  only in git on this branch. What is proven is that its expression returns the expected vector
  against live prod data and that promtool agrees about its behaviour over synthetic series.
  **Merging is what deploys it.**
 - **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
  any useful sense** — it describes a standing condition that has been true since the sidecars
  were deployed, so it would have been firing before, during and after. That is deliberate and it
  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
  instead of discovering it during the investigation. **No alert on this platform could have
  detected the incident itself, and none can today.**
 - **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
 - **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
  directory from the one the sidecar writes), and the rendered config contains **no
  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
  caught this incident, and is the single highest-value follow-up here — **would evaluate and
  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
  inert rule ConfigMap to look productive.**
 - **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
 - **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
  snapshot; anything older than three days is already gone. This is stated in the runbook because
  it bounds what an operator can still investigate.
 - **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
  requirement touched by this branch is certified. See the exit code below — **rc 3 is
  UNDETERMINED, not green.**
 ### Follow-ups, in order of value
 1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
   already being collected. The rule is one expression:
   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
 2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
   and it unlocks the ratio rule in Prometheus.
 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
   observable, and it belongs with whoever owns the sidecar args.
@@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 **The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
 `oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
 each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
 listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
 their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
 written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
 This matters because the gate's characteristic failure is invisible by construction: an
 unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
 against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
 where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
 spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
 into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
 `auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
 `OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
 `up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
 a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
 one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
 and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 **To close the gap for an app**, three things are needed together, on that app's own manifest:
 `--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
@@ -0,0 +1,117 @@
 ---
 # promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
 #
 # WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
 # failure modes of that shape are both silent: a label that does not survive `count by` makes the
 # right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
 # and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
 # visible by reading the expression, and neither is visible on a live cluster where the answer
 # happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
 # stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
 #
 # HOW TO RUN. The one repo-wide harness, by the companion convention:
 #   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
 # promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
 #
 # THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
 # copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
 # its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
 # ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
 # that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
 # pass while the real expression collapsed differently.
 #
 # ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
 # comparing it verbatim would couple this file to every wording change while proving nothing the
 # ALERTS series does not already prove.
 rule_files:
  - auth-visibility-alerts.rules.yaml
 evaluation_interval: 1m
 tests:
  # ---------------------------------------------------------------------------------------------
  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
  - interval: 1m
    name: an unscraped gate namespace fires after 1h; the scraped one never does
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The shared instance: kube-state-metrics sees both replicas...
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
        values: '1x130'
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
        values: '1x130'
      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
        values: '1x130'
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
        values: '1x130'
    promql_expr_test:
      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
      # the hold away and turning every rollout into a page.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 59m
        exp_samples: []
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
        eval_time: 59m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
            value: 1
      # Firing for jackett past the hold...
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # ...and never for the namespace that IS scraped. This is the assertion that catches an
      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
      # every gate namespace would pass the test above.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
        eval_time: 65m
        exp_samples: []
      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
      # that lost its `by` would collapse to a single unlabelled series.
      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
        eval_time: 65m
        exp_samples:
          - labels: '{namespace="jackett"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
  - interval: 1m
    name: the alert clears once the sidecar is actually scraped
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
      # exist yet.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
        values: '_x70 1x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
        eval_time: 80m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
  #    rule would page for every app on the platform the day someone widened the matcher.
  - interval: 1m
    name: namespaces without a gate sidecar are never reported
    input_series:
      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
        values: '1x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
        eval_time: 65m
        exp_samples: []
@@ -0,0 +1,113 @@
 ---
 # oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
 #
 # ============================================================================================
 # WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
 # ============================================================================================
 # Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
 # log in. The sidecar's own access log had the whole story:
 #
 #   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 #   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
 #   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
 #   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
 #   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
 #
 # NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
 # Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
 # without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
 # JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
 # is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
 # kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
 # gate had rejected every request ever made.
 #
 # ============================================================================================
 # WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
 # ============================================================================================
 # The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
 # user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
 # reason is this file's real subject:
 #
 #   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
 #
 #   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
 #     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
 #
 #   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
 #   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
 #   is never started; the container declares only port `http:4180`, there is no `metrics` port on
 #   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
 #   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
 #   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
 #   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
 #   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
 #   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
 #   temptation was refused here rather than ratcheted into a register.
 #
 #   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
 #   path for any sidecar-gated app — each app gates itself — and its current pods export no
 #   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
 #   the shared instance would not have fired on this incident either. Saying so is the point.
 #
 # So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
 # That condition is measurable today, from two already-scraped metric families, and it is the
 # precondition for ever being able to alert on the incident itself. It self-clears the moment a
 # sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
 # writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
 # by-hand equivalent an operator uses in the meantime.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: oauth2-proxy-auth-visibility
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
    app.kubernetes.io/name: oauth2-proxy
 spec:
  groups:
    - name: oauth2-proxy.auth-visibility
      rules:
        - alert: OAuth2ProxyGateUnobserved
          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
          # target for that container in that namespace.
          #
          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
          #     container exports anything, which is exactly the property needed here.
          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
          #     without knowing what job name it will pick.
          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
          # inert for the reason it is reporting.
          #
          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
          # the state the cluster is in; it is not a rule waiting for a hypothetical.
          expr: |
            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
              unless
            count by (namespace) (up{container="oauth2-proxy"})
          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
          # discovered yet, and this condition is a standing architectural gap, never an
          # emergency. Long enough that no deploy can produce it.
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
            description: >-
              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
              Prometheus has NO scrape target for it, so every authentication decision that gate
              makes — every redirect to Keycloak, every 403 on a group check, every session that
              silently fails to hold — is invisible to metrics and to alerting. This is not a
              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
              seconds later, and the only record anywhere was a line in the sidecar's stdout.
              No alert could have fired, because there is nothing to alert on. Until a scrape
              exists the runbook's LogQL queries are the substitute; the fix is
              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
              ServiceMonitor, after which the auth-churn rule becomes writable.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,7 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
  # tier, which is Ready before this tier's platform-config dependency completes.
  - auth-visibility-alerts.yaml
@@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
 | per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
 The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
 there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
 exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
 forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
 `--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
 and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
 namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
 (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
 `kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
 real scrape. See `oauth2-proxy-auth-failures`.
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
@@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
  rather than a 401, the browser follows it silently against a live SSO session, and the call
  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
  so these logs (72h of them) are the only record this class of failure leaves.
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -115,14 +135,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
  app never saw). Covers the cookie collision between per-app sidecars sharing the default
  `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
  *other* namespaces and is only legible with two logs side by side — the 5-minute access-token
  session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
  POST body silently.
 - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
@@ -85,11 +85,10 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
-      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
+      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
-      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
+      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
-      # presents as random session loss in ONE app's log and is only legible with both side by side,
+      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
-      # so the page has to be reachable at 3am without going looking for it in git.
+      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      - oauth2-proxy-session=oauth2-proxy-session.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
      # because the plex.tv claim window is short and is not a thing to go looking
@@ -0,0 +1,171 @@
 # oauth2-proxy auth failures — telling the three layers apart
 Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
 does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
 stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
 page is the diagnostic procedure, and it works whether or not the alert ever fired.
 ## What this means here
 Every app behind Keycloak on this platform has **three** places a request can be refused, and
 from a browser they look identical — because two of the three answer with **HTTP 200**.
 | Layer | What refuses | What the browser sees |
 |---|---|---|
 | 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
 | 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
 | 3. The app itself | the app's own login / API key | the app's own 401 or login page |
 **Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
 redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
 JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
 Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
 dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
 nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
 failed.
 The worked example, jackett on 2026-09-30, straight from the sidecar log:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
 signature of the session being **re-established over and over instead of held** — a cookie that
 another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
 down and not of the app being broken.
 **Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
 run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
 `metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
 `oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
 apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
 convenience — until that changes they are the **only** record that this class of failure happened.
 ## First commands
 `kubectl` does not exist on the workstation; everything runs in the pinned container, and always
 with an explicit context.
 **1. Which namespaces have an unobserved gate** (this is what the alert measures):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
 ```
 **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
 and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
 `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
 ```
 Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
 **3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
 you can ask about the whole fleet and about a window that has already scrolled out of
 `kubectl logs`. Port-forward Loki and query it:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
 ```
 and the denominator, which is what makes the number mean something:
 ```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
 ```
 **How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
 lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
 ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
 | namespace | `No valid authentication` | `AuthSuccess` | ratio |
 |---|---|---|---|
 | crafty | 81 | 1 | 81:1 |
 | radarr | 67 | 1 | 67:1 |
 | jackett | 39 | 4 | ~10:1 |
 | pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
 All three of those are the failure, not the baseline.
 **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
 and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
 page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
 shape, and `runbooks/crafty-admin-credential.md` covers it.
 **5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
 **same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
 other's cookie and the loser logs exactly the `No valid authentication` line above. List what
 each gate is configured to emit (arguments only — never print a Secret's data):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
 ```
 A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
 share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
 evicting each other, and the symptom is intermittent and depends on which app you visited last.
 ## Likely causes, most likely first
 1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
 2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
 3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
 4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
 5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
 6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
 ## When to escalate or stop
 - **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
 - **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
 - **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
 - **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.
@@ -1,195 +0,0 @@
 # oauth2-proxy-session — telling the auth layers apart from the sidecar log
 **Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
 (`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
 proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
 signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
 > The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
 > OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
 ## What this means here
 Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
 stacked on it, and the symptom rarely names the one that refused:
 1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
   form, or an error page from Keycloak's own hostname.
 2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
   this page is about reading it.
 3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
 The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
 Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
 `dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
 already changed once. Do not run two `dev.sh` invocations at the same time.
 ### The four lines, and what each one means
 | Log line | What actually happened | Which layer |
 |---|---|---|
 | `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
 | `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
 | `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
 | an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
 That last column is the one people miss. The upstream field sits between the method and the quoted
 path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
 it was proxied to the app. Count them:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
 If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
 log had `-`.
 ## First commands
 **1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
 go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
 **2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
 If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
 is not refreshing it:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
 The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
 spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
 **3. Is this a cookie collision?** Two checks, and you want both.
 *Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
 `.home.keithjackson.dev` domain is in the collision set with every other one:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
      | grep -E "cookie-name|cookie-domain"'
 An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
 flag being absent, not wrong.
 *The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
 line, and put the timestamps next to each other:
    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
 **4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
 instances that genuinely cannot read each other. Compare **hashes**, never values:
    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
 Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
 before.
 ### Do not rank apps by their failure ratio
 Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
 not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
 manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
 scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
 2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
 paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
 (43) — polling, not 156 user-visible failures.
 **Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
 that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
 deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
 start — so read it there:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
 then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
 to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
 ## Likely causes, most likely first
 1. **Cookie collision between two per-app proxies.** Several instances emit the default
   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
   scope — cross-host auth is its job.
 2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
   the proxy never spends the refresh token it holds, so the session expires with the access token.
   Set it to about half the measured token lifetime.
 3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
 4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
 5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
   by the app's own `/login` page is layer 3, not layer 2.
 ## When to escalate or stop
 - **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
  agent action.
 - **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
  erased by the next render anyway. Change the renderer, then regenerate:
      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
  the same commit. There is no way to change one app's sidecar arguments alone. Measured
  2026-09-30. Coordinate before starting — this is a single change, not five.
 - **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
 - **You cannot currently alert on any of this**, and that is worth knowing before you promise
  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
  log-reading only.
 - **Validate an argument change before you open the PR**, not after. The pinned image will tell
  you what it understood, without touching the cluster:
      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
  Running it with your full proposed argument list prints a `Cookie settings:` line and an
  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
  on a service you have just told someone is fixed.
 - **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
  that app and does nothing about the collision, which is caused by the *names* matching, not the
  secrets differing.
 - **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
  cookie flag will help.
 - **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
  has a unique `--cookie-name`. That combination should not occur; it means something is writing
  that name which is not in this inventory. Record what you measured.