feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy.*"}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 10:55:19 -05:00
14 changed files with 644 additions and 834 deletions
@@ -1,283 +0,0 @@
 ---
 date: 2026-09-30
 branch: feat/node-health-render-device
 title: The Intel render device is now self-re-measuring, not a fact someone checked once
 ---
 ## What changed
 ### Why this probe has value at all
 **Nothing else in the platform would notice the device going missing.** Plex exposes no
 ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
 manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
 *without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
 rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
 This check is the only thing in the system that would surface it, which is the actual
 justification for the role carrying it.
 ### The change
 `roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
 was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
 Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
 is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
 The failure this exists to catch is silent by construction. Plex hardware-transcodes through
 `/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
 the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
 back to software transcoding on its own and the only symptom is a hot CPU.
 Four files:
 - `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
  existing tool probe; both are optional and neither is depended on.
 - `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
  (INFO/SKIP).
 - `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
  check makes is data rather than task logic.
 - `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
  `roles/node_health/README.md` gained a section explaining the design and two new "known
  limits".
 Three design rulings worth stating, because each is the kind of thing a later edit would get
 wrong:
 1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
   the device and joins whichever group owns it, so drift genuinely is not a fault. The
   observed group name and number appear in the detail line, which makes drift *visible*
   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
 2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
   the bus and there is still no render node" is a **FAIL**.
 3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
   inside its container. Grading either would put a permanent WARN on a chosen configuration,
   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
   installs nothing.
 `xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
 drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
 bound to nothing.
 ## What I verified
 ### The real prod run (read-only, all three nodes)
 `--list-hosts` first, every time. It named exactly three hosts and nothing else:
    play #1 (k3s_prod): Inspect node hardware health (read-only)
        pattern: ['k3s_prod']
        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1
 Then `--syntax-check` (rc 0), then the run:
    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
    PLAY rc=0
 (The `changed=` counts are the *controller-side* artifact writes under
 `infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
 the collector task is `changed_when: false` and the role touches nothing.)
 All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
 (The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
 mine; none of them moved.)
 The new report section, **byte-identical on all three nodes**:
    --------------------------------------------------------------------------------
     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
    --------------------------------------------------------------------------------
     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
                  driver i915
     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
     dev  card0        character special  mode 0660  root:video gid 44
     dev  renderD128   character special  mode 0660  root:render gid 992
     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
                  readable without root (i915 module parameters are mode 0400)
                 no i915/xe parameter on the kernel cmdline
     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
 That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
 character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
 `0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
 CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
 **One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
 prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
 absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
 `node_baseline_packages` and depending on an undeclared package would be wrong — but the
 comments and README that asserted it was missing were factually wrong and are now corrected.
 Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
 ### The gate
    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
    GATE rc=3
 **rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
 three stages did not run, all three the known local ceiling from a worktree, and **no fourth
 skip**:
    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass
 Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
 (the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
 the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
 know the shim took effect rather than the git stages silently degrading.
 `ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
 `ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
 claiming to have matched a lint convention that does not exist.
 ### The CI gate on the PR — a green tick that is exit 3, read from the log
 The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
 on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
 "Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.
 **But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
 `run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
 log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
 Gitea, the web route serves it) and read the gate's own headline:
    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
    [fail]   Stages that did not run: spec-immutability
 **So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
 check is the tolerance swallowing that 3, not a rc 0.
 **CI verified two things my local run could not, and both came back green:**
    [ ok ] stage gitea-settings: pass
    [ ok ]   enable_push = false          enable_status_check = true
    [ ok ]   required_approvals = 1       block_admin_merge_override = true
    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
    [ ok ]   runners online: 1 (>= 1)
    [ ok ] Gitea settings match the declaration.
    [ ok ] stage approval-policy: pass
    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.
 My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
 `approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
 there would have failed the job. That is measured confirmation that the live Gitea matches
 `infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
 reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
 (`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.
 Corroborated independently: `GET /repos/.../branches/main` →
 `protected=true user_can_push=false enable_status_check=true required_approvals=1`,
 contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
 status=online`. And note the commit's **combined** state is `pending`, not success, because
 `gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
 not in the required list, so it does not block, but the PR is not "all green" either.
 The gate's `lint` stage passed with one `line too long` **warning** on
 `roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
 above my insertion point, and warnings do not fail the stage.
 ### Offline work (before the lock was released)
 - `python3 -m py_compile` on the collector: clean.
 - `collect_graphics()` executed for real against this workstation's hardware (an unrelated
  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
  structure with no exception, correctly reporting two cards, two render nodes, the bound
  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
  without root* rather than as a fabricated `0`.
 - That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
  earlier draft would have silently reported a null.
 - Two other bugs found and fixed by running it rather than reasoning about it:
  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
  excluded from the card scan.
 - All three checks and the new template section were rendered through a standalone Jinja
  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
  workstation; GPU present with the render node removed; no display device at all; `xe` bound
  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
  Ansible's `trim_blocks=True`.
  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
  dictionaries, not machines. They are not hardware tests and must not be written up as if
  they were.
 ## What I did NOT verify
 - **The FAIL path has never fired on a real node.** Every prod node has a working render
  device, and this role is read-only, so a device cannot be taken away to prove the check
  catches its absence. The failure branches were exercised against **synthetic fact sets
  only** (see above). I am not claiming the failure path is tested against hardware.
 - **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
 - **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
  is measured here. I did **not** confirm against kernel source that no other kernel version or
  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
  line, not a false verdict, which is why this failure mode is tolerable.
 - **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
 - **Not a transcode test.** These checks prove the render *device* exists, is a character
  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
  node-level check that installs nothing.
 - **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
 - **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
 - **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
  It is the expected, tolerated skip — but it is still a skip, not a pass.
 - **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
  passed — so those two I assert on measurement, not on this file's prose.
 - **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
  exists. That is real, it is out of this change's scope, and it is unclaimed work.
@@ -0,0 +1,175 @@
 ---
 date: 2026-09-30
 branch: feat/oauth2-proxy-auth-failure-visible
 title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
 ---
 ## What changed
 Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
 operator's report named the property that matters — *"reloading the page never fixes the issue and
 the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
 about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
 the alert rules. It is an absence of data.
 **The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
 That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
 jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
 `--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
 starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
 cluster selects them.
 So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
 a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
 exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
 permanently silent, and indistinguishable from health. It was not written.
 What was written instead alerts on the condition that made the incident invisible, from families
 that **are** scraped:
 - **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
        unless
      count by (namespace) (up{container="oauth2-proxy"})
  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
  clears per-namespace as each gate gains a real scrape.
 - **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
  prod, not invented.
 - **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
  `configMapGenerator` and listed in `runbooks/README.md`.
 - **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
  per-app sidecars, plus the measured reason.
 - **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
  three-part change that would close the gap.
 **Why the obvious alternative was rejected.** An alert on the *shared* proxy's
 `oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
 need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
 and — decisively — **it would not have fired on this incident**: the shared instance is not in the
 data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
 at all, having proxied nothing since they started (the counter exists in the TSDB only from an
 older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
 Shipping it would have looked like coverage of the reported failure and been none.
 ## What I verified
 Everything below was measured against live prod through the pinned container, read-only. No
 `apply`, no `edit`, no `patch`, no `flux reconcile`.
 - **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
  `1`, `namespace="oauth2-proxy"`.
 - **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
  `oauth2-proxy` container, with its declared ports: the two shared replicas have
  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
 - **No existing alert is inert *for this reason*, because no alert references these metrics at
  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
 - **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
  observable from the IdP side.
 - **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
 - **The alert expression fires against live data.** Run verbatim on prod:
  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
  radarr`, value 1 each; `oauth2-proxy` correctly absent.
 - **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
  runbook as the thing to compare against.
 - **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
 - **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
  `runbook_url` points at exactly that name.
 - **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
  per-stage outcomes in the next section — read them there, not here.
 ## What I did NOT verify
 - **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
  only in git on this branch. What is proven is that its expression returns the expected vector
  against live prod data and that promtool agrees about its behaviour over synthetic series.
  **Merging is what deploys it.**
 - **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
  any useful sense** — it describes a standing condition that has been true since the sidecars
  were deployed, so it would have been firing before, during and after. That is deliberate and it
  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
  instead of discovering it during the investigation. **No alert on this platform could have
  detected the incident itself, and none can today.**
 - **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
 - **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
  directory from the one the sidecar writes), and the rendered config contains **no
  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
  caught this incident, and is the single highest-value follow-up here — **would evaluate and
  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
  inert rule ConfigMap to look productive.**
 - **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
 - **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
  snapshot; anything older than three days is already gone. This is stated in the runbook because
  it bounds what an operator can still investigate.
 - **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
  requirement touched by this branch is certified. See the exit code below — **rc 3 is
  UNDETERMINED, not green.**
 ### Follow-ups, in order of value
 1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
   already being collected. The rule is one expression:
   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
 2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
   and it unlocks the ratio rule in Prometheus.
 3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
   observable, and it belongs with whoever owns the sidecar args.
@@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 ## Contract
- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`
+- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
-  (and `lspci` only if it happens to be installed), and — only when it can escalate —
+  and — only when it can escalate — `smartctl`. It writes nothing to the node and
-  `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
+  **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
  absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
  package installation is `node_baseline`'s job, never a health check's, never a manual
  `apt-get`).
@@ -50,9 +50,6 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
 | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) |
 | `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) |
 | `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
@@ -94,78 +91,6 @@ used to blame that state on "this node has not been baselined yet", which sent t
 back to re-run the role that had already run.
 ## The render-device check
 Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing,
 nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom
 is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard
 IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly
 that silent degradation.
 The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A
 fact measured once decays silently**, so it is re-measured on every health run instead of
 living in a runbook. Nothing is installed, no module is loaded, no group is changed — the
 role's read-only contract is unchanged.
 | fact | where it comes from | root? |
 |---|---|---|
 | `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no |
 | owning group **name** for the render node's gid | `grp.getgrgid()` | no |
 | bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no |
 | is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no |
 | media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no |
 | module load state | `/proc/modules` | no |
 | GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** |
 ### The PCI scan is separate from the DRM scan on purpose
 `/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on
 the bus with nothing attached has no `card0` at all — and that is precisely the failure being
 hunted. So the display device is looked for on the PCI bus independently, from sysfs, which
 is always there. That is what lets `graphics.render_device` distinguish
 - *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention
  `thermal.max` uses for "no sensor on this hardware"), from
 - *a GPU is on the bus and there is still no render node* → **FAIL**.
 `lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as
 optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing
 in this repo guarantees it. It did turn out to be present on all three prod nodes
 (`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending
 on that would mean depending on an undeclared package. Absent, `description` is `null`, the
 numeric IDs are printed instead, and no verdict changes.
 ### The render node's GID is reported, never asserted
 The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on
 2026-09-30. The check does **not** compare against `992`, and must not be changed to.
 `render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid
 from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own
 container entrypoint stats the device and joins whichever group owns it, so drift genuinely
 is not a fault. Recording the observed group name and gid in the detail line makes drift
 **visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a
 reinstall.
 What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver`
 (default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a
 **WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace
 media stack and a libva/iHD build that transcodes correctly against one is not automatically
 right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing.
 ### Two things that are informational and must stay that way
 - **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It
  costs low-power H.264 encode *quality* only; decode and encode both still run on the media
  engines. Grading it would put a permanent WARN on a chosen configuration.
 - **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and
  media driver inside its container, so the host needs neither. Flagging it would invite
  someone to `apt install vainfo` on a node to clear a health check — and this role installs
  nothing.
 `enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually
 cannot read it. It reports as an honest `NOT READ — <reason>` rather than a fabricated `0`,
 with `/proc/cmdline` as the world-readable second source for an explicit boot override.
 ## The EFI memattr check
 `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
@@ -348,10 +273,3 @@ stays `null` — no claim is made either way.
  but the wear/endurance context SKIPs rather than guessing.
 - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
  unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
 - `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod
  node has a working render device, and the role is read-only so one cannot be taken away to
  prove it. The failure branches were exercised against synthetic fact sets only. The same
  applies to the `xe`-bound WARN: no node here has `xe` bound.
 - `graphics.*` proves the render **device** exists and which driver owns it. It does **not**
  prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container,
  with Plex's own libva, which is out of scope for a node-level check that installs nothing.
@@ -35,24 +35,6 @@ node_health_try_privileged: true
 # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
 node_health_smartctl_args: "--json=c --all --nocheck=standby"
 # --- Graphics / render device ---------------------------------------------------------
 # The DRM driver this site EXPECTS to be bound to its display device. Measured on all three
 # prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP
 # EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to
 # nothing, and that difference matters enough to surface: the two drivers do not share a
 # userspace media stack (a libva/iHD build that works against i915 is not automatically the
 # right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD
 # or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic.
 #
 # WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning
 # group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static
 # one, so a rebuilt node may legitimately land on a different number — it was 992 on all
 # three prod nodes on 2026-09-30, and an assertion against that literal would be a false
 # alarm waiting to happen. Plex's own container entrypoint stats the device and joins
 # whichever group owns it, so drift genuinely is not a fault. The observed group name and GID
 # are therefore REPORTED in the check detail and never graded.
 node_health_graphics_expected_driver: i915
 # --- Thresholds ---------------------------------------------------------------------
 # All verdicts come from here, so a site can retune without touching task logic.
 node_health_thresholds:
@@ -35,13 +35,10 @@
 # =============================================================================
 import glob
 import grp
 import json
 import os
 import pwd
 import re
 import shutil
 import stat
 import subprocess
 import sys
 import time
@@ -759,212 +756,6 @@ def collect_pressure():
    return out
 # --- Graphics / DRM render device -----------------------------------------------------
 # WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that
 # node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that
 # binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software
 # transcoding and the only symptom is a hot CPU. So the render device is measured on every
 # health run rather than being a fact someone established once by hand.
 #
 # EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are
 # stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so
 # are the module parameters under /sys/module. Nothing here needs root, so nothing here
 # belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need
 # root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is
 # what actually decides whether the firmware is loaded at all.)
 #
 # THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists
 # once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no
 # card0 at all, and that is precisely the failure this is meant to catch — so the display
 # device is looked for on the PCI bus independently, from sysfs, which is always there.
 # `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as
 # optional because it IS optional: pciutils is not declared in node_baseline_packages, so
 # nothing in this repo guarantees it. It did turn out to be present on all three prod nodes
 # (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on
 # that would make the collector depend on an undeclared package. A missing lspci leaves
 # `description` null, the numeric IDs are reported instead, and no verdict changes.
 DISPLAY_PCI_CLASS_PREFIX = "0x03"  # PCI base class 03 = display controller (VGA/3D/other)
 # DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative
 # (a built-in driver never appears in /proc/modules); this is corroboration only, and it is
 # how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact.
 DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia")
 def _uid_name(uid):
    try:
        return pwd.getpwuid(uid).pw_name
    except (KeyError, OverflowError, TypeError, OSError):
        return None
 def _gid_name(gid):
    """Group NAME for a numeric gid, or None when nothing resolves it.
    Reported as an OBSERVED value, never asserted against a literal. The `render` group is
    a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's
    static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own
    container init stats the device and joins whatever group owns it, so a moved gid is
    drift to notice, not a fault to fail on."""
    try:
        return grp.getgrgid(gid).gr_name
    except (KeyError, OverflowError, TypeError, OSError):
        return None
 def _module_param(mod, param):
    """One /sys/module/<mod>/parameters/<param>, with WHY it is missing when it is.
    Three outcomes that must not be conflated: a real value; "the module exposes no such
    parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no
    enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915
    module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an
    unprivileged pass legitimately cannot read them. Returning null with the reason keeps
    that an honest UNKNOWN instead of a fabricated 0. read_text() also records the
    permission denial in UNREADABLE, so the summary lists it with everything else root
    would add."""
    path = "/sys/module/%s/parameters/%s" % (mod, param)
    if not os.path.exists(path):
        return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, "
                                         "built in, or no such knob on this driver)" % (mod, param)}
    value = read_text(path)
    if value is None:
        return {"value": None, "reason": "%s is not readable without root (i915 module "
                                         "parameters are mode 0400)" % path}
    return {"value": value, "reason": None}
 def _dev_node(path):
    """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is."""
    try:
        st = os.lstat(path)
    except OSError as exc:
        UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc})
        return None
    return {
        "path": path,
        "name": os.path.basename(path),
        "char_device": stat.S_ISCHR(st.st_mode),
        "mode": "%04o" % stat.S_IMODE(st.st_mode),
        "uid": st.st_uid,
        "gid": st.st_gid,
        "owner": _uid_name(st.st_uid),
        "group": _gid_name(st.st_gid),
    }
 def collect_graphics():
    dri_dir = "/dev/dri"
    dri_present = os.path.isdir(dri_dir)
    devices = []
    if dri_present:
        for path in sorted(glob.glob(dri_dir + "/*")):
            # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping
            # directories rather than filtering to char devices is deliberate: anything
            # in here that is NOT a character special file is an anomaly worth seeing, and
            # `char_device: false` is how it becomes visible.
            if os.path.isdir(path):
                continue
            entry = _dev_node(path)
            if entry is not None:
                devices.append(entry)
    render_nodes = [d for d in devices if d["name"].startswith("renderD")]
    # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR
    # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped.
    cards = []
    for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")):
        name = os.path.basename(cdir)
        if "-" in name:
            continue
        driver = None
        drvlink = cdir + "/device/driver"
        if os.path.islink(drvlink):
            driver = os.path.basename(os.path.realpath(drvlink))
        address = None
        devlink = cdir + "/device"
        if os.path.islink(devlink):
            address = os.path.basename(os.path.realpath(devlink))
        # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC)
        # engine and vecs* the video-enhancement engine: those two are what a transcode
        # actually runs on, so their presence is the difference between "a GPU is bound"
        # and "this GPU can do media work".
        engines = sorted(
            {
                os.path.basename(p)
                for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*"))
                if os.path.isdir(p)
            }
        )
        cards.append(
            {
                "card": name,
                "driver": driver,
                "pci_address": address,
                "vendor_id": read_text(cdir + "/device/vendor"),
                "device_id": read_text(cdir + "/device/device"),
                "engines": engines,
                "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))],
            }
        )
    # Display-class devices on the PCI bus, driver bound or not.
    pci_display = []
    for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")):
        class_id = read_text(pdir + "/class") or ""
        if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX):
            continue
        address = os.path.basename(pdir)
        driver = None
        if os.path.islink(pdir + "/driver"):
            driver = os.path.basename(os.path.realpath(pdir + "/driver"))
        description = None
        rc, out, _ = run(["lspci", "-nn", "-s", address])
        if rc == 0 and out.strip():
            description = out.strip().splitlines()[0]
        pci_display.append(
            {
                "address": address,
                "class_id": class_id or None,
                "vendor_id": read_text(pdir + "/vendor"),
                "device_id": read_text(pdir + "/device"),
                "revision": read_text(pdir + "/revision"),
                "driver": driver,
                "description": description,
            }
        )
    loaded = set()
    for line in (read_text("/proc/modules") or "").splitlines():
        fields = line.split()
        if fields:
            loaded.add(fields[0])
    modules = {m: (m in loaded) for m in DRM_MODULES}
    # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC
    # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the
    # one place the state is visible without root AND without debugfs — but the sysfs file
    # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel
    # command line is the world-readable second source: an explicit `i915.enable_guc=` there
    # is authoritative about what was ASKED for, even when the parameter file cannot be read.
    uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")}
    cmdline_params = [
        token
        for token in (read_text("/proc/cmdline") or "").split()
        if token.startswith(tuple("%s." % m for m in DRM_MODULES))
    ]
    return {
        "dri_present": dri_present,
        "devices": devices,
        "render_nodes": render_nodes,
        "drm_cards": cards,
        "pci_display": pci_display,
        "modules": modules,
        "uc": uc,
        "cmdline_params": cmdline_params,
    }
 # --- Storage ------------------------------------------------------------------------
 PSEUDO_FS = {
    "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore",
@@ -1304,15 +1095,7 @@ def collect_os():
 # we only DETECT, and the role skips the SMART checks with an accurate reason.
 def collect_tools():
    tools = {}
-    # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in
+    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
    # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod
    # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and
    # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo
    # is not needed on the host at all, because Plex ships its own libva and media driver
    # inside its container. They are probed so the report can state their presence or absence
    # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted
    # to install them. This role installs nothing.
    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"):
        tools[tool] = which(tool)
    return tools
@@ -1344,7 +1127,6 @@ def main():
        "os": collect_os(),
        "dmi": collect_dmi(),
        "cpu": collect_cpu(),
        "graphics": collect_graphics(),
        "memory": collect_memory(),
        "thermal": collect_thermal(),
        "pressure": collect_pressure(),
@@ -652,204 +652,6 @@
         ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
         ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
 # --- Graphics / render device ----------------------------------------------------------
 # WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
 # /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
 # to software transcoding on its own and the only symptom is a hot CPU and stuttering
 # clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
 # binds a different DRM driver all produce exactly that silent degradation. It was measured
 # by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
 # decays silently, so it is re-measured on every health run instead.
 #
 # READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
 # unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
 # /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
 # nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
 # README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
 # and it reports as an honest UNKNOWN with the reason rather than as a 0.
 #
 # THE THREE VERDICTS ARE SPLIT ON PURPOSE:
 #   graphics.render_device  the OUTCOME — can anything hardware-transcode on this node.
 #   graphics.drm_driver     the CAUSE side — what is bound, and is it what this site expects.
 #   graphics.media_stack    INFO only. GuC/HuC and host VAAPI userspace both belong here and
 #                           NEITHER is ever a failure; see the task's own comment for why.
 - name: Assess — locate the display device and its bound driver (once, shared by three checks)
  ansible.builtin.set_fact:
    node_health_render:
      # Display-class PCI devices (base class 0x03), driver bound or not. This is the
      # question "is there a GPU in this machine at all", and it is answered from the PCI
      # bus rather than from /sys/class/drm because a card* entry only exists once a driver
      # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
      # identical to a machine with no GPU.
      display_devices: "{{ g.pci_display }}"
      bound_drivers: >-
        {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
      # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
      # is an anomaly, not a device, and must not be counted as one.
      render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
      # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
      # difference between "a GPU is bound" and "this GPU can do media work".
      media_engines: >-
        {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
  vars:
    ansible_become: false
    g: "{{ node_health_facts.graphics }}"
 - name: Assess — DRM render node for hardware transcoding
  ansible.builtin.set_fact:
    node_health_checks: "{{ node_health_checks + [{
      'id': 'graphics.render_device',
      'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
      'status': render_status,
      'detail': render_detail }] }}"
  vars:
    ansible_become: false
    g: "{{ node_health_facts.graphics }}"
    r: "{{ node_health_render }}"
    # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
    # then nothing to grade, and the same convention thermal.max already uses for "no sensor
    # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
    # a GPU IS on the bus and there is still no render node.
    render_status: >-
      {{ 'SKIP' if (r.display_devices | length) == 0
         else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
    dri_listing: >-
      {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
         else 'nothing' }}
    render_detail: >-
      {%- if (r.display_devices | length) == 0 -%}
      this machine exposes no display-class PCI device (base class 0x03) at all, so there is
      no GPU here for a render node to belong to and hardware transcoding is not available on
      this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
      not assumed.
      {%- elif (r.render_chardevs | length) > 0 -%}
      {{ r.render_chardevs | length }} render node(s):
      {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
      owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
      gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
      /dev/dri holds: {{ dri_listing }}.
      THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
      `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
      node may legitimately land on a different number (all three prod nodes read gid 992 on
      2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
      owns it. Drift is therefore visible in this line without being a failure.
      {%- else -%}
      HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
      PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
      no usable character-special render node: /dev/dri
      {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
      {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
      ~ ' — present, but NOT character special, which is an anomaly in its own right. ')
         if (g.render_nodes | length) > 0 else '' }}
      Nothing fails loudly in this state — Plex silently falls back to software transcoding
      and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
      driver bound at all, then whether the onboard IGD is still enabled in BIOS.
      {%- endif -%}
 - name: Assess — DRM driver bound to the display device
  ansible.builtin.set_fact:
    node_health_checks: "{{ node_health_checks + [{
      'id': 'graphics.drm_driver',
      'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
      'status': driver_status,
      'detail': driver_detail }] }}"
  vars:
    ansible_become: false
    g: "{{ node_health_facts.graphics }}"
    r: "{{ node_health_render }}"
    expected: "{{ node_health_graphics_expected_driver }}"
    # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
    # still works, but the userspace media stack is not the same one, so it is a change the
    # operator must see before a transcode regression surfaces as a user complaint. FAIL is
    # for nothing bound at all, which is the cause of a missing render node.
    driver_status: >-
      {{ 'SKIP' if (r.display_devices | length) == 0
         else ('FAIL' if (r.bound_drivers | length) == 0
               else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
    driver_detail: >-
      {%- if (r.display_devices | length) == 0 -%}
      no display-class PCI device on this machine, so no DRM driver can be bound to one.
      Measured from /sys/bus/pci/devices.
      {%- else -%}
      {% for p in r.display_devices %}{{ p.address }}
      {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
      -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
      {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
           | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
         else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
      Module load state (/proc/modules, corroboration only — a built-in driver never appears
      there, the bound-driver symlink above is what decides):
      {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
         | map('join', '=') | join(', ') }}.
      {%- if (r.bound_drivers | length) == 0 %}
      NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
      subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
      module exists for this kernel and whether it was blacklisted or failed to probe.
      {%- elif expected in r.bound_drivers %}
      This is the expected driver for this site (node_health_graphics_expected_driver).
      {%- else %}
      EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
      works, so this is not an outage — but the two DRM drivers do not share a userspace
      media stack, and a libva/iHD build that transcodes correctly against one is not
      automatically right for the other. Confirm Plex still hardware-transcodes before
      treating this as benign.
      {%- endif %}
      {%- endif -%}
 # INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
 #
 #   GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
 #   costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
 #   through the media engines. Grading it would put a permanent WARN on a configuration the
 #   operator has chosen.
 #   No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
 #   driver inside its container, so the host needs neither. Flagging it would invite someone
 #   to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
 #
 # The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
 # (video-enhancement) are what a transcode actually runs on.
 - name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
  ansible.builtin.set_fact:
    node_health_checks: "{{ node_health_checks + [{
      'id': 'graphics.media_stack',
      'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
      'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
      'detail': media_detail }] }}"
  vars:
    ansible_become: false
    g: "{{ node_health_facts.graphics }}"
    r: "{{ node_health_render }}"
    # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
    # that set the expected driver to amdgpu must get a clean "no such parameters" line
    # instead of an undefined-key traceback.
    guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
    guc_text: >-
      {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
         else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
               if guc is mapping
               else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
    media_detail: >-
      {%- if (r.display_devices | length) == 0 -%}
      no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
      firmware and nothing to transcode with.
      {%- else -%}
      media engines:
      {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
         else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
      (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
      GuC/HuC: {{ guc_text }}.
      {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
         else 'No i915/xe parameter on the kernel cmdline' }}.
      GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
      low-power H.264 encode quality only, and both decode and encode still run on the
      engines above.
      Host VAAPI userspace: vainfo
      {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
         else 'NOT installed on this host' }}
      — also never graded, because Plex ships its own libva and media driver inside its
      container, so the host needs none. This role installs nothing.
      {%- endif -%}
 # --- Thermals -----------------------------------------------------------------------
 - name: Assess — temperatures
  ansible.builtin.set_fact:
@@ -136,37 +136,6 @@
 {% endfor %}
 {% endfor %}
 --------------------------------------------------------------------------------
 GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
 --------------------------------------------------------------------------------
 {% set gfx = node_health_facts.graphics %}
 {% if gfx.pci_display %}
 {% for p in gfx.pci_display %}
 pci  {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }}
              driver {{ p.driver | default('NONE BOUND', true) }}
 {% endfor %}
 {% else %}
 pci         no display-class device (PCI base class 0x03) on this machine — no GPU here
 {% endif %}
 {% for c in gfx.drm_cards %}
 drm  {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }}  engines {{ c.engines | join(' ') | default('none exposed', true) }}
 {% endfor %}
 {% if gfx.dri_present %}
 {% for d in gfx.devices %}
 dev  {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }}  {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }}
 {% endfor %}
 {% else %}
 dev         /dev/dri DOES NOT EXIST — no DRM device node on this node at all
 {% endif %}
 {# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may
   legitimately move it and Plex's container init joins whichever group owns the device. See
   defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #}
 {% set guc = gfx.uc.get(node_health_graphics_expected_driver) %}
 GuC/HuC     {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
             {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }}
 modules     {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides)
 vainfo      {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }}
 --------------------------------------------------------------------------------
 PRESSURE STALL (/proc/pressure — % of wall time stalled)
 --------------------------------------------------------------------------------
@@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2
 **The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
 `oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
 each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
 listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
 their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
 written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
 This matters because the gate's characteristic failure is invisible by construction: an
 unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
 against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
 where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
 spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
 into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
 `auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
 `OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
 `up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
 a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
 one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
 and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
 **To close the gap for an app**, three things are needed together, on that app's own manifest:
 `--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
 Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
 `code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
 session lifetime — become writable rather than decorative.
@@ -0,0 +1,117 @@
 ---
 # promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
 #
 # WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
 # failure modes of that shape are both silent: a label that does not survive `count by` makes the
 # right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
 # and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
 # visible by reading the expression, and neither is visible on a live cluster where the answer
 # happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
 # stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
 #
 # HOW TO RUN. The one repo-wide harness, by the companion convention:
 #   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
 # promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
 #
 # THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
 # copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
 # its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
 # ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
 # that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
 # pass while the real expression collapsed differently.
 #
 # ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
 # comparing it verbatim would couple this file to every wording change while proving nothing the
 # ALERTS series does not already prove.
 rule_files:
  - auth-visibility-alerts.rules.yaml
 evaluation_interval: 1m
 tests:
  # ---------------------------------------------------------------------------------------------
  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
  - interval: 1m
    name: an unscraped gate namespace fires after 1h; the scraped one never does
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # The shared instance: kube-state-metrics sees both replicas...
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
        values: '1x130'
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
        values: '1x130'
      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
        values: '1x130'
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
        values: '1x130'
    promql_expr_test:
      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
      # the hold away and turning every rollout into a page.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 59m
        exp_samples: []
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
        eval_time: 59m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
            value: 1
      # Firing for jackett past the hold...
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      # ...and never for the namespace that IS scraped. This is the assertion that catches an
      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
      # every gate namespace would pass the test above.
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
        eval_time: 65m
        exp_samples: []
      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
      # that lost its `by` would collapse to a single unlabelled series.
      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
        eval_time: 65m
        exp_samples:
          - labels: '{namespace="jackett"}'
            value: 1
  # ---------------------------------------------------------------------------------------------
  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
  - interval: 1m
    name: the alert clears once the sidecar is actually scraped
    input_series:
      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
        values: '1x130'
      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
      # exist yet.
      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
        values: '_x70 1x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
        eval_time: 65m
        exp_samples:
          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
        eval_time: 80m
        exp_samples: []
  # ---------------------------------------------------------------------------------------------
  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
  #    rule would page for every app on the platform the day someone widened the matcher.
  - interval: 1m
    name: namespaces without a gate sidecar are never reported
    input_series:
      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
        values: '1x130'
    promql_expr_test:
      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
        eval_time: 65m
        exp_samples: []
@@ -0,0 +1,113 @@
 ---
 # oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
 #
 # ============================================================================================
 # WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
 # ============================================================================================
 # Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
 # log in. The sidecar's own access log had the whole story:
 #
 #   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 #   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
 #   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
 #   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
 #   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
 #
 # NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
 # Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
 # without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
 # JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
 # is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
 # kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
 # gate had rejected every request ever made.
 #
 # ============================================================================================
 # WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
 # ============================================================================================
 # The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
 # user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
 # reason is this file's real subject:
 #
 #   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
 #
 #   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
 #     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
 #
 #   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
 #   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
 #   is never started; the container declares only port `http:4180`, there is no `metrics` port on
 #   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
 #   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
 #   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
 #   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
 #   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
 #   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
 #   temptation was refused here rather than ratcheted into a register.
 #
 #   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
 #   path for any sidecar-gated app — each app gates itself — and its current pods export no
 #   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
 #   the shared instance would not have fired on this incident either. Saying so is the point.
 #
 # So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
 # That condition is measurable today, from two already-scraped metric families, and it is the
 # precondition for ever being able to alert on the incident itself. It self-clears the moment a
 # sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
 # writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
 # by-hand equivalent an operator uses in the meantime.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: oauth2-proxy-auth-visibility
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
    app.kubernetes.io/name: oauth2-proxy
 spec:
  groups:
    - name: oauth2-proxy.auth-visibility
      rules:
        - alert: OAuth2ProxyGateUnobserved
          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
          # target for that container in that namespace.
          #
          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
          #     container exports anything, which is exactly the property needed here.
          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
          #     without knowing what job name it will pick.
          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
          # inert for the reason it is reporting.
          #
          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
          # the state the cluster is in; it is not a rule waiting for a hypothetical.
          expr: |
            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
              unless
            count by (namespace) (up{container="oauth2-proxy"})
          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
          # discovered yet, and this condition is a standing architectural gap, never an
          # emergency. Long enough that no deploy can produce it.
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
            description: >-
              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
              Prometheus has NO scrape target for it, so every authentication decision that gate
              makes — every redirect to Keycloak, every 403 on a group check, every session that
              silently fails to hold — is invisible to metrics and to alerting. This is not a
              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
              seconds later, and the only record anywhere was a line in the sidecar's stdout.
              No alert could have fired, because there is nothing to alert on. Until a scrape
              exists the runbook's LogQL queries are the substitute; the fix is
              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
              ServiceMonitor, after which the auth-churn rule becomes writable.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,7 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
  # tier, which is Ready before this tier's platform-config dependency completes.
  - auth-visibility-alerts.yaml
@@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
 | per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
 The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
 there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
 exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
 forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
 `--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
 and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
 namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
 (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
 `kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
 real scrape. See `oauth2-proxy-auth-failures`.
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
@@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
  rather than a 401, the browser follows it silently against a live SSO session, and the call
  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
  so these logs (72h of them) are the only record this class of failure leaves.
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -85,6 +85,10 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
      # because the plex.tv claim window is short and is not a thing to go looking
@@ -0,0 +1,171 @@
 # oauth2-proxy auth failures — telling the three layers apart
 Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
 does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
 stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
 page is the diagnostic procedure, and it works whether or not the alert ever fired.
 ## What this means here
 Every app behind Keycloak on this platform has **three** places a request can be refused, and
 from a browser they look identical — because two of the three answer with **HTTP 200**.
 | Layer | What refuses | What the browser sees |
 |---|---|---|
 | 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
 | 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
 | 3. The app itself | the app's own login / API key | the app's own 401 or login page |
 **Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
 redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
 JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
 Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
 dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
 nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
 failed.
 The worked example, jackett on 2026-09-30, straight from the sidecar log:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
 A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
 signature of the session being **re-established over and over instead of held** — a cookie that
 another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
 down and not of the app being broken.
 **Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
 run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
 `metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
 `oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
 apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
 convenience — until that changes they are the **only** record that this class of failure happened.
 ## First commands
 `kubectl` does not exist on the workstation; everything runs in the pinned container, and always
 with an explicit context.
 **1. Which namespaces have an unobserved gate** (this is what the alert measures):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
 ```
 **2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
 and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
 `kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
 ```
 Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
 **3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
 you can ask about the whole fleet and about a window that has already scrolled out of
 `kubectl logs`. Port-forward Loki and query it:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
 ```
 and the denominator, which is what makes the number mean something:
 ```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
 ```
 **How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
 lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
 ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
 | namespace | `No valid authentication` | `AuthSuccess` | ratio |
 |---|---|---|---|
 | crafty | 81 | 1 | 81:1 |
 | radarr | 67 | 1 | 67:1 |
 | jackett | 39 | 4 | ~10:1 |
 | pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
 All three of those are the failure, not the baseline.
 **4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
 and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
 page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
 shape, and `runbooks/crafty-admin-credential.md` covers it.
 **5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
 **same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
 other's cookie and the loser logs exactly the `No valid authentication` line above. List what
 each gate is configured to emit (arguments only — never print a Secret's data):
 ```
 infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
 ```
 A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
 share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
 evicting each other, and the symptom is intermittent and depends on which app you visited last.
 ## Likely causes, most likely first
 1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
 2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
 3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
 4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
 5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
 6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
 ## When to escalate or stop
 - **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
 - **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
 - **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
 - **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.