docs(decisions): record what the CI gate actually reported, not the tick

The PR's required check went green as "Successful in 1m40s" — a real run, not "Has been skipped". But .gitea/workflows/gate.yml:138 is `./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot distinguish them. Fetched the job log and read the gate's own headline: MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability). So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that plainly rather than leaving "CI green" to be read as rc 0. Also corrects a claim that had gone stale within the same entry: CI ran two stages my worktree run could not, and both passed — gitea-settings (the live Gitea matches infra/gitea/expected-branch-protection.json; runners online 1) and approval-policy with real PR context (no sensitive paths touched, so infra/ansible/** needs no human reviewer). My local run skipped both, and the handoff said branch protection was UNVERIFIED. It is now measured, so saying otherwise would have left a known-false line in the log. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(node-health): probe the Intel render device, read-only
2026-09-30 16:12:56 -05:00 · 2026-09-30 13:04:43 -05:00
14 changed files with 834 additions and 644 deletions
@@ -0,0 +1,283 @@
+---
+date: 2026-09-30
+branch: feat/node-health-render-device
+title: The Intel render device is now self-re-measuring, not a fact someone checked once
+---
+
+## What changed
+
+### Why this probe has value at all
+
+**Nothing else in the platform would notice the device going missing.** Plex exposes no
+ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
+manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
+*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
+rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
+This check is the only thing in the system that would surface it, which is the actual
+justification for the role carrying it.
+
+### The change
+
+`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
+was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
+Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
+is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
+
+The failure this exists to catch is silent by construction. Plex hardware-transcodes through
+`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
+the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
+back to software transcoding on its own and the only symptom is a hot CPU.
+
+Four files:
+
+- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
+  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
+  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
+  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
+  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
+  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
+  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
+  existing tool probe; both are optional and neither is depended on.
+- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
+  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
+  (INFO/SKIP).
+- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
+  check makes is data rather than task logic.
+- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
+  `roles/node_health/README.md` gained a section explaining the design and two new "known
+  limits".
+
+Three design rulings worth stating, because each is the kind of thing a later edit would get
+wrong:
+
+1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
+   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
+   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
+   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
+   the device and joins whichever group owns it, so drift genuinely is not a fault. The
+   observed group name and number appear in the detail line, which makes drift *visible*
+   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
+2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
+   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
+   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
+   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
+   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
+   the bus and there is still no render node" is a **FAIL**.
+3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
+   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
+   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
+   inside its container. Grading either would put a permanent WARN on a chosen configuration,
+   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
+   installs nothing.
+
+`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
+drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
+bound to nothing.
+
+## What I verified
+
+### The real prod run (read-only, all three nodes)
+
+`--list-hosts` first, every time. It named exactly three hosts and nothing else:
+
+    play #1 (k3s_prod): Inspect node hardware health (read-only)
+        pattern: ['k3s_prod']
+        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1
+
+Then `--syntax-check` (rc 0), then the run:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
+
+    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
+    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    PLAY rc=0
+
+(The `changed=` counts are the *controller-side* artifact writes under
+`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
+the collector task is `changed_when: false` and the role touches nothing.)
+
+All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
+
+    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
+    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
+    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
+
+    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
+    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
+    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+
+(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
+mine; none of them moved.)
+
+The new report section, **byte-identical on all three nodes**:
+
+    --------------------------------------------------------------------------------
+     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+    --------------------------------------------------------------------------------
+     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
+                  driver i915
+     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
+     dev  card0        character special  mode 0660  root:video gid 44
+     dev  renderD128   character special  mode 0660  root:render gid 992
+     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
+                  readable without root (i915 module parameters are mode 0400)
+                 no i915/xe parameter on the kernel cmdline
+     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
+     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
+
+That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
+character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
+`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
+CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
+
+**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
+prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
+absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
+`node_baseline_packages` and depending on an undeclared package would be wrong — but the
+comments and README that asserted it was missing were factually wrong and are now corrected.
+Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
+
+### The gate
+
+    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    GATE rc=3
+
+**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
+three stages did not run, all three the known local ceiling from a worktree, and **no fourth
+skip**:
+
+    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
+    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
+    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
+
+    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
+    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass
+
+Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
+(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
+the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
+know the shim took effect rather than the git stages silently degrading.
+
+`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
+`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
+claiming to have matched a lint convention that does not exist.
+
+### The CI gate on the PR — a green tick that is exit 3, read from the log
+
+The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
+on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
+"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.
+
+**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
+`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
+log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
+Gitea, the web route serves it) and read the gate's own headline:
+
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
+check is the tolerance swallowing that 3, not a rc 0.
+
+**CI verified two things my local run could not, and both came back green:**
+
+    [ ok ] stage gitea-settings: pass
+    [ ok ]   enable_push = false          enable_status_check = true
+    [ ok ]   required_approvals = 1       block_admin_merge_override = true
+    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
+    [ ok ]   runners online: 1 (>= 1)
+    [ ok ] Gitea settings match the declaration.
+    [ ok ] stage approval-policy: pass
+    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.
+
+My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
+`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
+there would have failed the job. That is measured confirmation that the live Gitea matches
+`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
+reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
+(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.
+
+Corroborated independently: `GET /repos/.../branches/main` →
+`protected=true user_can_push=false enable_status_check=true required_approvals=1`,
+contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
+status=online`. And note the commit's **combined** state is `pending`, not success, because
+`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
+not in the required list, so it does not block, but the PR is not "all green" either.
+
+The gate's `lint` stage passed with one `line too long` **warning** on
+`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
+above my insertion point, and warnings do not fail the stage.
+
+### Offline work (before the lock was released)
+
+- `python3 -m py_compile` on the collector: clean.
+- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
+  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
+  structure with no exception, correctly reporting two cards, two render nodes, the bound
+  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
+  without root* rather than as a fabricated `0`.
+- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
+  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
+  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
+  earlier draft would have silently reported a null.
+- Two other bugs found and fixed by running it rather than reasoning about it:
+  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
+  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
+  excluded from the card scan.
+- All three checks and the new template section were rendered through a standalone Jinja
+  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
+  workstation; GPU present with the render node removed; no display device at all; `xe` bound
+  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
+  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
+  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
+  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
+  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
+  Ansible's `trim_blocks=True`.
+  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
+  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
+  dictionaries, not machines. They are not hardware tests and must not be written up as if
+  they were.
+
+## What I did NOT verify
+
+- **The FAIL path has never fired on a real node.** Every prod node has a working render
+  device, and this role is read-only, so a device cannot be taken away to prove the check
+  catches its absence. The failure branches were exercised against **synthetic fact sets
+  only** (see above). I am not claiming the failure path is tested against hardware.
+- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
+  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
+  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
+  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
+  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
+- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
+  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
+  is measured here. I did **not** confirm against kernel source that no other kernel version or
+  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
+  line, not a false verdict, which is why this failure mode is tolerable.
+- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
+  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
+  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
+  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
+  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
+- **Not a transcode test.** These checks prove the render *device* exists, is a character
+  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
+  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
+  node-level check that installs nothing.
+- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
+  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
+  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
+  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
+  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
+- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
+  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
+- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
+  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
+  It is the expected, tolerated skip — but it is still a skip, not a pass.
+- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
+  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
+  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
+  passed — so those two I assert on measurement, not on this file's prose.
+- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
+  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
+  exists. That is real, it is out of this change's scope, and it is unclaimed work.
@@ -1,175 +0,0 @@
---
-date: 2026-09-30
-branch: feat/oauth2-proxy-auth-failure-visible
-title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
---
-
-## What changed
-
-Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
-operator's report named the property that matters — *"reloading the page never fixes the issue and
-the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
-about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
-the alert rules. It is an absence of data.
-
-**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
-That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
-jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
-`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
-starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
-cluster selects them.
-
-So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
-a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
-exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
-permanently silent, and indistinguishable from health. It was not written.
-
-What was written instead alerts on the condition that made the incident invisible, from families
-that **are** scraped:
-
- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
-
-      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-        unless
-      count by (namespace) (up{container="oauth2-proxy"})
-
-  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
-  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
-  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
-  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
-  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
-  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
-  clears per-namespace as each gate gains a real scrape.
- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
-  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
-  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
-  prod, not invented.
- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
-  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
-  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
-  `configMapGenerator` and listed in `runbooks/README.md`.
- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
-  per-app sidecars, plus the measured reason.
- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
-  three-part change that would close the gap.
-
-**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
-`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
-need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
-and — decisively — **it would not have fired on this incident**: the shared instance is not in the
-data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
-at all, having proxied nothing since they started (the counter exists in the TSDB only from an
-older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
-Shipping it would have looked like coverage of the reported failure and been none.
-
-## What I verified
-
-Everything below was measured against live prod through the pinned container, read-only. No
-`apply`, no `edit`, no `patch`, no `flux reconcile`.
-
- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
-  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
-  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
-  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
-  `1`, `namespace="oauth2-proxy"`.
- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
-  `oauth2-proxy` container, with its declared ports: the two shared replicas have
-  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
-  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
-  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
-  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
- **No existing alert is inert *for this reason*, because no alert references these metrics at
-  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
-  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
-  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
-  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
-  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
-  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
-  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
-  observable from the IdP side.
- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
-  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
-  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
-  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
- **The alert expression fires against live data.** Run verbatim on prod:
-  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
-  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
-  radarr`, value 1 each; `oauth2-proxy` correctly absent.
- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
-  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
-  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
-  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
-  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
-  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
-  runbook as the thing to compare against.
- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
-  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
-  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
-  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
-  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
-  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
-  `runbook_url` points at exactly that name.
- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
-  per-stage outcomes in the next section — read them there, not here.
-
-## What I did NOT verify
-
- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
-  only in git on this branch. What is proven is that its expression returns the expected vector
-  against live prod data and that promtool agrees about its behaviour over synthetic series.
-  **Merging is what deploys it.**
- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
-  any useful sense** — it describes a standing condition that has been true since the sidecars
-  were deployed, so it would have been firing before, during and after. That is deliberate and it
-  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
-  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
-  instead of discovering it during the investigation. **No alert on this platform could have
-  detected the incident itself, and none can today.**
- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
-  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
-  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
-  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
-  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
-  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
-  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
-  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
-  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
-  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
-  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
-  directory from the one the sidecar writes), and the rendered config contains **no
-  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
-  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
-  caught this incident, and is the single highest-value follow-up here — **would evaluate and
-  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
-  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
-  inert rule ConfigMap to look productive.**
- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
-  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
-  snapshot; anything older than three days is already gone. This is stated in the runbook because
-  it bounds what an operator can still investigate.
- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
-  requirement touched by this branch is certified. See the exit code below — **rc 3 is
-  UNDETERMINED, not green.**
-
-### Follow-ups, in order of value
-
-1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
-   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
-   already being collected. The rule is one expression:
-   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
-   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
-2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
-   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
-   and it unlocks the ratio rule in Prometheus.
-3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
-   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
-   observable, and it belongs with whoever owns the sidecar args.
@@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0

 ## Contract

- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
-  and — only when it can escalate — `smartctl`. It writes nothing to the node and
-  **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
+- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`
+  (and `lspci` only if it happens to be installed), and — only when it can escalate —
+  `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
  absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
  package installation is `node_baseline`'s job, never a health check's, never a manual
  `apt-get`).
@@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
 | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
+| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) |
+| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) |
+| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
@@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t
 back to re-run the role that had already run.


+## The render-device check
+
+Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing,
+nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom
+is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard
+IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly
+that silent degradation.
+
+The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A
+fact measured once decays silently**, so it is re-measured on every health run instead of
+living in a runbook. Nothing is installed, no module is loaded, no group is changed — the
+role's read-only contract is unchanged.
+
+| fact | where it comes from | root? |
+|---|---|---|
+| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no |
+| owning group **name** for the render node's gid | `grp.getgrgid()` | no |
+| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no |
+| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no |
+| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no |
+| module load state | `/proc/modules` | no |
+| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** |
+
+### The PCI scan is separate from the DRM scan on purpose
+
+`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on
+the bus with nothing attached has no `card0` at all — and that is precisely the failure being
+hunted. So the display device is looked for on the PCI bus independently, from sysfs, which
+is always there. That is what lets `graphics.render_device` distinguish
+
+- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention
+  `thermal.max` uses for "no sensor on this hardware"), from
+- *a GPU is on the bus and there is still no render node* → **FAIL**.
+
+`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as
+optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing
+in this repo guarantees it. It did turn out to be present on all three prod nodes
+(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending
+on that would mean depending on an undeclared package. Absent, `description` is `null`, the
+numeric IDs are printed instead, and no verdict changes.
+
+### The render node's GID is reported, never asserted
+
+The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on
+2026-09-30. The check does **not** compare against `992`, and must not be changed to.
+`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid
+from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own
+container entrypoint stats the device and joins whichever group owns it, so drift genuinely
+is not a fault. Recording the observed group name and gid in the detail line makes drift
+**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a
+reinstall.
+
+What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver`
+(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a
+**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace
+media stack and a libva/iHD build that transcodes correctly against one is not automatically
+right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing.
+
+### Two things that are informational and must stay that way
+
+- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It
+  costs low-power H.264 encode *quality* only; decode and encode both still run on the media
+  engines. Grading it would put a permanent WARN on a chosen configuration.
+- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and
+  media driver inside its container, so the host needs neither. Flagging it would invite
+  someone to `apt install vainfo` on a node to clear a health check — and this role installs
+  nothing.
+
+`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually
+cannot read it. It reports as an honest `NOT READ — <reason>` rather than a fabricated `0`,
+with `/proc/cmdline` as the world-readable second source for an explicit boot override.
+
 ## The EFI memattr check

 `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
@@ -273,3 +348,10 @@ stays `null` — no claim is made either way.
  but the wear/endurance context SKIPs rather than guessing.
 - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
  unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
+- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod
+  node has a working render device, and the role is read-only so one cannot be taken away to
+  prove it. The failure branches were exercised against synthetic fact sets only. The same
+  applies to the `xe`-bound WARN: no node here has `xe` bound.
+- `graphics.*` proves the render **device** exists and which driver owns it. It does **not**
+  prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container,
+  with Plex's own libva, which is out of scope for a node-level check that installs nothing.
@@ -35,6 +35,24 @@ node_health_try_privileged: true
 # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
 node_health_smartctl_args: "--json=c --all --nocheck=standby"

+# --- Graphics / render device ---------------------------------------------------------
+# The DRM driver this site EXPECTS to be bound to its display device. Measured on all three
+# prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP
+# EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to
+# nothing, and that difference matters enough to surface: the two drivers do not share a
+# userspace media stack (a libva/iHD build that works against i915 is not automatically the
+# right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD
+# or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic.
+#
+# WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning
+# group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static
+# one, so a rebuilt node may legitimately land on a different number — it was 992 on all
+# three prod nodes on 2026-09-30, and an assertion against that literal would be a false
+# alarm waiting to happen. Plex's own container entrypoint stats the device and joins
+# whichever group owns it, so drift genuinely is not a fault. The observed group name and GID
+# are therefore REPORTED in the check detail and never graded.
+node_health_graphics_expected_driver: i915
+
 # --- Thresholds ---------------------------------------------------------------------
 # All verdicts come from here, so a site can retune without touching task logic.
 node_health_thresholds:
@@ -35,10 +35,13 @@
 # =============================================================================

 import glob
+import grp
 import json
 import os
+import pwd
 import re
 import shutil
+import stat
 import subprocess
 import sys
 import time
@@ -756,6 +759,212 @@ def collect_pressure():
    return out


+# --- Graphics / DRM render device -----------------------------------------------------
+# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that
+# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that
+# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software
+# transcoding and the only symptom is a hot CPU. So the render device is measured on every
+# health run rather than being a fact someone established once by hand.
+#
+# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are
+# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so
+# are the module parameters under /sys/module. Nothing here needs root, so nothing here
+# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need
+# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is
+# what actually decides whether the firmware is loaded at all.)
+#
+# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists
+# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no
+# card0 at all, and that is precisely the failure this is meant to catch — so the display
+# device is looked for on the PCI bus independently, from sysfs, which is always there.
+# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as
+# optional because it IS optional: pciutils is not declared in node_baseline_packages, so
+# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes
+# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on
+# that would make the collector depend on an undeclared package. A missing lspci leaves
+# `description` null, the numeric IDs are reported instead, and no verdict changes.
+DISPLAY_PCI_CLASS_PREFIX = "0x03"  # PCI base class 03 = display controller (VGA/3D/other)
+# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative
+# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is
+# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact.
+DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia")
+
+
+def _uid_name(uid):
+    try:
+        return pwd.getpwuid(uid).pw_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _gid_name(gid):
+    """Group NAME for a numeric gid, or None when nothing resolves it.
+
+    Reported as an OBSERVED value, never asserted against a literal. The `render` group is
+    a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's
+    static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own
+    container init stats the device and joins whatever group owns it, so a moved gid is
+    drift to notice, not a fault to fail on."""
+    try:
+        return grp.getgrgid(gid).gr_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _module_param(mod, param):
+    """One /sys/module/<mod>/parameters/<param>, with WHY it is missing when it is.
+
+    Three outcomes that must not be conflated: a real value; "the module exposes no such
+    parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no
+    enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915
+    module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an
+    unprivileged pass legitimately cannot read them. Returning null with the reason keeps
+    that an honest UNKNOWN instead of a fabricated 0. read_text() also records the
+    permission denial in UNREADABLE, so the summary lists it with everything else root
+    would add."""
+    path = "/sys/module/%s/parameters/%s" % (mod, param)
+    if not os.path.exists(path):
+        return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, "
+                                         "built in, or no such knob on this driver)" % (mod, param)}
+    value = read_text(path)
+    if value is None:
+        return {"value": None, "reason": "%s is not readable without root (i915 module "
+                                         "parameters are mode 0400)" % path}
+    return {"value": value, "reason": None}
+
+
+def _dev_node(path):
+    """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is."""
+    try:
+        st = os.lstat(path)
+    except OSError as exc:
+        UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc})
+        return None
+    return {
+        "path": path,
+        "name": os.path.basename(path),
+        "char_device": stat.S_ISCHR(st.st_mode),
+        "mode": "%04o" % stat.S_IMODE(st.st_mode),
+        "uid": st.st_uid,
+        "gid": st.st_gid,
+        "owner": _uid_name(st.st_uid),
+        "group": _gid_name(st.st_gid),
+    }
+
+
+def collect_graphics():
+    dri_dir = "/dev/dri"
+    dri_present = os.path.isdir(dri_dir)
+    devices = []
+    if dri_present:
+        for path in sorted(glob.glob(dri_dir + "/*")):
+            # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping
+            # directories rather than filtering to char devices is deliberate: anything
+            # in here that is NOT a character special file is an anomaly worth seeing, and
+            # `char_device: false` is how it becomes visible.
+            if os.path.isdir(path):
+                continue
+            entry = _dev_node(path)
+            if entry is not None:
+                devices.append(entry)
+    render_nodes = [d for d in devices if d["name"].startswith("renderD")]
+
+    # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR
+    # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped.
+    cards = []
+    for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")):
+        name = os.path.basename(cdir)
+        if "-" in name:
+            continue
+        driver = None
+        drvlink = cdir + "/device/driver"
+        if os.path.islink(drvlink):
+            driver = os.path.basename(os.path.realpath(drvlink))
+        address = None
+        devlink = cdir + "/device"
+        if os.path.islink(devlink):
+            address = os.path.basename(os.path.realpath(devlink))
+        # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC)
+        # engine and vecs* the video-enhancement engine: those two are what a transcode
+        # actually runs on, so their presence is the difference between "a GPU is bound"
+        # and "this GPU can do media work".
+        engines = sorted(
+            {
+                os.path.basename(p)
+                for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*"))
+                if os.path.isdir(p)
+            }
+        )
+        cards.append(
+            {
+                "card": name,
+                "driver": driver,
+                "pci_address": address,
+                "vendor_id": read_text(cdir + "/device/vendor"),
+                "device_id": read_text(cdir + "/device/device"),
+                "engines": engines,
+                "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))],
+            }
+        )
+
+    # Display-class devices on the PCI bus, driver bound or not.
+    pci_display = []
+    for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")):
+        class_id = read_text(pdir + "/class") or ""
+        if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX):
+            continue
+        address = os.path.basename(pdir)
+        driver = None
+        if os.path.islink(pdir + "/driver"):
+            driver = os.path.basename(os.path.realpath(pdir + "/driver"))
+        description = None
+        rc, out, _ = run(["lspci", "-nn", "-s", address])
+        if rc == 0 and out.strip():
+            description = out.strip().splitlines()[0]
+        pci_display.append(
+            {
+                "address": address,
+                "class_id": class_id or None,
+                "vendor_id": read_text(pdir + "/vendor"),
+                "device_id": read_text(pdir + "/device"),
+                "revision": read_text(pdir + "/revision"),
+                "driver": driver,
+                "description": description,
+            }
+        )
+
+    loaded = set()
+    for line in (read_text("/proc/modules") or "").splitlines():
+        fields = line.split()
+        if fields:
+            loaded.add(fields[0])
+    modules = {m: (m in loaded) for m in DRM_MODULES}
+
+    # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC
+    # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the
+    # one place the state is visible without root AND without debugfs — but the sysfs file
+    # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel
+    # command line is the world-readable second source: an explicit `i915.enable_guc=` there
+    # is authoritative about what was ASKED for, even when the parameter file cannot be read.
+    uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")}
+    cmdline_params = [
+        token
+        for token in (read_text("/proc/cmdline") or "").split()
+        if token.startswith(tuple("%s." % m for m in DRM_MODULES))
+    ]
+
+    return {
+        "dri_present": dri_present,
+        "devices": devices,
+        "render_nodes": render_nodes,
+        "drm_cards": cards,
+        "pci_display": pci_display,
+        "modules": modules,
+        "uc": uc,
+        "cmdline_params": cmdline_params,
+    }
+
+
 # --- Storage ------------------------------------------------------------------------
 PSEUDO_FS = {
    "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore",
@@ -1095,7 +1304,15 @@ def collect_os():
 # we only DETECT, and the role skips the SMART checks with an accurate reason.
 def collect_tools():
    tools = {}
-    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
+    # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in
+    # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod
+    # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and
+    # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo
+    # is not needed on the host at all, because Plex ships its own libva and media driver
+    # inside its container. They are probed so the report can state their presence or absence
+    # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted
+    # to install them. This role installs nothing.
+    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"):
        tools[tool] = which(tool)
    return tools

@@ -1127,6 +1344,7 @@ def main():
        "os": collect_os(),
        "dmi": collect_dmi(),
        "cpu": collect_cpu(),
+        "graphics": collect_graphics(),
        "memory": collect_memory(),
        "thermal": collect_thermal(),
        "pressure": collect_pressure(),
@@ -652,6 +652,204 @@
         ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
         ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}

+# --- Graphics / render device ----------------------------------------------------------
+# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
+# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
+# to software transcoding on its own and the only symptom is a hot CPU and stuttering
+# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
+# binds a different DRM driver all produce exactly that silent degradation. It was measured
+# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
+# decays silently, so it is re-measured on every health run instead.
+#
+# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
+# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
+# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
+# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
+# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
+# and it reports as an honest UNKNOWN with the reason rather than as a 0.
+#
+# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
+#   graphics.render_device  the OUTCOME — can anything hardware-transcode on this node.
+#   graphics.drm_driver     the CAUSE side — what is bound, and is it what this site expects.
+#   graphics.media_stack    INFO only. GuC/HuC and host VAAPI userspace both belong here and
+#                           NEITHER is ever a failure; see the task's own comment for why.
+- name: Assess — locate the display device and its bound driver (once, shared by three checks)
+  ansible.builtin.set_fact:
+    node_health_render:
+      # Display-class PCI devices (base class 0x03), driver bound or not. This is the
+      # question "is there a GPU in this machine at all", and it is answered from the PCI
+      # bus rather than from /sys/class/drm because a card* entry only exists once a driver
+      # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
+      # identical to a machine with no GPU.
+      display_devices: "{{ g.pci_display }}"
+      bound_drivers: >-
+        {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
+      # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
+      # is an anomaly, not a device, and must not be counted as one.
+      render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
+      # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
+      # difference between "a GPU is bound" and "this GPU can do media work".
+      media_engines: >-
+        {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+
+- name: Assess — DRM render node for hardware transcoding
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.render_device',
+      'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
+      'status': render_status,
+      'detail': render_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
+    # then nothing to grade, and the same convention thermal.max already uses for "no sensor
+    # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
+    # a GPU IS on the bus and there is still no render node.
+    render_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
+    dri_listing: >-
+      {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
+         else 'nothing' }}
+    render_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      this machine exposes no display-class PCI device (base class 0x03) at all, so there is
+      no GPU here for a render node to belong to and hardware transcoding is not available on
+      this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
+      not assumed.
+      {%- elif (r.render_chardevs | length) > 0 -%}
+      {{ r.render_chardevs | length }} render node(s):
+      {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
+      owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
+      gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
+      /dev/dri holds: {{ dri_listing }}.
+      THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
+      `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
+      node may legitimately land on a different number (all three prod nodes read gid 992 on
+      2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
+      owns it. Drift is therefore visible in this line without being a failure.
+      {%- else -%}
+      HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
+      PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
+      no usable character-special render node: /dev/dri
+      {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
+      {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
+      ~ ' — present, but NOT character special, which is an anomaly in its own right. ')
+         if (g.render_nodes | length) > 0 else '' }}
+      Nothing fails loudly in this state — Plex silently falls back to software transcoding
+      and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
+      driver bound at all, then whether the onboard IGD is still enabled in BIOS.
+      {%- endif -%}
+
+- name: Assess — DRM driver bound to the display device
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.drm_driver',
+      'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
+      'status': driver_status,
+      'detail': driver_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    expected: "{{ node_health_graphics_expected_driver }}"
+    # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
+    # still works, but the userspace media stack is not the same one, so it is a change the
+    # operator must see before a transcode regression surfaces as a user complaint. FAIL is
+    # for nothing bound at all, which is the cause of a missing render node.
+    driver_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('FAIL' if (r.bound_drivers | length) == 0
+               else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
+    driver_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so no DRM driver can be bound to one.
+      Measured from /sys/bus/pci/devices.
+      {%- else -%}
+      {% for p in r.display_devices %}{{ p.address }}
+      {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
+      -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
+      {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
+           | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
+         else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
+      Module load state (/proc/modules, corroboration only — a built-in driver never appears
+      there, the bound-driver symlink above is what decides):
+      {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
+         | map('join', '=') | join(', ') }}.
+      {%- if (r.bound_drivers | length) == 0 %}
+      NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
+      subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
+      module exists for this kernel and whether it was blacklisted or failed to probe.
+      {%- elif expected in r.bound_drivers %}
+      This is the expected driver for this site (node_health_graphics_expected_driver).
+      {%- else %}
+      EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
+      works, so this is not an outage — but the two DRM drivers do not share a userspace
+      media stack, and a libva/iHD build that transcodes correctly against one is not
+      automatically right for the other. Confirm Plex still hardware-transcodes before
+      treating this as benign.
+      {%- endif %}
+      {%- endif -%}
+
+# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
+#
+#   GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
+#   costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
+#   through the media engines. Grading it would put a permanent WARN on a configuration the
+#   operator has chosen.
+#   No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
+#   driver inside its container, so the host needs neither. Flagging it would invite someone
+#   to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
+#
+# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
+# (video-enhancement) are what a transcode actually runs on.
+- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.media_stack',
+      'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
+      'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
+      'detail': media_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
+    # that set the expected driver to amdgpu must get a clean "no such parameters" line
+    # instead of an undefined-key traceback.
+    guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
+    guc_text: >-
+      {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
+         else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
+               if guc is mapping
+               else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
+    media_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
+      firmware and nothing to transcode with.
+      {%- else -%}
+      media engines:
+      {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
+         else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
+      (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
+      GuC/HuC: {{ guc_text }}.
+      {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
+         else 'No i915/xe parameter on the kernel cmdline' }}.
+      GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
+      low-power H.264 encode quality only, and both decode and encode still run on the
+      engines above.
+      Host VAAPI userspace: vainfo
+      {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
+         else 'NOT installed on this host' }}
+      — also never graded, because Plex ships its own libva and media driver inside its
+      container, so the host needs none. This role installs nothing.
+      {%- endif -%}
+
 # --- Thermals -----------------------------------------------------------------------
 - name: Assess — temperatures
  ansible.builtin.set_fact:
@@ -136,6 +136,37 @@
 {% endfor %}
 {% endfor %}

+--------------------------------------------------------------------------------
+ GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+--------------------------------------------------------------------------------
+{% set gfx = node_health_facts.graphics %}
+{% if gfx.pci_display %}
+{% for p in gfx.pci_display %}
+ pci  {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }}
+              driver {{ p.driver | default('NONE BOUND', true) }}
+{% endfor %}
+{% else %}
+ pci         no display-class device (PCI base class 0x03) on this machine — no GPU here
+{% endif %}
+{% for c in gfx.drm_cards %}
+ drm  {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }}  engines {{ c.engines | join(' ') | default('none exposed', true) }}
+{% endfor %}
+{% if gfx.dri_present %}
+{% for d in gfx.devices %}
+ dev  {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }}  {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }}
+{% endfor %}
+{% else %}
+ dev         /dev/dri DOES NOT EXIST — no DRM device node on this node at all
+{% endif %}
+{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may
+   legitimately move it and Plex's container init joins whichever group owns the device. See
+   defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #}
+{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %}
+ GuC/HuC     {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
+             {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }}
+ modules     {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides)
+ vainfo      {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }}
+
 --------------------------------------------------------------------------------
 PRESSURE STALL (/proc/pressure — % of wall time stalled)
 --------------------------------------------------------------------------------
@@ -88,39 +88,3 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
-
-## Observability: what this component can and cannot see (measured 2026-09-30)
-
-`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
-two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
-in the TSDB is:
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
-`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
-each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
-listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
-their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
-written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
-
-This matters because the gate's characteristic failure is invisible by construction: an
-unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
-against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
-where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
-spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
-into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
-
-`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
-`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
-`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
-a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
-one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
-and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
-
-**To close the gap for an app**, three things are needed together, on that app's own manifest:
-`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
-Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
-`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
-session lifetime — become writable rather than decorative.
@@ -1,117 +0,0 @@
---
-# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
-#
-# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
-# failure modes of that shape are both silent: a label that does not survive `count by` makes the
-# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
-# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
-# visible by reading the expression, and neither is visible on a live cluster where the answer
-# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
-# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
-#
-# HOW TO RUN. The one repo-wide harness, by the companion convention:
-#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
-# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
-#
-# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
-# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
-# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
-# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
-# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
-# pass while the real expression collapsed differently.
-#
-# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
-# comparing it verbatim would couple this file to every wording change while proving nothing the
-# ALERTS series does not already prove.
-rule_files:
-  - auth-visibility-alerts.rules.yaml
-evaluation_interval: 1m
-
-tests:
-  # ---------------------------------------------------------------------------------------------
-  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
-  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
-  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
-  - interval: 1m
-    name: an unscraped gate namespace fires after 1h; the scraped one never does
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # The shared instance: kube-state-metrics sees both replicas...
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
-        values: '1x130'
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
-        values: '1x130'
-      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
-        values: '1x130'
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
-        values: '1x130'
-    promql_expr_test:
-      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
-      # the hold away and turning every rollout into a page.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 59m
-        exp_samples: []
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
-        eval_time: 59m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
-            value: 1
-      # Firing for jackett past the hold...
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      # ...and never for the namespace that IS scraped. This is the assertion that catches an
-      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
-      # every gate namespace would pass the test above.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
-        eval_time: 65m
-        exp_samples: []
-      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
-      # that lost its `by` would collapse to a single unlabelled series.
-      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
-        eval_time: 65m
-        exp_samples:
-          - labels: '{namespace="jackett"}'
-            value: 1
-
-  # ---------------------------------------------------------------------------------------------
-  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
-  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
-  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
-  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
-  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
-  - interval: 1m
-    name: the alert clears once the sidecar is actually scraped
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
-      # exist yet.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
-        values: '_x70 1x60'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
-        eval_time: 80m
-        exp_samples: []
-
-  # ---------------------------------------------------------------------------------------------
-  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
-  #    rule would page for every app on the platform the day someone widened the matcher.
-  - interval: 1m
-    name: namespaces without a gate sidecar are never reported
-    input_series:
-      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
-        values: '1x130'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
-        eval_time: 65m
-        exp_samples: []
@@ -1,113 +0,0 @@
---
-# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
-#
-# ============================================================================================
-# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
-# ============================================================================================
-# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
-# log in. The sidecar's own access log had the whole story:
-#
-#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
-#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
-#
-# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
-# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
-# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
-# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
-# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
-# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
-# gate had rejected every request ever made.
-#
-# ============================================================================================
-# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
-# ============================================================================================
-# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
-# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
-# reason is this file's real subject:
-#
-#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
-#
-#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
-#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
-#
-#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
-#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
-#   is never started; the container declares only port `http:4180`, there is no `metrics` port on
-#   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
-#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
-#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
-#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
-#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
-#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
-#   temptation was refused here rather than ratcheted into a register.
-#
-#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
-#   path for any sidecar-gated app — each app gates itself — and its current pods export no
-#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
-#   the shared instance would not have fired on this incident either. Saying so is the point.
-#
-# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
-# That condition is measurable today, from two already-scraped metric families, and it is the
-# precondition for ever being able to alert on the incident itself. It self-clears the moment a
-# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
-# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
-# by-hand equivalent an operator uses in the meantime.
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: oauth2-proxy-auth-visibility
-  namespace: oauth2-proxy
-  labels:
-    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
-    app.kubernetes.io/name: oauth2-proxy
-spec:
-  groups:
-    - name: oauth2-proxy.auth-visibility
-      rules:
-        - alert: OAuth2ProxyGateUnobserved
-          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
-          # target for that container in that namespace.
-          #
-          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
-          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
-          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
-          #     container exports anything, which is exactly the property needed here.
-          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
-          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
-          #     without knowing what job name it will pick.
-          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
-          # inert for the reason it is reporting.
-          #
-          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
-          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
-          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
-          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
-          # the state the cluster is in; it is not a rule waiting for a hypothetical.
-          expr: |
-            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-              unless
-            count by (namespace) (up{container="oauth2-proxy"})
-          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
-          # discovered yet, and this condition is a standing architectural gap, never an
-          # emergency. Long enough that no deploy can produce it.
-          for: 1h
-          labels:
-            severity: warning
-          annotations:
-            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
-            description: >-
-              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
-              Prometheus has NO scrape target for it, so every authentication decision that gate
-              makes — every redirect to Keycloak, every 403 on a group check, every session that
-              silently fails to hold — is invisible to metrics and to alerting. This is not a
-              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
-              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
-              seconds later, and the only record anywhere was a line in the sidecar's stdout.
-              No alert could have fired, because there is nothing to alert on. Until a scrape
-              exists the runbook's LogQL queries are the substitute; the fix is
-              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
-              ServiceMonitor, after which the auth-churn rule becomes writable.
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,7 +14,3 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
-  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
-  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
-  # tier, which is Ready before this tier's platform-config dependency completes.
-  - auth-visibility-alerts.yaml
@@ -51,18 +51,6 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
-| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
-
-The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
-there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
-exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
-forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
-`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
-and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
-namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
-(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
-`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
-real scrape. See `oauth2-proxy-auth-failures`.

 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.

@@ -120,14 +108,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
-  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
-  rather than a 401, the browser follows it silently against a live SSO session, and the call
-  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
-  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
-  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
-  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
-  so these logs (72h of them) are the only record this class of failure leaves.

 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -85,10 +85,6 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
-      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
-      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
-      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
-      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
      # because the plex.tv claim window is short and is not a thing to go looking
@@ -1,171 +0,0 @@
-# oauth2-proxy auth failures — telling the three layers apart
-
-Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
-does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
-stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
-page is the diagnostic procedure, and it works whether or not the alert ever fired.
-
-## What this means here
-
-Every app behind Keycloak on this platform has **three** places a request can be refused, and
-from a browser they look identical — because two of the three answer with **HTTP 200**.
-
-| Layer | What refuses | What the browser sees |
-|---|---|---|
-| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
-| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
-| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
-
-**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
-redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
-JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
-Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
-dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
-nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
-failed.
-
-The worked example, jackett on 2026-09-30, straight from the sidecar log:
-
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-    ... ten of these at 15:30:53
-    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
-                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
-    GET "/UI/Dashboard"        200 44379
-    GET "/api/v2.0/indexers?_=..." 200 1501229
-    ... and 26 seconds later, inside that same 5-minute session:
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-
-A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
-signature of the session being **re-established over and over instead of held** — a cookie that
-another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
-down and not of the app being broken.
-
-**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
-run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
-`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
-`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
-apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
-convenience — until that changes they are the **only** record that this class of failure happened.
-
-## First commands
-
-`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
-with an explicit context.
-
-**1. Which namespaces have an unobserved gate** (this is what the alert measures):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
-```
-
-**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
-and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
-`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
-     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
-```
-
-Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
-
-**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
-you can ask about the whole fleet and about a window that has already scrolled out of
-`kubectl logs`. Port-forward Loki and query it:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
-   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
-     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
-```
-
-and the denominator, which is what makes the number mean something:
-
-```
-   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
-```
-
-**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
-lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
-ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
-
-| namespace | `No valid authentication` | `AuthSuccess` | ratio |
-|---|---|---|---|
-| crafty | 81 | 1 | 81:1 |
-| radarr | 67 | 1 | 67:1 |
-| jackett | 39 | 4 | ~10:1 |
-| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
-
-All three of those are the failure, not the baseline.
-
-**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
-and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
-page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
-shape, and `runbooks/crafty-admin-credential.md` covers it.
-
-**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
-**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
-other's cookie and the loser logs exactly the `No valid authentication` line above. List what
-each gate is configured to emit (arguments only — never print a Secret's data):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
-     kubectl --context k3s-prod -n $ns get deploy $ns \
-       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
-     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
-```
-
-A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
-share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
-evicting each other, and the symptom is intermittent and depends on which app you visited last.
-
-## Likely causes, most likely first
-
-1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
-   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
-   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
-   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
-   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
-2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
-   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
-   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
-   gate logs the same line. Look for numbered cookie names in the log, and for a session that
-   never survives a single navigation.
-3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
-   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
-   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
-   jackett trace above.
-4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
-   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
-   die early, the refresh is failing, not missing.
-5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
-   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
-   for the `groups:[...]` list on the `[AuthSuccess]` line.
-6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
-
-## When to escalate or stop
-
- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
-  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
-  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
-  reverts, and in the meantime you have an unreproducible cluster.
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
-  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
-  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
-  transcript here exactly this way.
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
-  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
-  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
-  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
-  report is older than three days, the evidence is gone and the only remaining move is to
-  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
-  the reason to close it rather than silence it.