docs(obs): record why CI is red — approval-policy on the one conformance entry

The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
2026-10-01 22:12:19 -05:00 · 2026-10-01 21:56:07 -05:00 · 2026-10-01 21:40:37 -05:00 · 2026-09-30 10:55:19 -05:00
23 changed files with 1284 additions and 1456 deletions
@@ -137,15 +137,14 @@ spec:
            secretKeyRef:
              name: plex-claim
              key: PLEX_CLAIM
-        resources:
-          limits:
-            gpu.intel.com/i915: 1
        volumeMounts:
        - name: config
          mountPath: /config
        - name: media
          mountPath: /media
          readOnly: true
+        - name: dri
+          mountPath: /dev/dri
      volumes:
      - name: config
        persistentVolumeClaim:
@@ -153,6 +152,10 @@ spec:
      - name: media
        persistentVolumeClaim:
          claimName: plex-media
+      - name: dri
+        hostPath:
+          path: /dev/dri
+          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1
@@ -81,49 +81,46 @@ env:
  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
-# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
-# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
-# How the platform hands the device over is `catalog/render.py`'s business and is documented in
-# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
-# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
-# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
-# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
-# whole point of the enum.
+# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
+# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
+# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
+# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
+# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
+# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
+# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
-# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
-# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
-# nodeSelector; placement is a separate concern with its own justification.
-#
-# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
-# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
-# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
-# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
-# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
-# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
-# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
-# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
-# carries that procedure for whoever needs it at 2am.
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
+# that is scheduler history rather than a requirement — this field deliberately renders no
+# nodeSelector, because placement is a separate concern with its own justification.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
-# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
-# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
-# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
-# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
+# CONTAINER, 2026-09-30: the image's own
+# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
+# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
-# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
-# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
-# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
-# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
-# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
-# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
-# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
-# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
+# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
+# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
+# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
+# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
+# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
+# securityContext for this field precisely so nobody adds an inert one back later.
 #
-# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
+# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
+# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
+# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
+# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
+# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
+# dashboard.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
@@ -131,7 +128,8 @@ env:
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
-# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
+# device-cgroup failure mode to expect first.
 hardware:
  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
@@ -166,14 +166,11 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
-# deliberately confined to this file. This file is the ONLY place the mechanism appears,
-# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
-# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
-#
-# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
-# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
-# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
-QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
+# deliberately confined to this file. Swapping to the Intel GPU device plugin
+# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
+# nowhere else -- no catalog entry moves.
+DRI_VOLUME_NAME = "dri"
+DRI_DEVICE_DIR = "/dev/dri"


 # =============================================================================
@@ -529,59 +526,41 @@ def r_deployment(s: dict) -> dict:
    # workload needs the host iGPU's render node for hardware video transcode. The SD says
    # only that; the wiring below is the platform's current mechanism for satisfying it.
    #
-    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
-    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
-    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
-    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
-    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
-    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
-    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
-    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
-    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
-    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
-    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
-    # because Kubernetes cannot infer one from a volume.
+    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
+    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
+    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
+    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
+    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
+    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
+    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
+    # directory the kubelet creates on such a node leaves nothing durable behind.
    #
-    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
-    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
-    # kubelet turns into both the device injection and the matching cgroup allow.
-    #
-    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
-    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
-    # requests-only extended resource is invalid. App container only, never a sidecar -- the
-    # oauth2-proxy sidecar does no transcoding.
-    #
-    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
-    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
-    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
-    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
-    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
-    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
-    # extended resource is a hard scheduling constraint, and it is also the only thing that
-    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
-    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
-    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
-    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
-    # unblocks. runbooks/plex-quicksync.md carries it.
+    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
    #
    # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
-    # would be a regression, not a hardening. MEASURED in the running container on prod-2
-    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
+    # 2026-09-30: the image's own cont-init hook
    # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
    # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
    # container when none exists, and `usermod -a -G` the app user into it -- every start,
-    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
-    # is why this swap needs no group work either. cont-init.d completes before services.d,
-    # so the account database s6 reads at exec time already carries the group: measured live,
-    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
-    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
-    # REPLACES the process group list from /etc/group rather than appending to it, so
-    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
-    # before the app ever starts. A supplementalGroups entry here would be inert AND would
-    # mislead whoever debugs this next.
+    # nothing hardcoded. cont-init.d completes before services.d, so the account database
+    # s6 reads at exec time already carries the group. The service is then started with
+    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
+    # /etc/group rather than appending to it: `Plex Media Server` runs with
+    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
+    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
+    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
+    # group handling is the image's job and it does it correctly. A supplementalGroups entry
+    # here would be inert AND would mislead whoever debugs this next.
    hw_gpu = (s.get("hardware") or {}).get("gpu")
    if hw_gpu == "intel-quicksync":
-        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
+        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
+        volumes.append(
+            {
+                "name": DRI_VOLUME_NAME,
+                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
+            }
+        )

    if volume_mounts:
        container["volumeMounts"] = volume_mounts
@@ -2600,13 +2579,10 @@ def _selftest() -> int:
        "an SD without podSecurityContext must not gain a pod securityContext",
    )

-    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
-    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
-    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
-    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
-    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
-    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
-    #     the emission site), and the absence assertion is what stops it coming back.
+    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
+    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
+    #     assertions pin the MECHANISM that currently satisfies it -- including what it
+    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
    hw_docs = render(
        {
            "name": "hw",
@@ -2617,36 +2593,24 @@ def _selftest() -> int:
    hw_pod = next(
        d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
-    hw_app = hw_pod["containers"][0]
-    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
-    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
-    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
-    # CPU/memory resources feature through this field (that would be scope creep, and this
-    # field is about the DEVICE only).
    check(
-        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
-        "hardware.gpu=intel-quicksync must render resources.limits "
-        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
+        {
+            "name": "dri",
+            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
+        }
+        in hw_pod.get("volumes", []),
+        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
+        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
    )
-    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
-    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
+    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
    check(
-        not any(
-            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
-        )
-        and not any(
-            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
-            for m in hw_app.get("volumeMounts", [])
-        ),
-        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
-        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
-        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
+        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
+        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
    )
-    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
-    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
-    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
-    # (see the emission site). Asserting the absence is what stops someone adding an inert
-    # one back.
+    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
+    # own cont-init hook adds the app user to the device's group inside the container, and
+    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
+    # emission site). Asserting the absence is what stops someone adding an inert one back.
    check(
        "securityContext" not in hw_pod,
        "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
@@ -2654,12 +2618,10 @@ def _selftest() -> int:
    )
    # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
    # rejects `privileged: true` in an app namespace, and placement is a separate concern
-    # with its own justification -- this field is about the DEVICE only. (The extended
-    # resource now constrains placement to nodes the plugin advertises, which is the
-    # scheduler's own doing, not a nodeSelector this renderer writes.)
+    # with its own justification -- this field is about the DEVICE only.
    check(
        "privileged" not in hw_pod.get("securityContext", {})
-        and "securityContext" not in hw_app
+        and "securityContext" not in hw_pod["containers"][0]
        and "nodeSelector" not in hw_pod,
        "hardware.gpu must emit no privileged securityContext and no nodeSelector",
    )
@@ -2672,10 +2634,8 @@ def _selftest() -> int:
    check(
        "volumes" not in nohw_pod
        and "volumeMounts" not in nohw_pod["containers"][0]
-        and "resources" not in nohw_pod["containers"][0]
        and "securityContext" not in nohw_pod,
-        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
-        "securityContext",
+        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
    )
    # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
    # hardware field neither adds to it nor rewrites it.
@@ -2692,35 +2652,9 @@ def _selftest() -> int:
        if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
    check(
-        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
-        and hwm_pod["containers"][0].get("resources")
-        == {"limits": {"gpu.intel.com/i915": 1}},
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
        "hardware.gpu must leave the SD's own podSecurityContext untouched",
    )
-    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
-    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
-    hws_containers = next(
-        d[1]
-        for d in render(
-            {
-                "name": "hws",
-                "source": {"image": "a:1"},
-                "hardware": {"gpu": "intel-quicksync"},
-                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
-                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
-            }
-        )
-        if d[1]["kind"] == "Deployment"
-    )["spec"]["template"]["spec"]["containers"]
-    check(
-        len(hws_containers) == 2
-        and hws_containers[0].get("resources")
-        == {"limits": {"gpu.intel.com/i915": 1}}
-        and "gpu.intel.com/i915"
-        not in hws_containers[1].get("resources", {}).get("limits", {}),
-        "hardware.gpu must put the i915 limit on the APP container only, never on the "
-        "oauth2-proxy sidecar",
-    )

    # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
    #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
@@ -1,104 +0,0 @@
---
-# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
-# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
-# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
-#
-# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
-#
-# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
-# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
-# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
-# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
-#
-#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
-#     you "the GPU plugin is wedged" from "longhorn is slow".
-#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
-#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
-#     plugin.
-#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
-#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
-#
-# And the consequence is worse here than for an average component, because of what sits downstream:
-# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
-# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
-# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
-# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
-#
-# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
-# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
-# path, and authors cannot self-approve. So this change needed the operator's approval either way;
-# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
-#
-# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
-#
-# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
-# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
-# it. It never uses the network and never talks to the API server; it talks to the local kubelet
-# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
-# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
-# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
-#
-# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
-# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
-# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
-# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
-# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
-# constraint on the next PR and not a formality.
-#
-# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
-# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
-# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
-#
-# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
-# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
-# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
-# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
-# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
-# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
-# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
-# the operator must answer before the consumer PR may merge.
-#
-# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
-# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
-# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
-# waves that would hand a device plugin the power to stall every app, which is the thing this file
-# exists to prevent. Think before adding that edge.
-#
-# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
-# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
-# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
-# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
-# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
-# no-op into a real outage. This PR's only job is to make the resource exist.
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: intel-gpu-plugin
-  namespace: flux-system
-spec:
-  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
-  # so this is only the drift-correction floor.
-  interval: 10m
-  retryInterval: 2m
-  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
-  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
-  # out that exact mismatch as one of its reasons for leaving the shared tier.
-  timeout: 10m
-  prune: true
-  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
-  wait: true
-  dependsOn:
-    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./platform/intel-gpu-plugin
-  # Retained for parity with every other tier / future overlay knobs. This component references no
-  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
-  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
-  postBuild:
-    substituteFrom:
-      - kind: ConfigMap
-        name: cluster-vars
-  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
-  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
@@ -12,15 +12,6 @@ resources:
  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
-  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
-  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
-  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
-  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
-  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
-  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
-  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
-  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
-  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
        "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
        "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
        "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
+        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
+        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
+        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
+        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
+        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
+        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
+        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
+        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
+        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
+        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
+        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
+        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
+        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
+        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
+        "resolves; the network path does not yet.",
 }

 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
@@ -0,0 +1,289 @@
+---
+date: 2026-09-30
+branch: feat/oauth2-proxy-auth-failure-visible
+title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
+---
+
+## What changed
+
+Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
+operator's report named the property that matters — *"reloading the page never fixes the issue and
+the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
+about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
+the alert rules. It is an absence of data.
+
+**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
+That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
+jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
+`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
+starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
+cluster selects them.
+
+So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
+a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
+exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
+permanently silent, and indistinguishable from health. It was not written.
+
+What was written instead alerts on the condition that made the incident invisible, from families
+that **are** scraped:
+
+- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
+
+      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+        unless
+      count by (namespace) (up{container="oauth2-proxy"})
+
+  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
+  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
+  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
+  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
+  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
+  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
+  clears per-namespace as each gate gains a real scrape.
+- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
+  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
+  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
+  prod, not invented.
+- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
+  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
+  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
+  `configMapGenerator` and listed in `runbooks/README.md`.
+- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
+  per-app sidecars, plus the measured reason.
+- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
+  three-part change that would close the gap.
+
+**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
+`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
+need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
+and — decisively — **it would not have fired on this incident**: the shared instance is not in the
+data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
+at all, having proxied nothing since they started (the counter exists in the TSDB only from an
+older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
+Shipping it would have looked like coverage of the reported failure and been none.
+
+## What I verified
+
+Everything below was measured against live prod through the pinned container, read-only. No
+`apply`, no `edit`, no `patch`, no `flux reconcile`.
+
+- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
+  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
+  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
+  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
+  `1`, `namespace="oauth2-proxy"`.
+- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
+  `oauth2-proxy` container, with its declared ports: the two shared replicas have
+  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
+  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
+  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
+  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
+- **No existing alert is inert *for this reason*, because no alert references these metrics at
+  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
+  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
+  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
+  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
+  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
+- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
+  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
+  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
+  observable from the IdP side.
+- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
+  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
+  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
+  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
+- **The alert expression fires against live data.** Run verbatim on prod:
+  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
+  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
+  radarr`, value 1 each; `oauth2-proxy` correctly absent.
+- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
+  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
+  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
+  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
+  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
+  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
+  runbook as the thing to compare against.
+- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
+  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
+  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
+- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
+  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
+  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
+  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
+  `runbook_url` points at exactly that name.
+- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
+  per-stage outcomes in the next section — read them there, not here.
+
+## What I did NOT verify
+
+- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
+  only in git on this branch. What is proven is that its expression returns the expected vector
+  against live prod data and that promtool agrees about its behaviour over synthetic series.
+  **Merging is what deploys it.**
+- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
+  any useful sense** — it describes a standing condition that has been true since the sidecars
+  were deployed, so it would have been firing before, during and after. That is deliberate and it
+  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
+  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
+  instead of discovering it during the investigation. **No alert on this platform could have
+  detected the incident itself, and none can today.**
+- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
+  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
+  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
+  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
+  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
+  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
+  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
+  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
+- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
+  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
+  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
+  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
+  directory from the one the sidecar writes), and the rendered config contains **no
+  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
+  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
+  caught this incident, and is the single highest-value follow-up here — **would evaluate and
+  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
+  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
+  inert rule ConfigMap to look productive.**
+- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
+  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
+- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
+  snapshot; anything older than three days is already gone. This is stated in the runbook because
+  it bounds what an operator can still investigate.
+- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
+  requirement touched by this branch is certified. See the exit code below — **rc 3 is
+  UNDETERMINED, not green.**
+
+### Follow-ups, in order of value
+
+1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
+   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
+   already being collected. The rule is one expression:
+   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
+   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
+2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
+   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
+   and it unlocks the ratio rule in Prometheus.
+3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
+   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
+   observable, and it belongs with whoever owns the sidecar args.
+
+---
+
+# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
+
+Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
+and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
+conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
+
+## What changed
+
+**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
+them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
+and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
+forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
+not an app list, so a sixth forward-auth app needs no edit.
+
+**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
+This is a real bug fix, and it is the most important thing in this round.** A target that is
+discovered but failing still publishes `up`, with value **0**. The series exists, so
+`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
+would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
+reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
+whose scrape is dropped, so the bare form would have silently cleared on this very commit.
+
+**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
+target that works and then breaks must re-fire rather than latch cleared.
+
+**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
+all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
+and the alert header, each showing the two-step history, because each step looked like the end of it.
+The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
+
+**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
+endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
+it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
+succeeds, not the day a listener exists.
+
+## What I verified
+
+- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
+  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
+  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
+  being in `args` and `/metrics` returning a body are different claims; both now hold.
+- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
+  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
+  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
+  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
+  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
+  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
+  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
+- **No Service exposes 44180** on any of the five (enumerated).
+- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
+  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
+- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
+  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
+  `jackett` from this tier would apply today and fail on a cold rebuild.
+- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
+  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
+  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
+- **Gate:** see the next section for the exit code and skips.
+
+## What I did NOT verify
+
+- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
+  measured pod labels and port name, not observed — no target list has been read with it in place,
+  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
+  the documented state, not a regression.
+- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
+  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
+  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
+  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
+  the catalog/renderer that is in flight on another branch. **It needs an owner.**
+- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
+  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
+  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
+  measured with this object.
+- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
+  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
+  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
+- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
+  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
+- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
+
+### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
+
+After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
+toolbox run was rc 3 with nothing failed. Measured cause, singular:
+
+    sensitive paths touched:
+      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
+    approving reviews: (no reviews at all)        author=gitea-admin
+
+`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
+`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
+`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
+
+**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
+short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
+touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
+signal. That is a general trap, not specific to this change, and it is worth knowing before someone
+concludes a red PR means broken code.
+
+To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
+status check, so the required context stays red until the job runs again. `required_approvals=1`
+already applied to every PR here, so the human was needed regardless; the conformance entry only
+turns the tick red while that is pending.
+
+The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
+check fails without the declaration). I did not take it: the entry is the "declare the limit"
+mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
+`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
+reviewer should make it knowingly.
@@ -1,198 +0,0 @@
---
-date: 2026-10-01
-branch: feat/intel-gpu-device-plugin
-title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
---
-
-## What changed
-
-Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
-**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
-Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
-device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
-1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
-hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
-response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
-it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
-`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
-`app-namespace=true`.
-
-New files:
-
- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
-  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
-  provenance header carrying the exact re-vendor command and both sha256s (generated body
-  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
-  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
-  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
-  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
-  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
-  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
-  kustomize base, for the other reason `external-snapshotter` records: a remote base is
-  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
-  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
-  non-choices, and the post-merge verification sequence.
- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
-  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
-
-**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
-cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
-`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
-split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
-the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
-called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
-consequence is worse than average here, because `platform` gates `platform-config` gates
-`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
-would stall every app, so a component added to fix a **silent no-op** would have gained the ability
-to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
-`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
-of path and authors cannot self-approve.
-
-`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
-never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
-is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
-`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
-`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
-eleven of its components, which shrinks the cold-start window in which a node has not yet
-advertised the resource.
-
-`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
-`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
-making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
-genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
-rendered graph), `wait: true` gates nothing while turning
-`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
-answered before the consumer PR may merge.
-
-Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
-(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
-`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
-believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
-a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
-*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
-nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
-already adds the `plex` user to each device's group and that works identically on a
-plugin-injected device.
-
-**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
-are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
-later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
-on all three nodes** — a request for a resource no node advertises makes the Plex pod
-Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
-stall the whole app tier and turn today's silent no-op into a real outage.
-
-## What I verified
-
-Mechanism, read from the v0.37.0 source rather than assumed:
-
- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
-  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
-  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
-  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
-  grant and no extra privilege — it does not need more than the upstream manifest gives it.
- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
-  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
-  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
-  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
-  nodes have `i915` bound.
- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
-  `false` and `setupLevelZeroService` returns early unless it is set.
- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
-
-Image pin, verified against the registry:
-
- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
-  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
-  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
-  latest upstream release (2026-09-16) and is not a prerelease.
- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
-  while GitHub release tags do.
-
-Policy scope, read from the policies rather than assumed:
-
- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
-  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
-  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
-  The new namespace is labelled `"false"`, so none of them match it.
- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
-  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
-  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
-  collision.
-
-Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
-bind-mounted read-only — never copied into the worktree):
-
- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
-  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
-  "NONE". That is the before-state this change is meant to move.
- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
-  measured evidence that the device-plugin API and the default kubelet socket path
-  (`/var/lib/kubelet/device-plugins`) work on these nodes.
-
-Offline rendering (the main proof available to a PR):
-
- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
-  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
-  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
-  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
-  (`clusters/prod`, `clusters/dev`) also render and carry it.
- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
-  so `wait: true` gates nothing.
- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
-  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
-  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
-  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
-  what was generated.
-
-Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
-gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
-(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
-not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
-other stage passed. **rc 3 is not green** and is not reported as such.
-
-## What I did NOT verify
-
-**Nothing about this component's runtime behaviour. It has never run.**
-
- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
-  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
-  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
-  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
-  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
-  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
-  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
-  admission.
- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
-  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
-  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
-  passing as uid 0 is weaker and must not be accepted as proof.
- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
-  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
-  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
-  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
-  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
-  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
-  the pods fail to start with an SELinux error, that line is the first suspect.
- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
-  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
-  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
-  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
-  no measurement of this plugin's footprint on these nodes. Worth a glance given
-  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
-  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
-  both overlays render the same base — but it is not a thing I could test.
- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
-  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
-  unconfirmed-by-me.
@@ -1,211 +0,0 @@
---
-date: 2026-10-01
-branch: feat/quicksync-via-device-plugin
-title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
---
-
-## What changed
-
-The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
-moved** — that was the whole design bet when this field was made an enum of capability classes
-instead of a device path, and this is the change that collects on it.
-
-**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
-**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
-in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
-
-| attempt | result |
-|---|---|
-| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-
-`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
-**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
-`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
-`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
-operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
-infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
-"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
-`restrict-host-namespaces` is Enforce on app namespaces).
-
-**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
-(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
-the **app container only** (never a sidecar):
-
-```yaml
-resources:
-  limits:
-    gpu.intel.com/i915: 1
-```
-
-and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
-`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
-a requests-only extended resource is invalid. This is the **only** app-container `resources` render
-path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
-along the way.
-
-**Self-tests** — the three existing ones updated plus one new:
- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
-  a stray CPU/memory addition through this field fails loudly;
- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
-  assertion is kept verbatim;
- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
-
-**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
-The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
-the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
-and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
-
-**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
-`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
-("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
-points at the contract, and keeps the measured facts that remain true (device on all three nodes,
-`i915` bound, the cont-init hook / `supplementalGroups` explanation).
-
-**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
-`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
-decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
-recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
-`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
-hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
-settable by any env var) is unchanged.
-
-**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
-added. Nothing else in `apps/` changed.
-
-### The one real regression, stated rather than sold
-
-This trades failure modes, and the new one is **worse in blast radius**:
-
-| | device/resource unavailable ⇒ |
-|---|---|
-| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
-| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
-
-It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
-thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
-schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
-yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
-all.
-
-**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
-and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
-the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
-someone will need at 2am.
-
-## What I verified
-
-**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
-DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
-(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
-`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
-`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
-themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
-PM, not by me** — see "What I did NOT verify".
-
-**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
-(a test that cannot fail is not a test):
-
-| mutant | caught by |
-|---|---|
-| `requests` instead of `limits` | exact-resources assertion |
-| emission dropped entirely | exact-resources assertion |
-| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
-| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
-| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
-| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
-| quantity as the string `"1"` | exact-resources assertion |
-| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
-
-**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
-else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
-hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
-`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
-from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
-diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
-
-**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
-filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
-`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
-
-**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
-not calling it green.** Run from this worktree in the pinned toolbox
-(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
-read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
-
-| stage | result |
-|---|---|
-| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
-| `decisions-log` | green |
-| `lint` | green |
-| `l0` | green |
-| `render-golden` | green |
-| `l1` | green |
-| `approval-policy` | green (`event='none'`, N/A off a PR event) |
-| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
-| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
-
-Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
-HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
-coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
-"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
-
-## What I did NOT verify
-
-**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
-it.** A green gate proves the **manifest renders**, nothing more.
-
- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
-  and Flux reconciles.** The whole case for this change is that the device plugin's
-  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
-  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
-  deliver it here, this change buys nothing and costs schedulability.
- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
-  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
-  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
-  no matter how the device arrives.
- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
-  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
-  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
-  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
-  that it had not been observed stable for any length of time.
- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
-  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
-  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
-  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
-  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
-  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
-  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
-  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
-  floating-image caveat.
-
-### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
-
-1. Merge the PR once the gate reports. Confirm Flux applied it — compare
-   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
-   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
-2. Confirm Plex **scheduled** (this is the new risk):
-   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
-   `Pending` ⇒ runbook "Plex will not schedule".
-3. Confirm it consumed the resource:
-   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
-   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
-4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
-   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
-   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
-   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
-   and uid 0 was one of the two `EPERM`s that killed the hostPath.
-5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
-   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
-   `Transcode (hw)` is the proof; plain `Transcode` is software.
-
-Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
-reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
-re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
-escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.
@@ -1,225 +0,0 @@
-# platform/intel-gpu-plugin
-
-**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
-[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
-pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
-*request* a GPU instead of hoping a `hostPath` is enough.
-
- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
-  consumer of an Intel iGPU.
- **Tier:** its **own** Flux Kustomization
-  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
-  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
-  *Placement* below.
- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
-  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
-  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
-  to the *local kubelet* over a unix socket.
-
-## What this component owns
-
-| File | Purpose |
-|---|---|
-| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
-| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
-| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
-
-## The problem this fixes
-
-Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
-the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
-
-```
-uid 0,    O_RDWR -> EPERM "Operation not permitted"
-uid 1000, O_RDWR -> EPERM "Operation not permitted"
-```
-
-`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
-
-1. root on the **host** opens the same device with `rc=0`;
-2. the node is unified `cgroup2fs`;
-3. the container's runc `config.json` reads
-   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
-   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
-   device.
-
-A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
-allow rule, because Kubernetes cannot infer one from a volume.
-
-**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
-(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
-per DRM node:
-
-```go
-specs = append(specs, pluginapi.DeviceSpec{
-    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
-})
-```
-
-and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
-cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
-also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
-under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
-2026-10-01). Correctly so.
-
-## Placement — why its own Kustomization
-
-The cheap option was one line in the shared `platform` tier. Neither that file nor a new
-`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
-have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
-already argued the case when Cilium was split out (read the comment in
-`platform/kustomization.yaml`):
-
-* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
-  "the GPU plugin is wedged" from "longhorn is slow".
-* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
-  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
-* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
-  rollout**. This is a 3-node DaemonSet rollout.
-
-The consequence is worse than average here because of what sits downstream: `platform` gates
-`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
-that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
-component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
-cause a *real outage*.
-
-The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
-`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
-path, and authors cannot self-approve. The approval was required either way.
-
-### `wait: true` is a health signal, not a gate
-
-This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
-because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
-on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
-it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
-genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
-answer to "did it come up on all three nodes", which is precisely the question that must be
-answered before the consumer PR may merge.
-
-Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
-`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
-power to stall every app, which is the thing this placement exists to prevent.
-
-## Deliberate non-choices
-
-**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
-controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
-three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
-already declared in git. For three identical nodes and one consumer that is strictly more blast
-radius for no gain.
-
-**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
-what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
-`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
-`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
-`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
-probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
-and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
-
-* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
-  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
-  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
-* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
-  `bypathFound = false`.
-
-**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
-card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
-
-* Plex is the only consumer today and wants one device;
-* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
-  it, so N containers get the same `renderD128` and contend for the same two media engines
-  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
-  "transcodes are mysteriously slow", which is neither;
-* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
-  right number will be known rather than guessed.
-
-**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
-630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
-something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
-rather than relying on the plugin's behaviour on such a node.
-
-**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
-`/dev/dri` char device and adds the `plex` user to its group — measured live as
-`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
-That mechanism is image-side and works identically on a plugin-injected device.
-
-## Resource name
-
-`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
-the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
-is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
-and a consumer's request would stop matching — worth knowing before that kernel arrives.
-
-## Versioning
-
-Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
-vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
-command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
-sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
-`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
-(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
-a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
-diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
-
-Vendored rather than referenced as a remote kustomize base for the same reason as
-`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
-fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
-then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
-which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
-was up.
-
-Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
-depName=...` immediately above the version literal), because `renovate.json` sets
-`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
-plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
-while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
-if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
-`renovate.json`, not a hand-edit.
-
-## Ordering — do not reorder the stack
-
-The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
-`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
-merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
-resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
-`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
-a real outage.
-
-## Post-merge verification
-
-Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
-the operator merges and Flux reconciles. Afterwards:
-
-```bash
-D=./infra/devcontainer/dev.sh
-
-# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
-$D run bash -c 'flux get kustomization intel-gpu-plugin'
-$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
-#    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
-
-# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
-$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
-  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
-#    want: three lines, each i915=1   (before this change: ABSENT on all three)
-
-# 3. If a node shows ABSENT, read that node's plugin logs.
-$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
-```
-
-### The decisive test
-
-Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
-the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
-through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
-request the resource:
-
-```bash
-$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
-```
-
-**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
-1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
@@ -1,117 +0,0 @@
---
-# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
-#   deployments/gpu_plugin/base
-#
-# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
-#   kubectl kustomize \
-#     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
-#     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
-#   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
-#   # and the volume set between minor releases before.
-#
-# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
-#   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
-# The raw upstream source file reviewed to produce it
-# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
-#   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
-# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
-# clean, so passing it through kustomize changed formatting only and no content.
-#
-# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
-# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
-# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
-# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
-# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
-# platform/external-snapshotter documents for the same reason.
-#
-# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
-# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
-# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
-# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
-# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
-# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
-# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
-# fragility; not this component's problem.)
-#
-# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
-# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
-# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
-# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
-# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  labels:
-    app: intel-gpu-plugin
-  name: intel-gpu-plugin
-spec:
-  selector:
-    matchLabels:
-      app: intel-gpu-plugin
-  template:
-    metadata:
-      labels:
-        app: intel-gpu-plugin
-    spec:
-      containers:
-      - env:
-        - name: NODE_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        - name: HOST_IP
-          valueFrom:
-            fieldRef:
-              fieldPath: status.hostIP
-        image: intel/intel-gpu-plugin:0.37.0
-        imagePullPolicy: IfNotPresent
-        name: intel-gpu-plugin
-        resources:
-          limits:
-            cpu: 100m
-            memory: 115Mi
-          requests:
-            cpu: 40m
-            memory: 45Mi
-        securityContext:
-          allowPrivilegeEscalation: false
-          capabilities:
-            drop:
-            - ALL
-          readOnlyRootFilesystem: true
-          seLinuxOptions:
-            type: container_device_plugin_t
-          seccompProfile:
-            type: RuntimeDefault
-        volumeMounts:
-        - mountPath: /dev/dri
-          name: devfs
-          readOnly: true
-        - mountPath: /sys/class/drm
-          name: sysfsdrm
-          readOnly: true
-        - mountPath: /var/lib/kubelet/device-plugins
-          name: kubeletsockets
-        - mountPath: /var/run/cdi
-          name: cdipath
-      nodeSelector:
-        kubernetes.io/arch: amd64
-      volumes:
-      - hostPath:
-          path: /dev/dri
-        name: devfs
-      - hostPath:
-          path: /sys/class/drm
-        name: sysfsdrm
-      - hostPath:
-          path: /var/lib/kubelet/device-plugins
-        name: kubeletsockets
-      - hostPath:
-          path: /var/run/cdi
-          type: DirectoryOrCreate
-        name: cdipath
-  updateStrategy:
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-    type: RollingUpdate
@@ -1,99 +0,0 @@
---
-# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
-#
-# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
-# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
-# Measured in the running pod on k3s-prod-2:
-#     uid 0,    O_RDWR -> EPERM "Operation not permitted"
-#     uid 1000, O_RDWR -> EPERM "Operation not permitted"
-# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
-# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
-# config.json carries one deny-all rule and nothing else:
-#     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
-#     linux.devices: None
-# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
-# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
-# Kubernetes cannot infer one from a volume.
-#
-# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
-# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
-# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
-# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
-# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
-# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
-# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
-#
-# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
-# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
-# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
-# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
-# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
-# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
-# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
-#
-# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
-# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
-# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
-# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
-# stat() demonstrably work today (that is how the Plex probe found the device before failing to
-# open it), so the plugin needs no cgroup grant of its own and no privilege.
-#   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
-#     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
-#     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
-#   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
-#   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
-#
-# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
-# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
-# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
-# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
-# request would stop matching — worth knowing before that kernel arrives.
-#
-# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
-# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
-# share one GPU. Deliberately not raised:
-#   * Plex is the only consumer and wants one device.
-#   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
-#     it — N containers get the same renderD128 and contend for the same two media engines
-#     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
-#     "transcodes are mysteriously slow", which is neither.
-#   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
-#     that point the right number will be known rather than guessed.
-# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
-# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
-# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
-#
-# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
-# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
-# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
-# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
-# or on the consumer would duplicate a thing that already works.
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: intel-gpu-plugin
-resources:
-  - namespace.yaml
-  - intel-gpu-plugin.yaml
-images:
-  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
-  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
-  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
-  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
-  # digest pinning is reserved for the conformance runner, where it is a security boundary.
-  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
-  # relies on containerd's docker.io default rather than saying so.
-  #
-  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
-  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
-  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
-  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
-  # `newTag` and not up in this header.
-  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
-  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
-  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
-  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
-  # renovate.json, not a hand-edit of this line.
-  - name: intel/intel-gpu-plugin
-    newName: docker.io/intel/intel-gpu-plugin
-    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
-    newTag: "0.37.0"
@@ -1,34 +0,0 @@
---
-# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
-#
-# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
-# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
-# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
-# reading the policies, not assumed:
-#   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
-#   * platform/kyverno/policies/require-egress-class-label.yaml
-#   * platform/kyverno/policies/require-auth-mode-label.yaml
-#   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
-#     app-namespace also gets a default-deny NetworkPolicy minted into it)
-# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
-# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
-# access] live in platform namespaces that do NOT carry the app-namespace label".
-#
-# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
-# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
-# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
-# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
-# in git, and because this component's correctness depends on the answer.
-#
-# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
-# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
-# placement is not a workaround for a policy — there is nothing to work around. It is here because
-# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
-# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
-# for a process that talks only to the local kubelet over a unix socket and never uses the network.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: intel-gpu-plugin
-  labels:
-    homelab.keithjackson.dev/app-namespace: "false"
@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
+
+## Observability: what this component can and cannot see (measured 2026-09-30)
+
+`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
+two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
+in the TSDB is:
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
+`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
+each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
+listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
+their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
+written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
+
+This matters because the gate's characteristic failure is invisible by construction: an
+unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
+against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
+where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
+spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
+into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
+
+`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
+`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
+`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
+a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
+one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
+and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
+
+**To close the gap for an app**, three things are needed together, on that app's own manifest:
+`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
+Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
+`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
+session lifetime — become writable rather than decorative.
+
+### Update 2026-10-01: the listener exists; the scrape path does not
+
+PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
+fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
+discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
+rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
+renderer.
+
+The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
+10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
+`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
+flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
+
+**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
+the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
+`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
+`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
+`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
+clause.
+
+So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
+and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
+and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
+ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
+(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
+cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
+
+Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
+present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
+would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
+as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
+fails it and nothing else.
@@ -0,0 +1,177 @@
+---
+# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
+#
+# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
+# failure modes of that shape are both silent: a label that does not survive `count by` makes the
+# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
+# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
+# visible by reading the expression, and neither is visible on a live cluster where the answer
+# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
+# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
+#
+# HOW TO RUN. The one repo-wide harness, by the companion convention:
+#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
+# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
+#
+# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
+# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
+# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
+# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
+# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
+# pass while the real expression collapsed differently.
+#
+# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
+# comparing it verbatim would couple this file to every wording change while proving nothing the
+# ALERTS series does not already prove.
+rule_files:
+  - auth-visibility-alerts.rules.yaml
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------------------------------
+  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
+  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
+  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
+  - interval: 1m
+    name: an unscraped gate namespace fires after 1h; the scraped one never does
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The shared instance: kube-state-metrics sees both replicas...
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
+        values: '1x130'
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
+        values: '1x130'
+      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
+        values: '1x130'
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
+        values: '1x130'
+    promql_expr_test:
+      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
+      # the hold away and turning every rollout into a page.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 59m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
+        eval_time: 59m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
+            value: 1
+      # Firing for jackett past the hold...
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # ...and never for the namespace that IS scraped. This is the assertion that catches an
+      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
+      # every gate namespace would pass the test above.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
+        eval_time: 65m
+        exp_samples: []
+      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
+      # that lost its `by` would collapse to a single unlabelled series.
+      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
+        eval_time: 65m
+        exp_samples:
+          - labels: '{namespace="jackett"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
+  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
+  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
+  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
+  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
+  - interval: 1m
+    name: the alert clears once the sidecar is actually scraped
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
+      # exist yet.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
+        values: '_x70 1x60'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
+        eval_time: 80m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
+  #
+  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
+  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
+  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
+  #
+  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
+  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
+  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
+  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
+  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
+  #    property that makes it worth having.
+  - interval: 1m
+    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
+      # scrape times out and `up` is published as 0 rather than going absent.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
+        values: '0x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
+      # mechanism the alert depends on, asserted directly so a future reader can see why the
+      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
+      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
+      # impossible to satisfy.)
+      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
+        eval_time: 65m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
+  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
+  - interval: 1m
+    name: a target that starts working and then fails re-fires
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
+        values: '1x250'
+      # Healthy for the first 60m, then every scrape fails for the rest.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
+        values: '1x60 0x180'
+    promql_expr_test:
+      # Quiet while it was working.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
+        eval_time: 55m
+        exp_samples: []
+      # Firing again an hour after it broke.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
+        eval_time: 125m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  #    rule would page for every app on the platform the day someone widened the matcher.
+  - interval: 1m
+    name: namespaces without a gate sidecar are never reported
+    input_series:
+      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
+        values: '1x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
+        eval_time: 65m
+        exp_samples: []
@@ -0,0 +1,137 @@
+---
+# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
+#
+# ============================================================================================
+# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
+# ============================================================================================
+# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
+# log in. The sidecar's own access log had the whole story:
+#
+#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
+#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
+#
+# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
+# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
+# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
+# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
+# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
+# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
+# gate had rejected every request ever made.
+#
+# ============================================================================================
+# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
+# ============================================================================================
+# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
+# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
+# reason is this file's real subject:
+#
+#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
+#
+#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
+#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
+#
+#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
+#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
+#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
+#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
+#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
+#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
+#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
+#
+#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
+#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
+#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
+#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
+#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
+#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
+#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
+#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
+#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
+#   temptation was refused here rather than ratcheted into a register.
+#
+#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
+#   path for any sidecar-gated app — each app gates itself — and its current pods export no
+#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
+#   the shared instance would not have fired on this incident either. Saying so is the point.
+#
+# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
+# That condition is measurable today, from two already-scraped metric families, and it is the
+# precondition for ever being able to alert on the incident itself. It self-clears the moment a
+# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
+# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
+# by-hand equivalent an operator uses in the meantime.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: oauth2-proxy-auth-visibility
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
+    app.kubernetes.io/name: oauth2-proxy
+spec:
+  groups:
+    - name: oauth2-proxy.auth-visibility
+      rules:
+        - alert: OAuth2ProxyGateUnobserved
+          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
+          # target for that container in that namespace.
+          #
+          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
+          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
+          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
+          #     container exports anything, which is exactly the property needed here.
+          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
+          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
+          #     without knowing what job name it will pick.
+          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
+          # inert for the reason it is reporting.
+          #
+          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
+          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
+          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
+          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
+          # the state the cluster is in; it is not a rule waiting for a hypothetical.
+          #
+          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
+          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
+          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
+          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
+          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
+          # branch's own disease — a green signal standing in for an absent one — reproduced
+          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
+          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
+          # the bare form would have silently cleared on the very next commit. `== 1` means
+          # "observed" requires a scrape that actually succeeds.
+          #
+          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
+          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
+          # down", this says "an authentication gate is unobserved", and during an incident those
+          # are different sentences. The overlap is the cheap half of the trade.
+          expr: |
+            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+              unless
+            count by (namespace) (up{container="oauth2-proxy"} == 1)
+          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
+          # discovered yet, and this condition is a standing architectural gap, never an
+          # emergency. Long enough that no deploy can produce it.
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
+            description: >-
+              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
+              Prometheus has NO scrape target for it, so every authentication decision that gate
+              makes — every redirect to Keycloak, every 403 on a group check, every session that
+              silently fails to hold — is invisible to metrics and to alerting. This is not a
+              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
+              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
+              seconds later, and the only record anywhere was a line in the sidecar's stdout.
+              No alert could have fired, because there is nothing to alert on. Until a scrape
+              exists the runbook's LogQL queries are the substitute; the fix is
+              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
+              ServiceMonitor, after which the auth-churn rule becomes writable.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,12 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
+  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
+  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
+  # tier, which is Ready before this tier's platform-config dependency completes.
+  - auth-visibility-alerts.yaml
+  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
+  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
+  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
+  # before concluding the targets should be up.
+  - podmonitor.yaml
@@ -0,0 +1,109 @@
+---
+# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
+#
+# ============================================================================================
+# WHY A PodMonitor AND NOT A ServiceMonitor
+# ============================================================================================
+# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
+# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
+# Services exposes 44180:
+#
+#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
+#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
+#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
+#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
+#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
+#
+# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
+# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
+# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
+#
+# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
+# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
+# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
+# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
+# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
+# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
+# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
+#
+# ============================================================================================
+# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# ============================================================================================
+# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
+# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+#
+#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
+#     GET /metrics -> http=200 bytes=10208
+#     # TYPE oauth2_proxy_requests_in_flight gauge
+#     # TYPE oauth2_proxy_requests_total counter
+#     # TYPE oauth2_proxy_response_duration_seconds histogram
+#     oauth2_proxy_requests_total{code="200"} 3
+#     oauth2_proxy_requests_total{code="302"} 2
+#
+#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
+#   Prometheus and so occupies the same policy position) to the jackett pod IP:
+#     http://10.0.0.189:44180/metrics   -> wget: download timed out
+#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
+#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
+#                                            has NO NetworkPolicy, which is the only reason the
+#                                            shared instance has ever been scrapable)
+#
+#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
+#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
+#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
+#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
+#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
+#
+# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
+# measured, documented state — NOT a regression introduced here, and NOT something to silence.
+# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
+# (see that file's header — this exact trap is why the `== 1` is there).
+#
+# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
+# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
+# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
+# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
+# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
+#
+# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
+#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
+#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
+#   apps-wave-1   dependsOn=['apps-wave-0']
+# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
+# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
+# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
+# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
+# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
+# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
+# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: oauth2-proxy-sidecars
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
+    app.kubernetes.io/name: oauth2-proxy
+    homelab.keithjackson.dev/purpose: metrics-scrape
+spec:
+  namespaceSelector:
+    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
+    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
+    any: true
+  selector:
+    matchLabels:
+      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
+      # 2026-10-01); absent from crafty, which has no metrics listener yet.
+      homelab.keithjackson.dev/auth-mode: forward-auth
+  podMetricsEndpoints:
+    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
+    # container, and a numeric port here would silently stop matching if the port were renumbered.
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
+      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
+      # the full interval.
+      scrapeTimeout: 10s
@@ -51,6 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
+
+The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
+because each step looked like the end of it.
+
+**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
+exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
+forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
+listener never started, and no ServiceMonitor or PodMonitor selected them.
+
+**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
+endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
+families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
+now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
+scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
+the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
+NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
+those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
+`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
+these apps it has to be rendered alongside each app's own policy rather than shipped from the
+oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
+namespaces existing yet.
+
+`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
+itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
+a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
+gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
+whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
+listener at all. See `oauth2-proxy-auth-failures`.

 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.

@@ -108,6 +138,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.
+- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
+  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
+  rather than a 401, the browser follows it silently against a live SSO session, and the call
+  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
+  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
+  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
+  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
+  so these logs (72h of them) are the only record this class of failure leaves.

 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -85,10 +85,17 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
+      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
+      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
+      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
+      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
      # presents as random session loss in ONE app's log and is only legible with both side by side,
      # so the page has to be reachable at 3am without going looking for it in git.
+      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
+      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
+      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
      - oauth2-proxy-session=oauth2-proxy-session.md
      # Operator procedure, not an alert page (no alert links to it — the
      # alert-runbooks-resolvable check treats that as advisory). It is served
@@ -0,0 +1,244 @@
+# oauth2-proxy auth failures — telling the three layers apart
+
+Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
+does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
+stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
+page is the diagnostic procedure, and it works whether or not the alert ever fired.
+
+## What this means here
+
+Every app behind Keycloak on this platform has **three** places a request can be refused, and
+from a browser they look identical — because two of the three answer with **HTTP 200**.
+
+| Layer | What refuses | What the browser sees |
+|---|---|---|
+| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
+| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
+| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
+
+**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
+redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
+JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
+Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
+dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
+nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
+failed.
+
+The worked example, jackett on 2026-09-30, straight from the sidecar log:
+
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+    ... ten of these at 15:30:53
+    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
+                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
+    GET "/UI/Dashboard"        200 44379
+    GET "/api/v2.0/indexers?_=..." 200 1501229
+    ... and 26 seconds later, inside that same 5-minute session:
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+
+A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
+signature of the session being **re-established over and over instead of held** — a cookie that
+another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
+down and not of the app being broken.
+
+**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
+*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
+them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
+targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
+the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
+for any of these apps. So there is still nothing to graph and nothing that could have alerted.
+**The logs below remain the only record that this class of failure happened.** Check which state
+you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
+
+## First commands
+
+`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
+with an explicit context.
+
+**1. Which namespaces have an unobserved gate** (this is what the alert measures):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
+```
+
+**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
+fixes and the alert deliberately does not distinguish them:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "up{container=\"oauth2-proxy\"}"'
+```
+
+No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
+`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
+value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s localhost:19090/api/v1/targets | python3 -c "
+import json,sys
+for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
+    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
+        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
+```
+
+`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
+endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
+bypasses policy entirely:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett port-forward \
+     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
+     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
+     http://127.0.0.1:14418/metrics'
+```
+
+`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
+2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
+`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
+
+**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
+and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
+`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
+     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
+```
+
+Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
+
+**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
+you can ask about the whole fleet and about a window that has already scrolled out of
+`kubectl logs`. Port-forward Loki and query it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
+   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
+     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
+```
+
+and the denominator, which is what makes the number mean something:
+
+```
+   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
+```
+
+**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
+lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
+ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
+
+| namespace | `No valid authentication` | `AuthSuccess` | ratio |
+|---|---|---|---|
+| crafty | 81 | 1 | 81:1 |
+| radarr | 67 | 1 | 67:1 |
+| jackett | 39 | 4 | ~10:1 |
+| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
+
+All three of those are the failure, not the baseline.
+
+**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
+jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
+session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
+The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
+them, **every single one was the same request**:
+
+    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
+
+That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
+reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
+is ever re-prompted; the tab just sits there looking like a working page that has stopped
+updating. **This is the entire failure class in one line**: a non-navigation request answered with
+a redirect, forever, with no error surfaced anywhere.
+
+So when you read a rejection count, **always group by request path before concluding anything**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
+     | grep -v "/ping" | grep "No valid authentication" -A 1 \
+     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
+```
+
+One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
+clears when the tab is closed. Many different paths from one session means the session itself is
+not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
+first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
+instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
+means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
+session-lifecycle conclusion can be drawn from that window at all.
+
+**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
+and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
+page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
+shape, and `runbooks/crafty-admin-credential.md` covers it.
+
+**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
+**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
+other's cookie and the loser logs exactly the `No valid authentication` line above. List what
+each gate is configured to emit (arguments only — never print a Secret's data):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
+     kubectl --context k3s-prod -n $ns get deploy $ns \
+       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
+     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
+```
+
+A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
+share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
+evicting each other, and the symptom is intermittent and depends on which app you visited last.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
+   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
+   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
+   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
+   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
+2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
+   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
+   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
+   gate logs the same line. Look for numbered cookie names in the log, and for a session that
+   never survives a single navigation.
+3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
+   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
+   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
+   jackett trace above.
+4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
+   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
+   die early, the refresh is failing, not missing.
+5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
+   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
+   for the `groups:[...]` list on the `[AuthSuccess]` line.
+6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
+
+## When to escalate or stop
+
+- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
+  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
+  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
+  reverts, and in the meantime you have an unreproducible cluster.
+- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
+  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
+  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
+  transcript here exactly this way.
+- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
+  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
+  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
+- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
+  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
+  report is older than three days, the evidence is gone and the only remaining move is to
+  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
+  the reason to close it rather than silence it.
@@ -5,24 +5,10 @@ not an incident page. It is indexed here because the feature it completes is **i
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
-rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
-mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
-`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
+rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.

-## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
-
-That is this feature's **one serious failure mode** and it **stalls the whole app tier**
-(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
-extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
-`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
-start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
-purpose.
-
-**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
-re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
-are in **"Plex will not schedule" → remedy B**.
-
 ## What this means here

 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
@@ -32,7 +18,7 @@ is in git**:

 | Half | Who does it | State |
 |---|---|---|
-| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
+| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |

 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
@@ -44,44 +30,10 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.

-### Why a resource request and not a `/dev/dri` hostPath
-
-Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
-will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
-**2026-10-01**: the device node was plainly **visible** in the container
-(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
-
-| attempt | result |
-|---|---|
-| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-
-`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
-from the outside and are not the same problem. Proven three ways: root on the **host** opened the
-same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
-`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
-deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
-device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
-
-Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
-does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
-`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
-
-The device plugin is what produces the allow rule — in the v0.37.0 source,
-`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
-into both the device injection and the matching cgroup allow.
-
-**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
-and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
-*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
-radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
-also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
-The one improvement is that the loss is no longer silent.
-
-What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
-hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
-renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
-advertises, which is the scheduler's doing and not a selector in this repo.
+What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
+not enable hardware transcoding, it does not prove the device can be *opened* from inside a
+non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
+of the three nodes and all three carry the device.

 ## Running the commands in this runbook

@@ -99,46 +51,15 @@ read of that file below greps a single named key — do not widen one.

 ## First commands

-**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
-```
-
-`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
-
-**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
-hostPath mounted" check, and it is now the first thing that can be wrong.
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod get nodes -o json \
-   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
-```
-
-Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
-node cannot run Plex. Then the DaemonSet behind it:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
-```
-
-Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
-symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
-
-**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
-GitOps half landed:
+**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
+     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
 ```

-A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
-mechanism that does not work. Compare
+`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare

 ```
 infra/devcontainer/dev.sh run bash -c \
@@ -148,29 +69,34 @@ infra/devcontainer/dev.sh run bash -c \

 against the merge commit before blaming anything else.

-**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
-of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.
+**2. Is the device actually visible inside the container?**

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```

-Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
-`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
-to exec into. Then the open, **as the app user**:
+Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
+here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
+had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
+transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```

-`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
-not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
-cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
-visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
-causes" #2.
+`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
+in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
+the outside and are not the same problem.

 **4. Is the toggle on?** Named key only — never the whole file:

@@ -246,32 +172,25 @@ When the dashboard still says plain `Transcode`:
   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
   investigating anything technical.

-2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
-   first-command #3 shows `renderD128` present and correctly owned, and the open returns
-   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
-   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
-   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
-   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
-   the pod really did consume the resource
+2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
+   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
+   character device on this k3s/containerd build **has not been established**; it can only be settled
+   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
+   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
+   correctly owned, step 3 returns `RW-OPEN-DENIED`.
+   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
+   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
+   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
+   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
+   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
+   **Enforce** mode and rejects it in app namespaces. Do not reach for it.

-   ```
-   infra/devcontainer/dev.sh run bash -c \
-     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
-   ```
-
-   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
-   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
-   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
-   that is the mechanism this replaced, and it was measured not to work.
-
-3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
-   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
-   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
-   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
-   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
-   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
-   **Fix the node.**
+3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
+   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
+   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
+   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
+   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
+   feature that is not load-bearing.

 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.
@@ -284,56 +203,6 @@ When the dashboard still says plain `Transcode`:
   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
   sessions show `(hw)` and others do not. Not a fault.

-## Plex will not schedule
-
-**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
-`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
-2am.
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
-```
-
-`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
-advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
-DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
-device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
-
-**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
-nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
-where to fix it. Check its Flux Kustomization reconciled:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
-```
-
-A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
-
-**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
-code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
-revert you):
-
-1. delete these two lines from `catalog/plex.yaml`:
-
-   ```yaml
-   hardware:
-     gpu: intel-quicksync
-   ```
-
-2. re-render the golden:
-
-   ```
-   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
-   ```
-
-   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
-3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
-
-Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
-`/config` PVC is affected either way — do **not** delete the PVC as part of this.
-
 ## The floating-image caveat

 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
@@ -374,12 +243,7 @@ hook exists.
  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
-  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
-  because root does not bypass the cgroup device gate.
- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
-  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
-  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
-  grants access; a volume cannot.
+  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
@@ -389,9 +253,9 @@ hook exists.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
-  confirmed) *under the current mechanism*. The next step is a platform change in
-  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
+- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
+  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
+  workload.

 ## Known gaps, deliberately not fixed here

@@ -415,8 +279,5 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
-reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
-Hardware transcoding has **still not been observed working**: the open test has not been re-run under
-the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
-toggle is still unset.
+(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
+of writing, and the Plex-side toggle was still unset.
@@ -165,115 +165,56 @@ platform must render, and cites the requirement each mapping satisfies. Changing
  (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
  ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
  over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
-  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
-  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
-  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
-  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
-  second route to the same one.
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:
+  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
+  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
+  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
+  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
+  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
+  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
+     render node read-write, so it is deliberately not `readOnly`).

-      resources:
-        limits:
-          gpu.intel.com/i915: 1
-
-  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
-  no privileged anything.
-  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
-    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
-    requests-only extended resource is rejected as invalid.
-  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
-    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
-    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
-    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
-  - This is the **only** app-container `resources` render path in the file, and it is deliberately
-    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
-    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
-    dict *exactly* so the attempt fails loudly.
- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
-  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
-  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
-  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
-  **could not be opened**:
-
-  | attempt | result |
-  |---|---|
-  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
-
-  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
-  identical from the outside and are not the same problem. Proven three ways: root on the **host**
-  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
-  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
-  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
-  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
-  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
-  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
-  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
-  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
-  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
-  into both the device injection and the matching cgroup allow.
- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
-  plainly rather than sold:
-
-  | | device/resource unavailable ⇒ |
-  |---|---|
-  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
-  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
-
-  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
-  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
-  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
-  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
-  transcoding working at all.
-  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
-    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
-    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
-    procedure, because it is what someone will need at 2am.
-  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
-    longer silent. There is still no metric and no alert (media apps here render no
-    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
-    "Plex is quietly software-transcoding" never was.
+  **Mounting the device is sufficient. That is the whole mapping.**
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
-  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
  move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
+  container on prod-2, 2026-09-30:** the image's own
  `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
  `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
-  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
-  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
-  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
-  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
-  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
-  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
-  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
-  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
-  own absence in `render.py --selftest`.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
+  `cont-init.d` completes before `services.d`, so the account database is already correct when the
+  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
+  **replaces** the process group list from `/etc/group` rather than appending to it:
+  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
+  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
+  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
+  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
  - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
    **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
    No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
-    established for an image that resolves the device group itself. A FUTURE consumer that runs
-    non-root and has no equivalent init hook would still need the group solved somehow — and the
-    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
-    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
-    The first such consumer must re-measure rather than assume this bullet covers it.
+  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
+    sufficient" was established for an image that resolves the device group itself. A FUTURE
+    consumer that runs non-root and has no equivalent init hook would still need the group solved
+    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
+    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
+    first such consumer must re-measure rather than assume this bullet covers it.
+- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
+  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
+  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
+  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
+  on such a node leaves nothing durable behind.
+  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
+    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
+    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
+    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
+    an operator observing it in the application's own dashboard.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
-  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
-  own justification — this field is about the device, not about where the pod lands; the extended
-  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
-  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
-  `resources` block carries the extended resource and nothing else).
- **Historical, kept so a reader recognises it rather than reintroduces it:**
-  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
-  what made a GPU-less node degrade to software transcoding instead of failing the mount and
-  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
-  the mechanism it protected never granted access in the first place.
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
-  (no `resources` key, no volume, no mount, no `securityContext`).
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
+  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
+  this field is about the device, not about where the pod lands); no resource requests/limits (there
+  is no app-container resource render path, and inventing one here would be scope creep).
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
  Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).

@@ -403,18 +344,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
   Derived from `exposure` alone; enforced on every gate run by
   `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
-   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
-   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
-   (never a sidecar) and no `securityContext` either. All directions are proven by
-   `render.py --selftest`, and each assertion is mutation-tested. The resource name
-   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
-   capability's implementation can be replaced without touching a single ServiceDefinition — which
-   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
-   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
-   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
-   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
-   to delete the `hardware:` block and re-render.
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
+   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
+   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
+   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
+   implementation can be replaced without touching a single ServiceDefinition.

 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
Author	SHA1	Message	Date
jakman85andClaude Opus 5	1e71b578d5	docs(obs): record why CI is red — approval-policy on the one conformance entry gate / gate (pull_request) Successful in 1m41s Details gate / conformance (pull_request) Has been skipped Details The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:12:19 -05:00
jakman85andClaude Opus 5	bce101d03f	feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it gate / gate (pull_request) Failing after 1m42s Details gate / conformance (pull_request) Has been skipped Details PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:56:07 -05:00
jakman85	ba79a1b59b	Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible # Conflicts: # runbooks/kustomization.yaml	2026-10-01 21:40:37 -05:00
jakman85andClaude Opus 5	fc7ec01628	feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics gate / gate (pull_request) Successful in 1m42s Details gate / conformance (pull_request) Has been skipped Details Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy."}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-30 10:55:19 -05:00