docs(decisions): handoff for the quicksync mechanism swap, including the regression it trades into

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/plex.yaml

@@ -137,14 +137,15 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
         - name: media
           mountPath: /media
           readOnly: true
-        - name: dri
-          mountPath: /dev/dri
       volumes:
       - name: config
         persistentVolumeClaim:
@@ -152,10 +153,6 @@ spec:
       - name: media
         persistentVolumeClaim:
           claimName: plex-media
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1

catalog/plex.yaml

@@ -81,46 +81,49 @@ env:
   # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
   # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
-# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
-# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
-# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
-# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
-# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
-# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
-# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
-# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
-# that is scheduler history rather than a requirement — this field deliberately renders no
-# nodeSelector, because placement is a separate concern with its own justification.
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
-# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
-# CONTAINER, 2026-09-30: the image's own
-# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
-# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
-# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
-# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
-# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
-# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
-# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
-# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
-# securityContext for this field precisely so nobody adds an inert one back later.
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
 #
-# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
-# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
-# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
-# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
-# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
-# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
-# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
-# dashboard.
-#
-# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
@@ -128,8 +131,7 @@ env:
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
-# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
-# device-cgroup failure mode to expect first.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
 hardware:
   gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)

catalog/render.py

@@ -166,11 +166,14 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
-# deliberately confined to this file. Swapping to the Intel GPU device plugin
-# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
-# nowhere else -- no catalog entry moves.
-DRI_VOLUME_NAME = "dri"
-DRI_DEVICE_DIR = "/dev/dri"
+# deliberately confined to this file. This file is the ONLY place the mechanism appears,
+# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
+# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
+#
+# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
+# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
+# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
+QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
 
 
 # =============================================================================
@@ -526,41 +529,59 @@ def r_deployment(s: dict) -> dict:
     # workload needs the host iGPU's render node for hardware video transcode. The SD says
     # only that; the wiring below is the platform's current mechanism for satisfying it.
     #
-    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
-    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
-    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
-    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
-    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
-    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
-    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
-    # directory the kubelet creates on such a node leaves nothing durable behind.
+    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
+    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
+    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
+    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
+    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
+    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
+    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
+    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
+    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
+    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
+    # because Kubernetes cannot infer one from a volume.
     #
-    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
+    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
+    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
+    # kubelet turns into both the device injection and the matching cgroup allow.
+    #
+    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
+    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
+    # requests-only extended resource is invalid. App container only, never a sidecar -- the
+    # oauth2-proxy sidecar does no transcoding.
+    #
+    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
+    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
+    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
+    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
+    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
+    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
+    # extended resource is a hard scheduling constraint, and it is also the only thing that
+    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
+    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
+    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
+    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
+    # unblocks. runbooks/plex-quicksync.md carries it.
     #
     # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
-    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
-    # 2026-09-30: the image's own cont-init hook
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2
+    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
     # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
     # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
     # container when none exists, and `usermod -a -G` the app user into it -- every start,
-    # nothing hardcoded. cont-init.d completes before services.d, so the account database
-    # s6 reads at exec time already carries the group. The service is then started with
-    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
-    # /etc/group rather than appending to it: `Plex Media Server` runs with
-    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
-    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
-    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
-    # group handling is the image's job and it does it correctly. A supplementalGroups entry
-    # here would be inert AND would mislead whoever debugs this next.
+    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
+    # is why this swap needs no group work either. cont-init.d completes before services.d,
+    # so the account database s6 reads at exec time already carries the group: measured live,
+    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
+    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
+    # REPLACES the process group list from /etc/group rather than appending to it, so
+    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
+    # before the app ever starts. A supplementalGroups entry here would be inert AND would
+    # mislead whoever debugs this next.
     hw_gpu = (s.get("hardware") or {}).get("gpu")
     if hw_gpu == "intel-quicksync":
-        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
-        volumes.append(
-            {
-                "name": DRI_VOLUME_NAME,
-                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
-            }
-        )
+        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
 
     if volume_mounts:
         container["volumeMounts"] = volume_mounts
@@ -2579,10 +2600,13 @@ def _selftest() -> int:
         "an SD without podSecurityContext must not gain a pod securityContext",
     )
 
-    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
-    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
-    #     assertions pin the MECHANISM that currently satisfies it -- including what it
-    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
+    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
+    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
+    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
+    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
+    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
+    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
+    #     the emission site), and the absence assertion is what stops it coming back.
     hw_docs = render(
         {
             "name": "hw",
@@ -2593,24 +2617,36 @@ def _selftest() -> int:
     hw_pod = next(
         d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
+    hw_app = hw_pod["containers"][0]
+    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
+    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
+    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
+    # CPU/memory resources feature through this field (that would be scope creep, and this
+    # field is about the DEVICE only).
     check(
-        {
-            "name": "dri",
-            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
-        }
-        in hw_pod.get("volumes", []),
-        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
-        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
+        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
+        "hardware.gpu=intel-quicksync must render resources.limits "
+        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
     )
-    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
+    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
+    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
     check(
-        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
-        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
+        not any(
+            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
+        )
+        and not any(
+            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
+            for m in hw_app.get("volumeMounts", [])
+        ),
+        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
+        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
+        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
     )
-    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
-    # own cont-init hook adds the app user to the device's group inside the container, and
-    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
-    # emission site). Asserting the absence is what stops someone adding an inert one back.
+    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
+    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
+    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
+    # (see the emission site). Asserting the absence is what stops someone adding an inert
+    # one back.
     check(
         "securityContext" not in hw_pod,
         "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
@@ -2618,10 +2654,12 @@ def _selftest() -> int:
     )
     # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
     # rejects `privileged: true` in an app namespace, and placement is a separate concern
-    # with its own justification -- this field is about the DEVICE only.
+    # with its own justification -- this field is about the DEVICE only. (The extended
+    # resource now constrains placement to nodes the plugin advertises, which is the
+    # scheduler's own doing, not a nodeSelector this renderer writes.)
     check(
         "privileged" not in hw_pod.get("securityContext", {})
-        and "securityContext" not in hw_pod["containers"][0]
+        and "securityContext" not in hw_app
         and "nodeSelector" not in hw_pod,
         "hardware.gpu must emit no privileged securityContext and no nodeSelector",
     )
@@ -2634,8 +2672,10 @@ def _selftest() -> int:
     check(
         "volumes" not in nohw_pod
         and "volumeMounts" not in nohw_pod["containers"][0]
+        and "resources" not in nohw_pod["containers"][0]
         and "securityContext" not in nohw_pod,
-        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
+        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
+        "securityContext",
     )
     # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
     # hardware field neither adds to it nor rewrites it.
@@ -2652,9 +2692,35 @@ def _selftest() -> int:
         if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
     check(
-        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
+        and hwm_pod["containers"][0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}},
         "hardware.gpu must leave the SD's own podSecurityContext untouched",
     )
+    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
+    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
+    hws_containers = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hws",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
+                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]["containers"]
+    check(
+        len(hws_containers) == 2
+        and hws_containers[0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}}
+        and "gpu.intel.com/i915"
+        not in hws_containers[1].get("resources", {}).get("limits", {}),
+        "hardware.gpu must put the i915 limit on the APP container only, never on the "
+        "oauth2-proxy sidecar",
+    )
 
     # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
     #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kustomization.yaml

@@ -12,6 +12,15 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,22 +129,6 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
-    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
-        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
-        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
-        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
-        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
-        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
-        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
-        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
-        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
-        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
-        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
-        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
-        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
-        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
-        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
-        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -1,289 +0,0 @@
----
-date: 2026-09-30
-branch: feat/oauth2-proxy-auth-failure-visible
-title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
----
-
-## What changed
-
-Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
-operator's report named the property that matters — *"reloading the page never fixes the issue and
-the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
-about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
-the alert rules. It is an absence of data.
-
-**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
-That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
-jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
-`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
-starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
-cluster selects them.
-
-So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
-a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
-exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
-permanently silent, and indistinguishable from health. It was not written.
-
-What was written instead alerts on the condition that made the incident invisible, from families
-that **are** scraped:
-
-- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
-
-      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-        unless
-      count by (namespace) (up{container="oauth2-proxy"})
-
-  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
-  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
-  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
-  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
-  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
-  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
-  clears per-namespace as each gate gains a real scrape.
-- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
-  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
-  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
-  prod, not invented.
-- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
-  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
-  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
-  `configMapGenerator` and listed in `runbooks/README.md`.
-- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
-  per-app sidecars, plus the measured reason.
-- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
-  three-part change that would close the gap.
-
-**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
-`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
-need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
-and — decisively — **it would not have fired on this incident**: the shared instance is not in the
-data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
-at all, having proxied nothing since they started (the counter exists in the TSDB only from an
-older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
-Shipping it would have looked like coverage of the reported failure and been none.
-
-## What I verified
-
-Everything below was measured against live prod through the pinned container, read-only. No
-`apply`, no `edit`, no `patch`, no `flux reconcile`.
-
-- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
-  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
-  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
-  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
-  `1`, `namespace="oauth2-proxy"`.
-- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
-  `oauth2-proxy` container, with its declared ports: the two shared replicas have
-  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
-  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
-  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
-  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
-- **No existing alert is inert *for this reason*, because no alert references these metrics at
-  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
-  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
-  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
-  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
-  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
-- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
-  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
-  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
-  observable from the IdP side.
-- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
-  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
-  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
-  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
-- **The alert expression fires against live data.** Run verbatim on prod:
-  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
-  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
-  radarr`, value 1 each; `oauth2-proxy` correctly absent.
-- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
-  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
-  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
-  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
-  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
-  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
-  runbook as the thing to compare against.
-- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
-  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
-  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
-- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
-  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
-  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
-  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
-  `runbook_url` points at exactly that name.
-- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
-  per-stage outcomes in the next section — read them there, not here.
-
-## What I did NOT verify
-
-- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
-  only in git on this branch. What is proven is that its expression returns the expected vector
-  against live prod data and that promtool agrees about its behaviour over synthetic series.
-  **Merging is what deploys it.**
-- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
-  any useful sense** — it describes a standing condition that has been true since the sidecars
-  were deployed, so it would have been firing before, during and after. That is deliberate and it
-  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
-  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
-  instead of discovering it during the investigation. **No alert on this platform could have
-  detected the incident itself, and none can today.**
-- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
-  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
-  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
-  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
-  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
-  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
-  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
-  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
-- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
-  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
-  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
-  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
-  directory from the one the sidecar writes), and the rendered config contains **no
-  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
-  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
-  caught this incident, and is the single highest-value follow-up here — **would evaluate and
-  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
-  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
-  inert rule ConfigMap to look productive.**
-- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
-  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
-- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
-  snapshot; anything older than three days is already gone. This is stated in the runbook because
-  it bounds what an operator can still investigate.
-- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
-  requirement touched by this branch is certified. See the exit code below — **rc 3 is
-  UNDETERMINED, not green.**
-
-### Follow-ups, in order of value
-
-1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
-   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
-   already being collected. The rule is one expression:
-   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
-   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
-2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
-   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
-   and it unlocks the ratio rule in Prometheus.
-3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
-   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
-   observable, and it belongs with whoever owns the sidecar args.
-
----
-
-# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
-
-Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
-and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
-conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
-
-## What changed
-
-**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
-`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
-them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
-and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
-forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
-not an app list, so a sixth forward-auth app needs no edit.
-
-**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
-This is a real bug fix, and it is the most important thing in this round.** A target that is
-discovered but failing still publishes `up`, with value **0**. The series exists, so
-`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
-would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
-reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
-whose scrape is dropped, so the bare form would have silently cleared on this very commit.
-
-**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
-target that works and then breaks must re-fire rather than latch cleared.
-
-**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
-all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
-and the alert header, each showing the two-step history, because each step looked like the end of it.
-The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
-
-**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
-endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
-it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
-succeeds, not the day a listener exists.
-
-## What I verified
-
-- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
-  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
-  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
-  being in `args` and `/metrics` returning a body are different claims; both now hold.
-- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
-  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
-  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
-  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
-  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
-  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
-  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
-- **No Service exposes 44180** on any of the five (enumerated).
-- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
-  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
-- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
-  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
-  `jackett` from this tier would apply today and fail on a cold rebuild.
-- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
-  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
-  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
-- **Gate:** see the next section for the exit code and skips.
-
-## What I did NOT verify
-
-- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
-  measured pod labels and port name, not observed — no target list has been read with it in place,
-  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
-  the documented state, not a regression.
-- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
-  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
-  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
-  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
-  the catalog/renderer that is in flight on another branch. **It needs an owner.**
-- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
-  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
-  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
-  measured with this object.
-- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
-  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
-  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
-- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
-  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
-- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
-
-### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
-
-After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
-toolbox run was rc 3 with nothing failed. Measured cause, singular:
-
-    sensitive paths touched:
-      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
-    approving reviews: (no reviews at all)        author=gitea-admin
-
-`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
-`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
-`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
-
-**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
-short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
-touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
-signal. That is a general trap, not specific to this change, and it is worth knowing before someone
-concludes a red PR means broken code.
-
-To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
-status check, so the required context stays red until the job runs again. `required_approvals=1`
-already applied to every PR here, so the human was needed regardless; the conformance entry only
-turns the tick red while that is pending.
-
-The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
-check fails without the declaration). I did not take it: the entry is the "declare the limit"
-mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
-`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
-reviewer should make it knowingly.

decisions/2026-10-01-intel-gpu-device-plugin.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-01
+branch: feat/intel-gpu-device-plugin
+title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
+---
+
+## What changed
+
+Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
+**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
+Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
+device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
+1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
+hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
+response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
+it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
+`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
+`app-namespace=true`.
+
+New files:
+
+- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
+  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
+  provenance header carrying the exact re-vendor command and both sha256s (generated body
+  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
+  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
+  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
+  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
+  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
+  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
+  kustomize base, for the other reason `external-snapshotter` records: a remote base is
+  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
+- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
+  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
+- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
+- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
+  non-choices, and the post-merge verification sequence.
+- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
+  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
+- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
+
+**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
+cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
+`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
+split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
+the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
+called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
+consequence is worse than average here, because `platform` gates `platform-config` gates
+`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
+would stall every app, so a component added to fix a **silent no-op** would have gained the ability
+to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
+`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
+of path and authors cannot self-approve.
+
+`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
+never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
+is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
+`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
+`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
+eleven of its components, which shrinks the cold-start window in which a node has not yet
+advertised the resource.
+
+`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
+`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
+making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
+genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
+rendered graph), `wait: true` gates nothing while turning
+`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
+answered before the consumer PR may merge.
+
+Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
+(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
+`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
+believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
+a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
+*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
+nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
+already adds the `plex` user to each device's group and that works identically on a
+plugin-injected device.
+
+**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
+are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
+later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
+on all three nodes** — a request for a resource no node advertises makes the Plex pod
+Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
+stall the whole app tier and turn today's silent no-op into a real outage.
+
+## What I verified
+
+Mechanism, read from the v0.37.0 source rather than assumed:
+
+- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
+  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
+  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
+  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
+  grant and no extra privilege — it does not need more than the upstream manifest gives it.
+- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
+  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
+  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
+- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
+  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
+  nodes have `i915` bound.
+- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
+  `false` and `setupLevelZeroService` returns early unless it is set.
+- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
+
+Image pin, verified against the registry:
+
+- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
+  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
+  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
+  latest upstream release (2026-09-16) and is not a prerelease.
+- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
+  while GitHub release tags do.
+
+Policy scope, read from the policies rather than assumed:
+
+- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
+  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
+  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
+  The new namespace is labelled `"false"`, so none of them match it.
+- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
+  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
+  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
+  collision.
+
+Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
+bind-mounted read-only — never copied into the worktree):
+
+- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
+- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
+  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
+  "NONE". That is the before-state this change is meant to move.
+- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
+  measured evidence that the device-plugin API and the default kubelet socket path
+  (`/var/lib/kubelet/device-plugins`) work on these nodes.
+
+Offline rendering (the main proof available to a PR):
+
+- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
+  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
+  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
+- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
+  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
+  (`clusters/prod`, `clusters/dev`) also render and carry it.
+- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
+  so `wait: true` gates nothing.
+- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
+  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
+  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
+  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
+  what was generated.
+
+Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
+gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
+(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
+not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
+other stage passed. **rc 3 is not green** and is not reported as such.
+
+## What I did NOT verify
+
+**Nothing about this component's runtime behaviour. It has never run.**
+
+- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
+  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
+  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
+- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
+  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
+  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
+- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
+  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
+  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
+  admission.
+- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
+  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
+  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
+  passing as uid 0 is weaker and must not be accepted as proof.
+- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
+  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
+  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
+  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
+- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
+  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
+  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
+  the pods fail to start with an SELinux error, that line is the first suspect.
+- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
+  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
+  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
+  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
+- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
+  no measurement of this plugin's footprint on these nodes. Worth a glance given
+  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
+- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
+  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
+  both overlays render the same base — but it is not a thing I could test.
+- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
+  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
+  unconfirmed-by-me.

decisions/2026-10-01-quicksync-via-device-plugin.md

@@ -0,0 +1,211 @@
+---
+date: 2026-10-01
+branch: feat/quicksync-via-device-plugin
+title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
+---
+
+## What changed
+
+The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
+moved** — that was the whole design bet when this field was made an enum of capability classes
+instead of a device path, and this is the change that collects on it.
+
+**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
+**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
+in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
+**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
+operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
+infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
+"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces).
+
+**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
+(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
+the **app container only** (never a sidecar):
+
+```yaml
+resources:
+  limits:
+    gpu.intel.com/i915: 1
+```
+
+and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
+`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
+a requests-only extended resource is invalid. This is the **only** app-container `resources` render
+path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
+along the way.
+
+**Self-tests** — the three existing ones updated plus one new:
+- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
+  a stray CPU/memory addition through this field fails loudly;
+- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
+- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
+- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
+  assertion is kept verbatim;
+- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
+
+**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
+The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
+the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
+and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
+
+**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
+`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
+("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
+points at the contract, and keeps the measured facts that remain true (device on all three nodes,
+`i915` bound, the cont-init hook / `supplementalGroups` explanation).
+
+**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
+`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
+decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
+recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
+`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
+hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
+settable by any env var) is unchanged.
+
+**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
+added. Nothing else in `apps/` changed.
+
+### The one real regression, stated rather than sold
+
+This trades failure modes, and the new one is **worse in blast radius**:
+
+| | device/resource unavailable ⇒ |
+|---|---|
+| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
+| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
+thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
+schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
+yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
+all.
+
+**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
+and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
+the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
+someone will need at 2am.
+
+## What I verified
+
+**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
+DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
+(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
+`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
+`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
+themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
+PM, not by me** — see "What I did NOT verify".
+
+**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
+(a test that cannot fail is not a test):
+
+| mutant | caught by |
+|---|---|
+| `requests` instead of `limits` | exact-resources assertion |
+| emission dropped entirely | exact-resources assertion |
+| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
+| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
+| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
+| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
+| quantity as the string `"1"` | exact-resources assertion |
+| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
+
+**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
+--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
+else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
+hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
+`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
+from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
+diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
+
+**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
+filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
+`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
+
+**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
+not calling it green.** Run from this worktree in the pinned toolbox
+(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
+read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
+
+| stage | result |
+|---|---|
+| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
+| `decisions-log` | green |
+| `lint` | green |
+| `l0` | green |
+| `render-golden` | green |
+| `l1` | green |
+| `approval-policy` | green (`event='none'`, N/A off a PR event) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
+
+Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
+HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
+coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
+"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
+
+## What I did NOT verify
+
+**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
+it.** A green gate proves the **manifest renders**, nothing more.
+
+- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
+  and Flux reconciles.** The whole case for this change is that the device plugin's
+  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
+  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
+  deliver it here, this change buys nothing and costs schedulability.
+- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
+  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
+  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
+  no matter how the device arrives.
+- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
+  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
+  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
+  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
+  that it had not been observed stable for any length of time.
+- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
+  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
+  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
+  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
+- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
+  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
+- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
+  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
+  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
+- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
+  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
+  floating-image caveat.
+
+### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
+
+1. Merge the PR once the gate reports. Confirm Flux applied it — compare
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
+   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
+2. Confirm Plex **scheduled** (this is the new risk):
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
+   `Pending` ⇒ runbook "Plex will not schedule".
+3. Confirm it consumed the resource:
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
+   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
+4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
+   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
+   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
+   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
+   and uid 0 was one of the two `EPERM`s that killed the hostPath.
+5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
+   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
+   `Transcode (hw)` is the proof; plain `Transcode` is software.
+
+Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
+reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
+re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
+escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.

platform/intel-gpu-plugin/README.md

@@ -0,0 +1,225 @@
+# platform/intel-gpu-plugin
+
+**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
+[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
+pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
+*request* a GPU instead of hoping a `hostPath` is enough.
+
+- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
+  consumer of an Intel iGPU.
+- **Tier:** its **own** Flux Kustomization
+  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
+  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
+  *Placement* below.
+- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
+  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
+  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
+- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
+  to the *local kubelet* over a unix socket.
+
+## What this component owns
+
+| File | Purpose |
+|---|---|
+| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
+| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
+| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
+
+## The problem this fixes
+
+Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
+the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
+
+```
+uid 0,    O_RDWR -> EPERM "Operation not permitted"
+uid 1000, O_RDWR -> EPERM "Operation not permitted"
+```
+
+`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
+
+1. root on the **host** opens the same device with `rc=0`;
+2. the node is unified `cgroup2fs`;
+3. the container's runc `config.json` reads
+   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
+   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
+   device.
+
+A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
+allow rule, because Kubernetes cannot infer one from a volume.
+
+**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
+(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
+per DRM node:
+
+```go
+specs = append(specs, pluginapi.DeviceSpec{
+    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
+})
+```
+
+and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
+cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
+also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
+under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
+2026-10-01). Correctly so.
+
+## Placement — why its own Kustomization
+
+The cheap option was one line in the shared `platform` tier. Neither that file nor a new
+`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
+have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
+already argued the case when Cilium was split out (read the comment in
+`platform/kustomization.yaml`):
+
+* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
+  "the GPU plugin is wedged" from "longhorn is slow".
+* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
+  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
+* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
+  rollout**. This is a 3-node DaemonSet rollout.
+
+The consequence is worse than average here because of what sits downstream: `platform` gates
+`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
+that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
+component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
+cause a *real outage*.
+
+The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
+`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
+path, and authors cannot self-approve. The approval was required either way.
+
+### `wait: true` is a health signal, not a gate
+
+This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
+because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
+on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
+it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
+genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
+answer to "did it come up on all three nodes", which is precisely the question that must be
+answered before the consumer PR may merge.
+
+Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
+`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
+power to stall every app, which is the thing this placement exists to prevent.
+
+## Deliberate non-choices
+
+**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
+controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
+three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
+already declared in git. For three identical nodes and one consumer that is strictly more blast
+radius for no gain.
+
+**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
+what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
+`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
+`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
+`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
+probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
+and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
+
+* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
+  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
+  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
+* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
+  `bypathFound = false`.
+
+**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
+card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
+
+* Plex is the only consumer today and wants one device;
+* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
+  it, so N containers get the same `renderD128` and contend for the same two media engines
+  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
+  "transcodes are mysteriously slow", which is neither;
+* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
+  right number will be known rather than guessed.
+
+**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
+630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
+something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
+rather than relying on the plugin's behaviour on such a node.
+
+**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
+`/dev/dri` char device and adds the `plex` user to its group — measured live as
+`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
+That mechanism is image-side and works identically on a plugin-injected device.
+
+## Resource name
+
+`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
+the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
+is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
+and a consumer's request would stop matching — worth knowing before that kernel arrives.
+
+## Versioning
+
+Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
+vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
+command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
+sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
+`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
+(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
+a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
+diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
+
+Vendored rather than referenced as a remote kustomize base for the same reason as
+`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
+fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
+then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
+which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
+was up.
+
+Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
+depName=...` immediately above the version literal), because `renovate.json` sets
+`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
+plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
+while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
+if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
+`renovate.json`, not a hand-edit.
+
+## Ordering — do not reorder the stack
+
+The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
+`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
+merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
+resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
+`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
+a real outage.
+
+## Post-merge verification
+
+Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
+the operator merges and Flux reconciles. Afterwards:
+
+```bash
+D=./infra/devcontainer/dev.sh
+
+# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
+$D run bash -c 'flux get kustomization intel-gpu-plugin'
+$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
+#    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
+
+# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
+$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
+  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+#    want: three lines, each i915=1   (before this change: ABSENT on all three)
+
+# 3. If a node shows ABSENT, read that node's plugin logs.
+$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
+```
+
+### The decisive test
+
+Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
+the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
+through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
+request the resource:
+
+```bash
+$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
+```
+
+**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
+1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.

platform/intel-gpu-plugin/intel-gpu-plugin.yaml

@@ -0,0 +1,117 @@
+---
+# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
+#   deployments/gpu_plugin/base
+#
+# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
+#   kubectl kustomize \
+#     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
+#     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
+#   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
+#   # and the volume set between minor releases before.
+#
+# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
+#   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
+# The raw upstream source file reviewed to produce it
+# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
+#   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
+# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
+# clean, so passing it through kustomize changed formatting only and no content.
+#
+# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
+# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
+# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
+# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
+# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
+# platform/external-snapshotter documents for the same reason.
+#
+# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
+# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
+# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
+# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
+# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
+# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
+# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
+# fragility; not this component's problem.)
+#
+# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
+# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
+# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
+# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
+# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    app: intel-gpu-plugin
+  name: intel-gpu-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-gpu-plugin
+  template:
+    metadata:
+      labels:
+        app: intel-gpu-plugin
+    spec:
+      containers:
+      - env:
+        - name: NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        - name: HOST_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.hostIP
+        image: intel/intel-gpu-plugin:0.37.0
+        imagePullPolicy: IfNotPresent
+        name: intel-gpu-plugin
+        resources:
+          limits:
+            cpu: 100m
+            memory: 115Mi
+          requests:
+            cpu: 40m
+            memory: 45Mi
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop:
+            - ALL
+          readOnlyRootFilesystem: true
+          seLinuxOptions:
+            type: container_device_plugin_t
+          seccompProfile:
+            type: RuntimeDefault
+        volumeMounts:
+        - mountPath: /dev/dri
+          name: devfs
+          readOnly: true
+        - mountPath: /sys/class/drm
+          name: sysfsdrm
+          readOnly: true
+        - mountPath: /var/lib/kubelet/device-plugins
+          name: kubeletsockets
+        - mountPath: /var/run/cdi
+          name: cdipath
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      volumes:
+      - hostPath:
+          path: /dev/dri
+        name: devfs
+      - hostPath:
+          path: /sys/class/drm
+        name: sysfsdrm
+      - hostPath:
+          path: /var/lib/kubelet/device-plugins
+        name: kubeletsockets
+      - hostPath:
+          path: /var/run/cdi
+          type: DirectoryOrCreate
+        name: cdipath
+  updateStrategy:
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+    type: RollingUpdate

platform/intel-gpu-plugin/kustomization.yaml

@@ -0,0 +1,99 @@
+---
+# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
+#
+# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
+# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
+# Measured in the running pod on k3s-prod-2:
+#     uid 0,    O_RDWR -> EPERM "Operation not permitted"
+#     uid 1000, O_RDWR -> EPERM "Operation not permitted"
+# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
+# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
+# config.json carries one deny-all rule and nothing else:
+#     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
+#     linux.devices: None
+# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
+# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
+# Kubernetes cannot infer one from a volume.
+#
+# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
+# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
+# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
+# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
+# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
+# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
+# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
+#
+# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
+# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
+# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
+# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
+# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
+# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
+# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
+#
+# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
+# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
+# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
+# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
+# stat() demonstrably work today (that is how the Plex probe found the device before failing to
+# open it), so the plugin needs no cgroup grant of its own and no privilege.
+#   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
+#     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
+#     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
+#   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
+#   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
+#
+# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
+# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
+# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
+# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
+# request would stop matching — worth knowing before that kernel arrives.
+#
+# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
+# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
+# share one GPU. Deliberately not raised:
+#   * Plex is the only consumer and wants one device.
+#   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
+#     it — N containers get the same renderD128 and contend for the same two media engines
+#     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
+#     "transcodes are mysteriously slow", which is neither.
+#   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
+#     that point the right number will be known rather than guessed.
+# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
+# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
+# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
+#
+# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
+# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
+# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
+# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
+# or on the consumer would duplicate a thing that already works.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: intel-gpu-plugin
+resources:
+  - namespace.yaml
+  - intel-gpu-plugin.yaml
+images:
+  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
+  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
+  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
+  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
+  # digest pinning is reserved for the conformance runner, where it is a security boundary.
+  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
+  # relies on containerd's docker.io default rather than saying so.
+  #
+  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
+  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
+  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
+  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
+  # `newTag` and not up in this header.
+  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
+  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
+  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
+  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
+  # renovate.json, not a hand-edit of this line.
+  - name: intel/intel-gpu-plugin
+    newName: docker.io/intel/intel-gpu-plugin
+    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
+    newTag: "0.37.0"

platform/intel-gpu-plugin/namespace.yaml

@@ -0,0 +1,34 @@
+---
+# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
+#
+# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
+# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
+# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
+# reading the policies, not assumed:
+#   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
+#   * platform/kyverno/policies/require-egress-class-label.yaml
+#   * platform/kyverno/policies/require-auth-mode-label.yaml
+#   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
+#     app-namespace also gets a default-deny NetworkPolicy minted into it)
+# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
+# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
+# access] live in platform namespaces that do NOT carry the app-namespace label".
+#
+# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
+# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
+# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
+# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
+# in git, and because this component's correctness depends on the answer.
+#
+# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
+# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
+# placement is not a workaround for a policy — there is nothing to work around. It is here because
+# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
+# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
+# for a process that talks only to the local kubelet over a unix socket and never uses the network.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: intel-gpu-plugin
+  labels:
+    homelab.keithjackson.dev/app-namespace: "false"

platform/oauth2-proxy/README.md

@@ -88,72 +88,3 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
    protected UIs).
-
-## Observability: what this component can and cannot see (measured 2026-09-30)
-
-`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
-two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
-in the TSDB is:
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
-`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
-each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
-listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
-their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
-written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
-
-This matters because the gate's characteristic failure is invisible by construction: an
-unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
-against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
-where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
-spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
-into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
-
-`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
-`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
-`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
-a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
-one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
-and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
-
-**To close the gap for an app**, three things are needed together, on that app's own manifest:
-`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
-Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
-`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
-session lifetime — become writable rather than decorative.
-
-### Update 2026-10-01: the listener exists; the scrape path does not
-
-PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
-fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
-discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
-rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
-renderer.
-
-The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
-10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
-`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
-flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
-
-**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
-the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
-`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
-`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
-`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
-clause.
-
-So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
-and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
-and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
-ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
-(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
-cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
-
-Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
-present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
-would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
-as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
-fails it and nothing else.

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -1,177 +0,0 @@
----
-# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
-#
-# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
-# failure modes of that shape are both silent: a label that does not survive `count by` makes the
-# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
-# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
-# visible by reading the expression, and neither is visible on a live cluster where the answer
-# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
-# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
-#
-# HOW TO RUN. The one repo-wide harness, by the companion convention:
-#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
-# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
-#
-# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
-# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
-# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
-# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
-# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
-# pass while the real expression collapsed differently.
-#
-# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
-# comparing it verbatim would couple this file to every wording change while proving nothing the
-# ALERTS series does not already prove.
-rule_files:
-  - auth-visibility-alerts.rules.yaml
-evaluation_interval: 1m
-
-tests:
-  # ---------------------------------------------------------------------------------------------
-  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
-  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
-  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
-  - interval: 1m
-    name: an unscraped gate namespace fires after 1h; the scraped one never does
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # The shared instance: kube-state-metrics sees both replicas...
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
-        values: '1x130'
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
-        values: '1x130'
-      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
-        values: '1x130'
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
-        values: '1x130'
-    promql_expr_test:
-      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
-      # the hold away and turning every rollout into a page.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 59m
-        exp_samples: []
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
-        eval_time: 59m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
-            value: 1
-      # Firing for jackett past the hold...
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      # ...and never for the namespace that IS scraped. This is the assertion that catches an
-      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
-      # every gate namespace would pass the test above.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
-        eval_time: 65m
-        exp_samples: []
-      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
-      # that lost its `by` would collapse to a single unlabelled series.
-      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
-        eval_time: 65m
-        exp_samples:
-          - labels: '{namespace="jackett"}'
-            value: 1
-
-  # ---------------------------------------------------------------------------------------------
-  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
-  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
-  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
-  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
-  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
-  - interval: 1m
-    name: the alert clears once the sidecar is actually scraped
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
-      # exist yet.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
-        values: '_x70 1x60'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
-        eval_time: 80m
-        exp_samples: []
-
-  # ---------------------------------------------------------------------------------------------
-  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
-  #
-  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
-  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
-  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
-  #
-  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
-  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
-  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
-  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
-  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
-  #    property that makes it worth having.
-  - interval: 1m
-    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
-      # scrape times out and `up` is published as 0 rather than going absent.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
-        values: '0x130'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
-      # mechanism the alert depends on, asserted directly so a future reader can see why the
-      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
-      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
-      # impossible to satisfy.)
-      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
-        eval_time: 65m
-        exp_samples: []
-
-  # ---------------------------------------------------------------------------------------------
-  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
-  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
-  - interval: 1m
-    name: a target that starts working and then fails re-fires
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
-        values: '1x250'
-      # Healthy for the first 60m, then every scrape fails for the rest.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
-        values: '1x60 0x180'
-    promql_expr_test:
-      # Quiet while it was working.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
-        eval_time: 55m
-        exp_samples: []
-      # Firing again an hour after it broke.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
-        eval_time: 125m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
-            value: 1
-
-  # ---------------------------------------------------------------------------------------------
-  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
-  #    rule would page for every app on the platform the day someone widened the matcher.
-  - interval: 1m
-    name: namespaces without a gate sidecar are never reported
-    input_series:
-      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
-        values: '1x130'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
-        eval_time: 65m
-        exp_samples: []

platform/oauth2-proxy/auth-visibility-alerts.yaml

@@ -1,137 +0,0 @@
----
-# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
-#
-# ============================================================================================
-# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
-# ============================================================================================
-# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
-# log in. The sidecar's own access log had the whole story:
-#
-#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
-#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
-#
-# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
-# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
-# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
-# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
-# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
-# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
-# gate had rejected every request ever made.
-#
-# ============================================================================================
-# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
-# ============================================================================================
-# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
-# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
-# reason is this file's real subject:
-#
-#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
-#
-#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
-#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
-#
-#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
-#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
-#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
-#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
-#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
-#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
-#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
-#
-#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
-#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
-#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
-#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
-#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
-#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
-#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
-#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
-#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
-#   temptation was refused here rather than ratcheted into a register.
-#
-#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
-#   path for any sidecar-gated app — each app gates itself — and its current pods export no
-#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
-#   the shared instance would not have fired on this incident either. Saying so is the point.
-#
-# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
-# That condition is measurable today, from two already-scraped metric families, and it is the
-# precondition for ever being able to alert on the incident itself. It self-clears the moment a
-# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
-# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
-# by-hand equivalent an operator uses in the meantime.
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: oauth2-proxy-auth-visibility
-  namespace: oauth2-proxy
-  labels:
-    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
-    app.kubernetes.io/name: oauth2-proxy
-spec:
-  groups:
-    - name: oauth2-proxy.auth-visibility
-      rules:
-        - alert: OAuth2ProxyGateUnobserved
-          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
-          # target for that container in that namespace.
-          #
-          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
-          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
-          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
-          #     container exports anything, which is exactly the property needed here.
-          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
-          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
-          #     without knowing what job name it will pick.
-          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
-          # inert for the reason it is reporting.
-          #
-          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
-          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
-          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
-          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
-          # the state the cluster is in; it is not a rule waiting for a hypothetical.
-          #
-          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
-          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
-          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
-          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
-          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
-          # branch's own disease — a green signal standing in for an absent one — reproduced
-          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
-          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
-          # the bare form would have silently cleared on the very next commit. `== 1` means
-          # "observed" requires a scrape that actually succeeds.
-          #
-          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
-          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
-          # down", this says "an authentication gate is unobserved", and during an incident those
-          # are different sentences. The overlap is the cheap half of the trade.
-          expr: |
-            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-              unless
-            count by (namespace) (up{container="oauth2-proxy"} == 1)
-          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
-          # discovered yet, and this condition is a standing architectural gap, never an
-          # emergency. Long enough that no deploy can produce it.
-          for: 1h
-          labels:
-            severity: warning
-          annotations:
-            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
-            description: >-
-              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
-              Prometheus has NO scrape target for it, so every authentication decision that gate
-              makes — every redirect to Keycloak, every 403 on a group check, every session that
-              silently fails to hold — is invisible to metrics and to alerting. This is not a
-              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
-              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
-              seconds later, and the only record anywhere was a line in the sidecar's stdout.
-              No alert could have fired, because there is nothing to alert on. Until a scrape
-              exists the runbook's LogQL queries are the substitute; the fix is
-              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
-              ServiceMonitor, after which the auth-churn rule becomes writable.
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"

platform/oauth2-proxy/kustomization.yaml

@@ -14,12 +14,3 @@ resources:
   - oauth2-proxy-credentials.sops.yaml
   - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
   - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
-  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
-  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
-  # tier, which is Ready before this tier's platform-config dependency completes.
-  - auth-visibility-alerts.yaml
-  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
-  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
-  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
-  # before concluding the targets should be up.
-  - podmonitor.yaml

platform/oauth2-proxy/podmonitor.yaml

@@ -1,109 +0,0 @@
----
-# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
-#
-# ============================================================================================
-# WHY A PodMonitor AND NOT A ServiceMonitor
-# ============================================================================================
-# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
-# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
-# Services exposes 44180:
-#
-#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
-#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
-#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
-#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
-#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
-#
-# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
-# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
-# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
-#
-# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
-# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
-# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
-# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
-# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
-# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
-# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
-#
-# ============================================================================================
-# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
-# ============================================================================================
-# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
-# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
-#
-#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
-#     GET /metrics -> http=200 bytes=10208
-#     # TYPE oauth2_proxy_requests_in_flight gauge
-#     # TYPE oauth2_proxy_requests_total counter
-#     # TYPE oauth2_proxy_response_duration_seconds histogram
-#     oauth2_proxy_requests_total{code="200"} 3
-#     oauth2_proxy_requests_total{code="302"} 2
-#
-#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
-#   Prometheus and so occupies the same policy position) to the jackett pod IP:
-#     http://10.0.0.189:44180/metrics   -> wget: download timed out
-#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
-#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
-#                                            has NO NetworkPolicy, which is the only reason the
-#                                            shared instance has ever been scrapable)
-#
-#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
-#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
-#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
-#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
-#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
-#
-# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
-# measured, documented state — NOT a regression introduced here, and NOT something to silence.
-# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
-# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
-# (see that file's header — this exact trap is why the `== 1` is there).
-#
-# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
-# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
-# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
-# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
-# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
-#
-# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
-# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
-#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
-#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
-#   apps-wave-1   dependsOn=['apps-wave-0']
-# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
-# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
-# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
-# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
-# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
-# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
-# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
-apiVersion: monitoring.coreos.com/v1
-kind: PodMonitor
-metadata:
-  name: oauth2-proxy-sidecars
-  namespace: oauth2-proxy
-  labels:
-    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
-    app.kubernetes.io/name: oauth2-proxy
-    homelab.keithjackson.dev/purpose: metrics-scrape
-spec:
-  namespaceSelector:
-    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
-    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
-    any: true
-  selector:
-    matchLabels:
-      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
-      # 2026-10-01); absent from crafty, which has no metrics listener yet.
-      homelab.keithjackson.dev/auth-mode: forward-auth
-  podMetricsEndpoints:
-    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
-    # container, and a numeric port here would silently stop matching if the port were renumbered.
-    - port: metrics
-      path: /metrics
-      interval: 30s
-      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
-      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
-      # the full interval.
-      scrapeTimeout: 10s

runbooks/README.md

@@ -51,36 +51,6 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
-| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
-
-The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
-because each step looked like the end of it.
-
-**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
-exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
-forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
-listener never started, and no ServiceMonitor or PodMonitor selected them.
-
-**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
-`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
-endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
-families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
-now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
-scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
-the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
-NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
-those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
-`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
-these apps it has to be rendered alongside each app's own policy rather than shipped from the
-oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
-namespaces existing yet.
-
-`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
-itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
-a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
-gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
-whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
-listener at all. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 
@@ -138,14 +108,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
-- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
-  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
-  rather than a 401, the browser follows it silently against a live SSO session, and the call
-  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
-  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
-  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
-  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
-  so these logs (72h of them) are the only record this class of failure leaves.
 
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only

runbooks/kustomization.yaml

@@ -85,17 +85,10 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
-      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
-      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
-      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
-      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
       # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
       # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
       # presents as random session loss in ONE app's log and is only legible with both side by side,
       # so the page has to be reachable at 3am without going looking for it in git.
-      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
-      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
-      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
       - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served

runbooks/oauth2-proxy-auth-failures.md

@@ -1,244 +0,0 @@
-# oauth2-proxy auth failures — telling the three layers apart
-
-Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
-does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
-stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
-page is the diagnostic procedure, and it works whether or not the alert ever fired.
-
-## What this means here
-
-Every app behind Keycloak on this platform has **three** places a request can be refused, and
-from a browser they look identical — because two of the three answer with **HTTP 200**.
-
-| Layer | What refuses | What the browser sees |
-|---|---|---|
-| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
-| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
-| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
-
-**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
-redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
-JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
-Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
-dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
-nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
-failed.
-
-The worked example, jackett on 2026-09-30, straight from the sidecar log:
-
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-    ... ten of these at 15:30:53
-    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
-                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
-    GET "/UI/Dashboard"        200 44379
-    GET "/api/v2.0/indexers?_=..." 200 1501229
-    ... and 26 seconds later, inside that same 5-minute session:
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-
-A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
-signature of the session being **re-established over and over instead of held** — a cookie that
-another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
-down and not of the app being broken.
-
-**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
-*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
-them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
-targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
-the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
-for any of these apps. So there is still nothing to graph and nothing that could have alerted.
-**The logs below remain the only record that this class of failure happened.** Check which state
-you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
-
-## First commands
-
-`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
-with an explicit context.
-
-**1. Which namespaces have an unobserved gate** (this is what the alert measures):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
-```
-
-**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
-fixes and the alert deliberately does not distinguish them:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "up{container=\"oauth2-proxy\"}"'
-```
-
-No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
-`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
-value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
-   curl -s localhost:19090/api/v1/targets | python3 -c "
-import json,sys
-for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
-    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
-        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
-```
-
-`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
-endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
-bypasses policy entirely:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n jackett port-forward \
-     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
-     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
-     http://127.0.0.1:14418/metrics'
-```
-
-`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
-2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
-`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
-
-**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
-and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
-`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
-     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
-```
-
-Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
-
-**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
-you can ask about the whole fleet and about a window that has already scrolled out of
-`kubectl logs`. Port-forward Loki and query it:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
-   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
-     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
-```
-
-and the denominator, which is what makes the number mean something:
-
-```
-   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
-```
-
-**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
-lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
-ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
-
-| namespace | `No valid authentication` | `AuthSuccess` | ratio |
-|---|---|---|---|
-| crafty | 81 | 1 | 81:1 |
-| radarr | 67 | 1 | 67:1 |
-| jackett | 39 | 4 | ~10:1 |
-| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
-
-All three of those are the failure, not the baseline.
-
-**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
-jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
-session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
-The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
-them, **every single one was the same request**:
-
-    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
-
-That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
-reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
-is ever re-prompted; the tab just sits there looking like a working page that has stopped
-updating. **This is the entire failure class in one line**: a non-navigation request answered with
-a redirect, forever, with no error surfaced anywhere.
-
-So when you read a rejection count, **always group by request path before concluding anything**:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
-     | grep -v "/ping" | grep "No valid authentication" -A 1 \
-     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
-```
-
-One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
-clears when the tab is closed. Many different paths from one session means the session itself is
-not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
-first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
-instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
-means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
-session-lifecycle conclusion can be drawn from that window at all.
-
-**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
-and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
-page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
-shape, and `runbooks/crafty-admin-credential.md` covers it.
-
-**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
-**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
-other's cookie and the loser logs exactly the `No valid authentication` line above. List what
-each gate is configured to emit (arguments only — never print a Secret's data):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
-     kubectl --context k3s-prod -n $ns get deploy $ns \
-       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
-     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
-```
-
-A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
-share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
-evicting each other, and the symptom is intermittent and depends on which app you visited last.
-
-## Likely causes, most likely first
-
-1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
-   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
-   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
-   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
-   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
-2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
-   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
-   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
-   gate logs the same line. Look for numbered cookie names in the log, and for a session that
-   never survives a single navigation.
-3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
-   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
-   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
-   jackett trace above.
-4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
-   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
-   die early, the refresh is failing, not missing.
-5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
-   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
-   for the `groups:[...]` list on the `[AuthSuccess]` line.
-6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
-
-## When to escalate or stop
-
-- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
-  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
-  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
-  reverts, and in the meantime you have an unreproducible cluster.
-- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
-  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
-  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
-  transcript here exactly this way.
-- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
-  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
-  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
-- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
-  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
-  report is older than three days, the evidence is gone and the only remaining move is to
-  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
-  the reason to close it rather than silence it.

runbooks/plex-quicksync.md

@@ -5,10 +5,24 @@ not an incident page. It is indexed here because the feature it completes is **i
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
-rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
-mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
+rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
+`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
 
+## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
+
+That is this feature's **one serious failure mode** and it **stalls the whole app tier**
+(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
+extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
+`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
+start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
+purpose.
+
+**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
+re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
+are in **"Plex will not schedule" → remedy B**.
+
 ## What this means here
 
 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
@@ -18,7 +32,7 @@ is in git**:
 
 | Half | Who does it | State |
 |---|---|---|
-| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
+| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
 
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
@@ -30,10 +44,44 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.
 
-What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
-not enable hardware transcoding, it does not prove the device can be *opened* from inside a
-non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
-of the three nodes and all three carry the device.
+### Why a resource request and not a `/dev/dri` hostPath
+
+Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
+will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
+**2026-10-01**: the device node was plainly **visible** in the container
+(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
+from the outside and are not the same problem. Proven three ways: root on the **host** opened the
+same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
+`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
+deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
+device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
+
+Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
+does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
+`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+
+The device plugin is what produces the allow rule — in the v0.37.0 source,
+`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+into both the device injection and the matching cgroup allow.
+
+**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
+and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
+*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
+radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
+also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
+The one improvement is that the loss is no longer silent.
+
+What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
+hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
+renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+advertises, which is the scheduler's doing and not a selector in this repo.
 
 ## Running the commands in this runbook
 
@@ -51,15 +99,46 @@ read of that file below greps a single named key — do not widen one.
 
 ## First commands
 
-**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
+**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
+
+**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
+hostPath mounted" check, and it is now the first thing that can be wrong.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o json \
+   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+```
+
+Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
+node cannot run Plex. Then the DaemonSet behind it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
+```
+
+Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
+symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
+
+**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
+GitOps half landed:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
+     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
 ```
 
-`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
+A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
+mechanism that does not work. Compare
 
 ```
 infra/devcontainer/dev.sh run bash -c \
@@ -69,34 +148,29 @@ infra/devcontainer/dev.sh run bash -c \
 
 against the merge commit before blaming anything else.
 
-**2. Is the device actually visible inside the container?**
+**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
+of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```
 
-Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
-here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
-had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
-transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
-```
-
-**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
+Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
+`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
+to exec into. Then the open, **as the app user**:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```
 
-`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
-in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
-the outside and are not the same problem.
+`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
+not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
+cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
+visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
+causes" #2.
 
 **4. Is the toggle on?** Named key only — never the whole file:
 
@@ -172,25 +246,32 @@ When the dashboard still says plain `Transcode`:
    First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
    investigating anything technical.
 
-2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
-   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
-   character device on this k3s/containerd build **has not been established**; it can only be settled
-   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
-   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
-   correctly owned, step 3 returns `RW-OPEN-DENIED`.
-   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
-   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
-   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
-   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
-   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
-   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
+2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
+   first-command #3 shows `renderD128` present and correctly owned, and the open returns
+   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
+   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
+   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
+   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
+   the pod really did consume the resource
 
-3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
-   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
-   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
-   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
-   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
-   feature that is not load-bearing.
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
+   ```
+
+   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
+   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
+   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
+   that is the mechanism this replaced, and it was measured not to work.
+
+3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
+   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
+   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
+   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
+   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
+   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
+   **Fix the node.**
 
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
    mechanism this feature leans on with no commit in this repo.
@@ -203,6 +284,56 @@ When the dashboard still says plain `Transcode`:
    Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
    sessions show `(hw)` and others do not. Not a fault.
 
+## Plex will not schedule
+
+**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
+`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
+2am.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
+```
+
+`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
+advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
+DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
+device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
+
+**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
+nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
+where to fix it. Check its Flux Kustomization reconciled:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
+```
+
+A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
+
+**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
+code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
+revert you):
+
+1. delete these two lines from `catalog/plex.yaml`:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+2. re-render the golden:
+
+   ```
+   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+   ```
+
+   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
+3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
+
+Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
+`/config` PVC is affected either way — do **not** delete the PVC as part of this.
+
 ## The floating-image caveat
 
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
@@ -243,7 +374,12 @@ hook exists.
   `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
   discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
-  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
+  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
+  because root does not bypass the cgroup device gate.
+- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
+  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
+  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
+  grants access; a volume cannot.
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
   `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
   `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
@@ -253,9 +389,9 @@ hook exists.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
   watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
   lose the DB.
-- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
-  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
-  workload.
+- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
+  confirmed) *under the current mechanism*. The next step is a platform change in
+  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
 
 ## Known gaps, deliberately not fixed here
 
@@ -279,5 +415,8 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
-of writing, and the Plex-side toggle was still unset.
+(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
+reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
+Hardware transcoding has **still not been observed working**: the open test has not been re-run under
+the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
+toggle is still unset.

schemas/renderer-contract.md

@@ -165,56 +165,115 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
   ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
   over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
-  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
-  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
-  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
-- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
-  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
-  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
-     render node read-write, so it is deliberately not `readOnly`).
+  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
+  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
+  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
+  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
+  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
+  second route to the same one.
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:
 
-  **Mounting the device is sufficient. That is the whole mapping.**
+      resources:
+        limits:
+          gpu.intel.com/i915: 1
+
+  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
+  no privileged anything.
+  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
+    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
+    requests-only extended resource is rejected as invalid.
+  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
+    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
+    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
+    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
+  - This is the **only** app-container `resources` render path in the file, and it is deliberately
+    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
+    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
+    dict *exactly* so the attempt fails loudly.
+- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
+  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
+  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
+  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
+  **could not be opened**:
+
+  | attempt | result |
+  |---|---|
+  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
+  identical from the outside and are not the same problem. Proven three ways: root on the **host**
+  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
+  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
+  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
+  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
+  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
+  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+  into both the device injection and the matching cgroup allow.
+- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
+  plainly rather than sold:
+
+  | | device/resource unavailable ⇒ |
+  |---|---|
+  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
+  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
+  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
+  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
+  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
+  transcoding working at all.
+  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
+    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
+    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
+    procedure, because it is what someone will need at 2am.
+  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
+    longer silent. There is still no metric and no alert (media apps here render no
+    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
+    "Plex is quietly software-transcoding" never was.
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
+  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
   move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2, 2026-09-30:** the image's own
+  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
   `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
   `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
-  `cont-init.d` completes before `services.d`, so the account database is already correct when the
-  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
-  **replaces** the process group list from `/etc/group` rather than appending to it:
-  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
-  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
-  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
-  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
+  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
+  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
+  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
+  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
+  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
+  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
+  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
+  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
+  own absence in `render.py --selftest`.
   - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
     **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
     No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
-    sufficient" was established for an image that resolves the device group itself. A FUTURE
-    consumer that runs non-root and has no equivalent init hook would still need the group solved
-    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
-    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
-    first such consumer must re-measure rather than assume this bullet covers it.
-- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
-  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
-  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
-  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
-  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
-  on such a node leaves nothing durable behind.
-  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
-    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
-    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
-    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
-    an operator observing it in the application's own dashboard.
+  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
+    established for an image that resolves the device group itself. A FUTURE consumer that runs
+    non-root and has no equivalent init hook would still need the group solved somehow — and the
+    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
+    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
+    The first such consumer must re-measure rather than assume this bullet covers it.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
-  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
-  this field is about the device, not about where the pod lands); no resource requests/limits (there
-  is no app-container resource render path, and inventing one here would be scope creep).
-- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
+  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
+  own justification — this field is about the device, not about where the pod lands; the extended
+  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
+  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
+  `resources` block carries the extended resource and nothing else).
+- **Historical, kept so a reader recognises it rather than reintroduces it:**
+  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
+  what made a GPU-less node degrade to software transcoding instead of failing the mount and
+  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
+  the mechanism it protected never granted access in the first place.
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
+  (no `resources` key, no volume, no mount, no `securityContext`).
   Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
 
@@ -344,11 +403,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing
    `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
    Derived from `exposure` alone; enforced on every gate run by
    `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
-   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
-   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
-   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
-   implementation can be replaced without touching a single ServiceDefinition.
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
+   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
+   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
+   (never a sidecar) and no `securityContext` either. All directions are proven by
+   `render.py --selftest`, and each assertion is mutation-tested. The resource name
+   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
+   capability's implementation can be replaced without touching a single ServiceDefinition — which
+   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
+   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
+   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
+   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
+   to delete the `hardware:` block and re-render.
 
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02