docs(decisions): handoff for the quicksync mechanism swap, including the regression it trades into

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/plex.yaml

@@ -137,14 +137,15 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
         - name: media
           mountPath: /media
           readOnly: true
-        - name: dri
-          mountPath: /dev/dri
       volumes:
       - name: config
         persistentVolumeClaim:
@@ -152,10 +153,6 @@ spec:
       - name: media
         persistentVolumeClaim:
           claimName: plex-media
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1

catalog/plex.yaml

@@ -81,46 +81,49 @@ env:
   # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
   # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
-# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
-# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
-# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
-# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
-# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
-# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
-# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
-# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
-# that is scheduler history rather than a requirement — this field deliberately renders no
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
-# nodeSelector, because placement is a separate concern with its own justification.
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
-# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
-# CONTAINER, 2026-09-30: the image's own
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
-# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
-# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
-# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
-# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
-# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
-# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
-# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
-# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
-# securityContext for this field precisely so nobody adds an inert one back later.
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
 #
-# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
-# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
-# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
-# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
-# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
-# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
-# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
-# dashboard.
-#
-# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
@@ -128,8 +131,7 @@ env:
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
-# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
-# device-cgroup failure mode to expect first.
 hardware:
   gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)

catalog/render.py

@@ -166,11 +166,14 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
-# deliberately confined to this file. Swapping to the Intel GPU device plugin
+# deliberately confined to this file. This file is the ONLY place the mechanism appears,
-# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
+# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
-# nowhere else -- no catalog entry moves.
+# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
-DRI_VOLUME_NAME = "dri"
+#
-DRI_DEVICE_DIR = "/dev/dri"
+# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
+# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
+# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
+QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
 
 
 # =============================================================================
@@ -526,41 +529,59 @@ def r_deployment(s: dict) -> dict:
     # workload needs the host iGPU's render node for hardware video transcode. The SD says
     # only that; the wiring below is the platform's current mechanism for satisfying it.
     #
-    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
+    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
-    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
+    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
-    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
+    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
-    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
+    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
-    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
+    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
-    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
+    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
-    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
+    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
-    # directory the kubelet creates on such a node leaves nothing durable behind.
+    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
+    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
+    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
+    # because Kubernetes cannot infer one from a volume.
     #
-    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
+    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
+    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
+    # kubelet turns into both the device injection and the matching cgroup allow.
+    #
+    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
+    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
+    # requests-only extended resource is invalid. App container only, never a sidecar -- the
+    # oauth2-proxy sidecar does no transcoding.
+    #
+    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
+    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
+    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
+    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
+    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
+    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
+    # extended resource is a hard scheduling constraint, and it is also the only thing that
+    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
+    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
+    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
+    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
+    # unblocks. runbooks/plex-quicksync.md carries it.
     #
     # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
-    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2
-    # 2026-09-30: the image's own cont-init hook
+    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
     # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
     # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
     # container when none exists, and `usermod -a -G` the app user into it -- every start,
-    # nothing hardcoded. cont-init.d completes before services.d, so the account database
+    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
-    # s6 reads at exec time already carries the group. The service is then started with
+    # is why this swap needs no group work either. cont-init.d completes before services.d,
-    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
+    # so the account database s6 reads at exec time already carries the group: measured live,
-    # /etc/group rather than appending to it: `Plex Media Server` runs with
+    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
-    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
+    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
-    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
+    # REPLACES the process group list from /etc/group rather than appending to it, so
-    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
+    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
-    # group handling is the image's job and it does it correctly. A supplementalGroups entry
+    # before the app ever starts. A supplementalGroups entry here would be inert AND would
-    # here would be inert AND would mislead whoever debugs this next.
+    # mislead whoever debugs this next.
     hw_gpu = (s.get("hardware") or {}).get("gpu")
     if hw_gpu == "intel-quicksync":
-        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
+        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
-        volumes.append(
-            {
-                "name": DRI_VOLUME_NAME,
-                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
-            }
-        )
 
     if volume_mounts:
         container["volumeMounts"] = volume_mounts
@@ -2579,10 +2600,13 @@ def _selftest() -> int:
         "an SD without podSecurityContext must not gain a pod securityContext",
     )
 
-    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
+    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
-    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
+    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
-    #     assertions pin the MECHANISM that currently satisfies it -- including what it
+    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
-    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
+    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
+    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
+    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
+    #     the emission site), and the absence assertion is what stops it coming back.
     hw_docs = render(
         {
             "name": "hw",
@@ -2593,24 +2617,36 @@ def _selftest() -> int:
     hw_pod = next(
         d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
+    hw_app = hw_pod["containers"][0]
+    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
+    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
+    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
+    # CPU/memory resources feature through this field (that would be scope creep, and this
+    # field is about the DEVICE only).
     check(
-        {
+        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
-            "name": "dri",
+        "hardware.gpu=intel-quicksync must render resources.limits "
-            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
+        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
-        }
-        in hw_pod.get("volumes", []),
-        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
-        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
     )
-    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
+    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
+    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
     check(
-        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
+        not any(
-        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
+            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
+        )
+        and not any(
+            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
+            for m in hw_app.get("volumeMounts", [])
+        ),
+        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
+        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
+        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
     )
-    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
+    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
-    # own cont-init hook adds the app user to the device's group inside the container, and
+    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
-    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
+    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
-    # emission site). Asserting the absence is what stops someone adding an inert one back.
+    # (see the emission site). Asserting the absence is what stops someone adding an inert
+    # one back.
     check(
         "securityContext" not in hw_pod,
         "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
@@ -2618,10 +2654,12 @@ def _selftest() -> int:
     )
     # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
     # rejects `privileged: true` in an app namespace, and placement is a separate concern
-    # with its own justification -- this field is about the DEVICE only.
+    # with its own justification -- this field is about the DEVICE only. (The extended
+    # resource now constrains placement to nodes the plugin advertises, which is the
+    # scheduler's own doing, not a nodeSelector this renderer writes.)
     check(
         "privileged" not in hw_pod.get("securityContext", {})
-        and "securityContext" not in hw_pod["containers"][0]
+        and "securityContext" not in hw_app
         and "nodeSelector" not in hw_pod,
         "hardware.gpu must emit no privileged securityContext and no nodeSelector",
     )
@@ -2634,8 +2672,10 @@ def _selftest() -> int:
     check(
         "volumes" not in nohw_pod
         and "volumeMounts" not in nohw_pod["containers"][0]
+        and "resources" not in nohw_pod["containers"][0]
         and "securityContext" not in nohw_pod,
-        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
+        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
+        "securityContext",
     )
     # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
     # hardware field neither adds to it nor rewrites it.
@@ -2652,9 +2692,35 @@ def _selftest() -> int:
         if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
     check(
-        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
+        and hwm_pod["containers"][0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}},
         "hardware.gpu must leave the SD's own podSecurityContext untouched",
     )
+    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
+    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
+    hws_containers = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hws",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
+                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]["containers"]
+    check(
+        len(hws_containers) == 2
+        and hws_containers[0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}}
+        and "gpu.intel.com/i915"
+        not in hws_containers[1].get("resources", {}).get("limits", {}),
+        "hardware.gpu must put the i915 limit on the APP container only, never on the "
+        "oauth2-proxy sidecar",
+    )
 
     # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
     #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.

decisions/2026-10-01-quicksync-via-device-plugin.md

@@ -0,0 +1,211 @@
+---
+date: 2026-10-01
+branch: feat/quicksync-via-device-plugin
+title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
+---
+
+## What changed
+
+The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
+moved** — that was the whole design bet when this field was made an enum of capability classes
+instead of a device path, and this is the change that collects on it.
+
+**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
+**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
+in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
+**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
+operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
+infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
+"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces).
+
+**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
+(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
+the **app container only** (never a sidecar):
+
+```yaml
+resources:
+  limits:
+    gpu.intel.com/i915: 1
+```
+
+and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
+`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
+a requests-only extended resource is invalid. This is the **only** app-container `resources` render
+path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
+along the way.
+
+**Self-tests** — the three existing ones updated plus one new:
+- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
+  a stray CPU/memory addition through this field fails loudly;
+- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
+- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
+- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
+  assertion is kept verbatim;
+- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
+
+**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
+The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
+the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
+and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
+
+**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
+`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
+("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
+points at the contract, and keeps the measured facts that remain true (device on all three nodes,
+`i915` bound, the cont-init hook / `supplementalGroups` explanation).
+
+**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
+`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
+decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
+recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
+`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
+hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
+settable by any env var) is unchanged.
+
+**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
+added. Nothing else in `apps/` changed.
+
+### The one real regression, stated rather than sold
+
+This trades failure modes, and the new one is **worse in blast radius**:
+
+| | device/resource unavailable ⇒ |
+|---|---|
+| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
+| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
+thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
+schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
+yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
+all.
+
+**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
+and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
+the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
+someone will need at 2am.
+
+## What I verified
+
+**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
+DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
+(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
+`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
+`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
+themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
+PM, not by me** — see "What I did NOT verify".
+
+**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
+(a test that cannot fail is not a test):
+
+| mutant | caught by |
+|---|---|
+| `requests` instead of `limits` | exact-resources assertion |
+| emission dropped entirely | exact-resources assertion |
+| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
+| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
+| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
+| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
+| quantity as the string `"1"` | exact-resources assertion |
+| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
+
+**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
+--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
+else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
+hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
+`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
+from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
+diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
+
+**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
+filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
+`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
+
+**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
+not calling it green.** Run from this worktree in the pinned toolbox
+(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
+read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
+
+| stage | result |
+|---|---|
+| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
+| `decisions-log` | green |
+| `lint` | green |
+| `l0` | green |
+| `render-golden` | green |
+| `l1` | green |
+| `approval-policy` | green (`event='none'`, N/A off a PR event) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
+
+Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
+HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
+coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
+"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
+
+## What I did NOT verify
+
+**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
+it.** A green gate proves the **manifest renders**, nothing more.
+
+- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
+  and Flux reconciles.** The whole case for this change is that the device plugin's
+  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
+  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
+  deliver it here, this change buys nothing and costs schedulability.
+- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
+  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
+  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
+  no matter how the device arrives.
+- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
+  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
+  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
+  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
+  that it had not been observed stable for any length of time.
+- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
+  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
+  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
+  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
+- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
+  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
+- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
+  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
+  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
+- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
+  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
+  floating-image caveat.
+
+### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
+
+1. Merge the PR once the gate reports. Confirm Flux applied it — compare
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
+   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
+2. Confirm Plex **scheduled** (this is the new risk):
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
+   `Pending` ⇒ runbook "Plex will not schedule".
+3. Confirm it consumed the resource:
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
+   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
+4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
+   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
+   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
+   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
+   and uid 0 was one of the two `EPERM`s that killed the hostPath.
+5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
+   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
+   `Transcode (hw)` is the proof; plain `Transcode` is software.
+
+Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
+reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
+re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
+escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.

runbooks/plex-quicksync.md

@@ -5,10 +5,24 @@ not an incident page. It is indexed here because the feature it completes is **i
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
-rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
+rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
-mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
+`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
 
+## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
+
+That is this feature's **one serious failure mode** and it **stalls the whole app tier**
+(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
+extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
+`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
+start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
+purpose.
+
+**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
+re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
+are in **"Plex will not schedule" → remedy B**.
+
 ## What this means here
 
 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
@@ -18,7 +32,7 @@ is in git**:
 
 | Half | Who does it | State |
 |---|---|---|
-| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
+| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
 
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
@@ -30,10 +44,44 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.
 
-What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
+### Why a resource request and not a `/dev/dri` hostPath
-not enable hardware transcoding, it does not prove the device can be *opened* from inside a
+
-non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
+Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
-of the three nodes and all three carry the device.
+will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
+**2026-10-01**: the device node was plainly **visible** in the container
+(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
+from the outside and are not the same problem. Proven three ways: root on the **host** opened the
+same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
+`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
+deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
+device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
+
+Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
+does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
+`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+
+The device plugin is what produces the allow rule — in the v0.37.0 source,
+`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+into both the device injection and the matching cgroup allow.
+
+**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
+and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
+*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
+radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
+also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
+The one improvement is that the loss is no longer silent.
+
+What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
+hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
+renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+advertises, which is the scheduler's doing and not a selector in this repo.
 
 ## Running the commands in this runbook
 
@@ -51,15 +99,46 @@ read of that file below greps a single named key — do not widen one.
 
 ## First commands
 
-**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
+**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
+
+**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
+hostPath mounted" check, and it is now the first thing that can be wrong.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o json \
+   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+```
+
+Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
+node cannot run Plex. Then the DaemonSet behind it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
+```
+
+Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
+symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
+
+**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
+GitOps half landed:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
+     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
 ```
 
-`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
+A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
+mechanism that does not work. Compare
 
 ```
 infra/devcontainer/dev.sh run bash -c \
@@ -69,34 +148,29 @@ infra/devcontainer/dev.sh run bash -c \
 
 against the merge commit before blaming anything else.
 
-**2. Is the device actually visible inside the container?**
+**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
+of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```
 
-Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
+Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
-here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
+`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
-had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
+to exec into. Then the open, **as the app user**:
-transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
-```
-
-**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```
 
-`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
+`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
-in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
+not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
-the outside and are not the same problem.
+cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
+visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
+causes" #2.
 
 **4. Is the toggle on?** Named key only — never the whole file:
 
@@ -172,25 +246,32 @@ When the dashboard still says plain `Transcode`:
    First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
    investigating anything technical.
 
-2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
+2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
-   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
+   first-command #3 shows `renderD128` present and correctly owned, and the open returns
-   character device on this k3s/containerd build **has not been established**; it can only be settled
+   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
-   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
+   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
-   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
+   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
-   correctly owned, step 3 returns `RW-OPEN-DENIED`.
+   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
-   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
+   the pod really did consume the resource
-   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
-   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
-   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
-   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
-   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
 
-3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
+   ```
-   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
+   infra/devcontainer/dev.sh run bash -c \
-   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
+     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
+        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
-   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
+   ```
-   feature that is not load-bearing.
+
+   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
+   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
+   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
+   that is the mechanism this replaced, and it was measured not to work.
+
+3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
+   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
+   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
+   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
+   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
+   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
+   **Fix the node.**
 
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
    mechanism this feature leans on with no commit in this repo.
@@ -203,6 +284,56 @@ When the dashboard still says plain `Transcode`:
    Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
    sessions show `(hw)` and others do not. Not a fault.
 
+## Plex will not schedule
+
+**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
+`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
+2am.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
+```
+
+`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
+advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
+DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
+device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
+
+**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
+nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
+where to fix it. Check its Flux Kustomization reconciled:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
+```
+
+A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
+
+**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
+code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
+revert you):
+
+1. delete these two lines from `catalog/plex.yaml`:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+2. re-render the golden:
+
+   ```
+   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+   ```
+
+   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
+3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
+
+Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
+`/config` PVC is affected either way — do **not** delete the PVC as part of this.
+
 ## The floating-image caveat
 
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
@@ -243,7 +374,12 @@ hook exists.
   `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
   discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
-  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
+  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
+  because root does not bypass the cgroup device gate.
+- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
+  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
+  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
+  grants access; a volume cannot.
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
   `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
   `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
@@ -253,9 +389,9 @@ hook exists.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
   watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
   lose the DB.
-- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
+- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
-  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
+  confirmed) *under the current mechanism*. The next step is a platform change in
-  workload.
+  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
 
 ## Known gaps, deliberately not fixed here
 
@@ -279,5 +415,8 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
+(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
-of writing, and the Plex-side toggle was still unset.
+reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
+Hardware transcoding has **still not been observed working**: the open test has not been re-run under
+the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
+toggle is still unset.

schemas/renderer-contract.md

@@ -165,56 +165,115 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
   ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
   over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
+  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
-  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
+  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
-  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
+  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
-  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
+  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
-- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
+  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
-  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
+  second route to the same one.
-  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:
-     render node read-write, so it is deliberately not `readOnly`).
 
-  **Mounting the device is sufficient. That is the whole mapping.**
+      resources:
+        limits:
+          gpu.intel.com/i915: 1
+
+  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
+  no privileged anything.
+  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
+    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
+    requests-only extended resource is rejected as invalid.
+  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
+    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
+    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
+    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
+  - This is the **only** app-container `resources` render path in the file, and it is deliberately
+    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
+    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
+    dict *exactly* so the attempt fails loudly.
+- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
+  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
+  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
+  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
+  **could not be opened**:
+
+  | attempt | result |
+  |---|---|
+  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
+  identical from the outside and are not the same problem. Proven three ways: root on the **host**
+  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
+  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
+  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
+  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
+  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
+  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+  into both the device injection and the matching cgroup allow.
+- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
+  plainly rather than sold:
+
+  | | device/resource unavailable ⇒ |
+  |---|---|
+  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
+  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
+  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
+  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
+  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
+  transcoding working at all.
+  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
+    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
+    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
+    procedure, because it is what someone will need at 2am.
+  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
+    longer silent. There is still no metric and no alert (media apps here render no
+    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
+    "Plex is quietly software-transcoding" never was.
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
+  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
   move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2, 2026-09-30:** the image's own
+  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
   `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
   `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
-  `cont-init.d` completes before `services.d`, so the account database is already correct when the
+  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
-  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
+  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
-  **replaces** the process group list from `/etc/group` rather than appending to it:
+  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
-  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
+  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
-  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
+  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
-  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
+  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
-  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
+  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
+  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
+  own absence in `render.py --selftest`.
   - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
     **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
     No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
+  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
-    sufficient" was established for an image that resolves the device group itself. A FUTURE
+    established for an image that resolves the device group itself. A FUTURE consumer that runs
-    consumer that runs non-root and has no equivalent init hook would still need the group solved
+    non-root and has no equivalent init hook would still need the group solved somehow — and the
-    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
+    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
-    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
+    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
-    first such consumer must re-measure rather than assume this bullet covers it.
+    The first such consumer must re-measure rather than assume this bullet covers it.
-- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
-  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
-  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
-  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
-  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
-  on such a node leaves nothing durable behind.
-  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
-    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
-    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
-    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
-    an operator observing it in the application's own dashboard.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
-  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
+  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
-  this field is about the device, not about where the pod lands); no resource requests/limits (there
+  own justification — this field is about the device, not about where the pod lands; the extended
-  is no app-container resource render path, and inventing one here would be scope creep).
+  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
-- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
+  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
+  `resources` block carries the extended resource and nothing else).
+- **Historical, kept so a reader recognises it rather than reintroduces it:**
+  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
+  what made a GPU-less node degrade to software transcoding instead of failing the mount and
+  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
+  the mechanism it protected never granted access in the first place.
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
+  (no `resources` key, no volume, no mount, no `securityContext`).
   Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
 
@@ -344,11 +403,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing
    `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
    Derived from `exposure` alone; enforced on every gate run by
    `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
-   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
+   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
-   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
+   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
-   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
+   (never a sidecar) and no `securityContext` either. All directions are proven by
-   implementation can be replaced without touching a single ServiceDefinition.
+   `render.py --selftest`, and each assertion is mutation-tested. The resource name
+   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
+   capability's implementation can be replaced without touching a single ServiceDefinition — which
+   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
+   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
+   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
+   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
+   to delete the `hardware:` block and re-render.
 
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02