docs(decisions): handoff for the quicksync mechanism swap, including the regression it trades into

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/plex.yaml

@@ -137,6 +137,9 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config

catalog/plex.yaml

@@ -80,6 +80,60 @@ env:
   # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
   # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
   # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
+#
+# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
+# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
+# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
+#
+# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
+# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
+# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
+# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
+# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
+# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
+# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
+# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+hardware:
+  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/render.py

@@ -166,11 +166,14 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
-# deliberately confined to this file. Swapping to the Intel GPU device plugin
+# deliberately confined to this file. This file is the ONLY place the mechanism appears,
-# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
+# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
-# nowhere else -- no catalog entry moves.
+# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
-DRI_VOLUME_NAME = "dri"
+#
-DRI_DEVICE_DIR = "/dev/dri"
+# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
+# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
+# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
+QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
 
 
 # =============================================================================
@@ -526,41 +529,59 @@ def r_deployment(s: dict) -> dict:
     # workload needs the host iGPU's render node for hardware video transcode. The SD says
     # only that; the wiring below is the platform's current mechanism for satisfying it.
     #
-    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
+    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
-    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
+    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
-    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
+    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
-    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
+    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
-    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
+    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
-    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
+    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
-    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
+    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
-    # directory the kubelet creates on such a node leaves nothing durable behind.
+    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
+    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
+    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
+    # because Kubernetes cannot infer one from a volume.
     #
-    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
+    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
+    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
+    # kubelet turns into both the device injection and the matching cgroup allow.
+    #
+    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
+    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
+    # requests-only extended resource is invalid. App container only, never a sidecar -- the
+    # oauth2-proxy sidecar does no transcoding.
+    #
+    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
+    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
+    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
+    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
+    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
+    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
+    # extended resource is a hard scheduling constraint, and it is also the only thing that
+    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
+    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
+    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
+    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
+    # unblocks. runbooks/plex-quicksync.md carries it.
     #
     # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
-    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2
-    # 2026-09-30: the image's own cont-init hook
+    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
     # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
     # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
     # container when none exists, and `usermod -a -G` the app user into it -- every start,
-    # nothing hardcoded. cont-init.d completes before services.d, so the account database
+    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
-    # s6 reads at exec time already carries the group. The service is then started with
+    # is why this swap needs no group work either. cont-init.d completes before services.d,
-    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
+    # so the account database s6 reads at exec time already carries the group: measured live,
-    # /etc/group rather than appending to it: `Plex Media Server` runs with
+    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
-    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
+    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
-    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
+    # REPLACES the process group list from /etc/group rather than appending to it, so
-    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
+    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
-    # group handling is the image's job and it does it correctly. A supplementalGroups entry
+    # before the app ever starts. A supplementalGroups entry here would be inert AND would
-    # here would be inert AND would mislead whoever debugs this next.
+    # mislead whoever debugs this next.
     hw_gpu = (s.get("hardware") or {}).get("gpu")
     if hw_gpu == "intel-quicksync":
-        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
+        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
-        volumes.append(
-            {
-                "name": DRI_VOLUME_NAME,
-                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
-            }
-        )
 
     if volume_mounts:
         container["volumeMounts"] = volume_mounts
@@ -2579,10 +2600,13 @@ def _selftest() -> int:
         "an SD without podSecurityContext must not gain a pod securityContext",
     )
 
-    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
+    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
-    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
+    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
-    #     assertions pin the MECHANISM that currently satisfies it -- including what it
+    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
-    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
+    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
+    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
+    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
+    #     the emission site), and the absence assertion is what stops it coming back.
     hw_docs = render(
         {
             "name": "hw",
@@ -2593,24 +2617,36 @@ def _selftest() -> int:
     hw_pod = next(
         d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
+    hw_app = hw_pod["containers"][0]
+    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
+    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
+    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
+    # CPU/memory resources feature through this field (that would be scope creep, and this
+    # field is about the DEVICE only).
     check(
-        {
+        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
-            "name": "dri",
+        "hardware.gpu=intel-quicksync must render resources.limits "
-            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
+        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
-        }
-        in hw_pod.get("volumes", []),
-        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
-        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
     )
-    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
+    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
+    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
     check(
-        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
+        not any(
-        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
+            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
+        )
+        and not any(
+            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
+            for m in hw_app.get("volumeMounts", [])
+        ),
+        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
+        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
+        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
     )
-    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
+    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
-    # own cont-init hook adds the app user to the device's group inside the container, and
+    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
-    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
+    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
-    # emission site). Asserting the absence is what stops someone adding an inert one back.
+    # (see the emission site). Asserting the absence is what stops someone adding an inert
+    # one back.
     check(
         "securityContext" not in hw_pod,
         "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
@@ -2618,10 +2654,12 @@ def _selftest() -> int:
     )
     # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
     # rejects `privileged: true` in an app namespace, and placement is a separate concern
-    # with its own justification -- this field is about the DEVICE only.
+    # with its own justification -- this field is about the DEVICE only. (The extended
+    # resource now constrains placement to nodes the plugin advertises, which is the
+    # scheduler's own doing, not a nodeSelector this renderer writes.)
     check(
         "privileged" not in hw_pod.get("securityContext", {})
-        and "securityContext" not in hw_pod["containers"][0]
+        and "securityContext" not in hw_app
         and "nodeSelector" not in hw_pod,
         "hardware.gpu must emit no privileged securityContext and no nodeSelector",
     )
@@ -2634,8 +2672,10 @@ def _selftest() -> int:
     check(
         "volumes" not in nohw_pod
         and "volumeMounts" not in nohw_pod["containers"][0]
+        and "resources" not in nohw_pod["containers"][0]
         and "securityContext" not in nohw_pod,
-        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
+        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
+        "securityContext",
     )
     # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
     # hardware field neither adds to it nor rewrites it.
@@ -2652,9 +2692,35 @@ def _selftest() -> int:
         if d[1]["kind"] == "Deployment"
     )["spec"]["template"]["spec"]
     check(
-        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
+        and hwm_pod["containers"][0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}},
         "hardware.gpu must leave the SD's own podSecurityContext untouched",
     )
+    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
+    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
+    hws_containers = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hws",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
+                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]["containers"]
+    check(
+        len(hws_containers) == 2
+        and hws_containers[0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}}
+        and "gpu.intel.com/i915"
+        not in hws_containers[1].get("resources", {}).get("limits", {}),
+        "hardware.gpu must put the i915 limit on the APP container only, never on the "
+        "oauth2-proxy sidecar",
+    )
 
     # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
     #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kustomization.yaml

@@ -12,6 +12,15 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any

decisions/2026-09-30-plex-quicksync.md

@@ -0,0 +1,174 @@
+---
+date: 2026-09-30
+branch: feat/plex-quicksync
+title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
+---
+
+## What changed
+
+Three things, and the third is the point of the branch.
+
+1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+   placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
+   comment in the file's established style. The comment records: the field is **intent, not
+   mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
+   not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
+   **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
+   work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
+   why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
+   `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
+   that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
+   pointing at the new runbook.
+
+2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
+   tree:
+
+   ```
+           - name: dri
+             mountPath: /dev/dri
+         - name: dri
+           hostPath:
+             path: /dev/dri
+             type: DirectoryOrCreate
+   ```
+
+3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
+   `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
+   under "Operator procedures". **The file alone would have been a 404** — the pod serves a
+   ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
+   precisely because "the file is in git" and "the URL resolves" are different claims. The page
+   covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
+   acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
+   keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
+   the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
+   with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
+   caveat with a digest check.
+
+**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
+cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
+is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
+duplicating its changes here would have been worse than showing them. **This must merge after #27.**
+
+Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
+would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
+does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
+
+### Two follow-ups left unclaimed, with the reasoning
+
+- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
+  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
+  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
+  transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
+  candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
+  (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
+  emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
+  have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
+  working around it — which is exactly why it deserves its own PR and its own review rather than
+  being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
+- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
+  quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
+  **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
+  documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
+  well clear of this PR.
+
+### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
+
+Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
+`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
+exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
+ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
+
+**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
+branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
+(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
+entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
+**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
+grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
+`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
+adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
+`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
+`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
+
+Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
+reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
+`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
+`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
+oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
+(`auth: native`, no sidecar) is unaffected by it.
+
+## What I verified
+
+- **The render is exactly the intended change, and nothing else moved.** After
+  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
+  silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
+  `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
+  is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
+  the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
+  are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
+  `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
+  diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
+  plus my new files.
+- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
+  `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
+  read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
+  `nodeSelector`, no resources in the diff.
+- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
+  carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
+  L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
+  lookup. Also indexed in `runbooks/README.md`.
+- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
+  alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
+  page): all four required sections (`## What this means here`, `## First commands`,
+  `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
+  floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
+- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
+- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
+  page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
+
+## What I did NOT verify
+
+**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
+does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
+runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
+
+- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
+  containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
+  device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
+  uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
+  the two failures look identical from the outside. If it is refused, the fallback is the Intel
+  Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
+  **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
+  `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
+- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
+  **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
+  operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
+  That is the whole reason the runbook exists.
+- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
+  no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
+  rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
+  `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
+  alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
+  exactly that. Closing the gap honestly means an exporter first.
+- **I did not re-derive this session's node or in-container measurements.** The three-node
+  `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
+  replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
+  `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
+  re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
+  the cluster before relying on any of it.
+- **The image is a floating tag.** Everything known about the hook came from digest
+  `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
+  `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
+  "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
+  the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
+  enforcement.
+- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
+  online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
+  query it.
+- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
+  an HTTP 200; it does not make the request, and I did not either.

decisions/2026-10-01-intel-gpu-device-plugin.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-01
+branch: feat/intel-gpu-device-plugin
+title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
+---
+
+## What changed
+
+Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
+**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
+Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
+device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
+1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
+hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
+response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
+it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
+`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
+`app-namespace=true`.
+
+New files:
+
+- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
+  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
+  provenance header carrying the exact re-vendor command and both sha256s (generated body
+  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
+  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
+  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
+  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
+  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
+  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
+  kustomize base, for the other reason `external-snapshotter` records: a remote base is
+  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
+- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
+  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
+- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
+- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
+  non-choices, and the post-merge verification sequence.
+- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
+  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
+- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
+
+**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
+cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
+`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
+split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
+the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
+called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
+consequence is worse than average here, because `platform` gates `platform-config` gates
+`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
+would stall every app, so a component added to fix a **silent no-op** would have gained the ability
+to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
+`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
+of path and authors cannot self-approve.
+
+`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
+never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
+is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
+`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
+`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
+eleven of its components, which shrinks the cold-start window in which a node has not yet
+advertised the resource.
+
+`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
+`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
+making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
+genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
+rendered graph), `wait: true` gates nothing while turning
+`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
+answered before the consumer PR may merge.
+
+Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
+(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
+`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
+believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
+a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
+*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
+nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
+already adds the `plex` user to each device's group and that works identically on a
+plugin-injected device.
+
+**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
+are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
+later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
+on all three nodes** — a request for a resource no node advertises makes the Plex pod
+Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
+stall the whole app tier and turn today's silent no-op into a real outage.
+
+## What I verified
+
+Mechanism, read from the v0.37.0 source rather than assumed:
+
+- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
+  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
+  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
+  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
+  grant and no extra privilege — it does not need more than the upstream manifest gives it.
+- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
+  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
+  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
+- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
+  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
+  nodes have `i915` bound.
+- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
+  `false` and `setupLevelZeroService` returns early unless it is set.
+- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
+
+Image pin, verified against the registry:
+
+- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
+  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
+  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
+  latest upstream release (2026-09-16) and is not a prerelease.
+- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
+  while GitHub release tags do.
+
+Policy scope, read from the policies rather than assumed:
+
+- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
+  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
+  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
+  The new namespace is labelled `"false"`, so none of them match it.
+- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
+  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
+  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
+  collision.
+
+Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
+bind-mounted read-only — never copied into the worktree):
+
+- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
+- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
+  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
+  "NONE". That is the before-state this change is meant to move.
+- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
+  measured evidence that the device-plugin API and the default kubelet socket path
+  (`/var/lib/kubelet/device-plugins`) work on these nodes.
+
+Offline rendering (the main proof available to a PR):
+
+- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
+  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
+  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
+- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
+  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
+  (`clusters/prod`, `clusters/dev`) also render and carry it.
+- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
+  so `wait: true` gates nothing.
+- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
+  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
+  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
+  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
+  what was generated.
+
+Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
+gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
+(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
+not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
+other stage passed. **rc 3 is not green** and is not reported as such.
+
+## What I did NOT verify
+
+**Nothing about this component's runtime behaviour. It has never run.**
+
+- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
+  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
+  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
+- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
+  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
+  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
+- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
+  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
+  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
+  admission.
+- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
+  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
+  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
+  passing as uid 0 is weaker and must not be accepted as proof.
+- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
+  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
+  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
+  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
+- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
+  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
+  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
+  the pods fail to start with an SELinux error, that line is the first suspect.
+- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
+  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
+  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
+  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
+- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
+  no measurement of this plugin's footprint on these nodes. Worth a glance given
+  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
+- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
+  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
+  both overlays render the same base — but it is not a thing I could test.
+- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
+  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
+  unconfirmed-by-me.

decisions/2026-10-01-quicksync-via-device-plugin.md

@@ -0,0 +1,211 @@
+---
+date: 2026-10-01
+branch: feat/quicksync-via-device-plugin
+title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
+---
+
+## What changed
+
+The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
+moved** — that was the whole design bet when this field was made an enum of capability classes
+instead of a device path, and this is the change that collects on it.
+
+**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
+**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
+in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
+**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
+operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
+infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
+"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces).
+
+**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
+(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
+the **app container only** (never a sidecar):
+
+```yaml
+resources:
+  limits:
+    gpu.intel.com/i915: 1
+```
+
+and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
+`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
+a requests-only extended resource is invalid. This is the **only** app-container `resources` render
+path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
+along the way.
+
+**Self-tests** — the three existing ones updated plus one new:
+- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
+  a stray CPU/memory addition through this field fails loudly;
+- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
+- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
+- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
+  assertion is kept verbatim;
+- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
+
+**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
+The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
+the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
+and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
+
+**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
+`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
+("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
+points at the contract, and keeps the measured facts that remain true (device on all three nodes,
+`i915` bound, the cont-init hook / `supplementalGroups` explanation).
+
+**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
+`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
+decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
+recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
+`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
+hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
+settable by any env var) is unchanged.
+
+**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
+added. Nothing else in `apps/` changed.
+
+### The one real regression, stated rather than sold
+
+This trades failure modes, and the new one is **worse in blast radius**:
+
+| | device/resource unavailable ⇒ |
+|---|---|
+| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
+| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
+thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
+schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
+yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
+all.
+
+**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
+and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
+the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
+someone will need at 2am.
+
+## What I verified
+
+**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
+DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
+(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
+`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
+`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
+themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
+PM, not by me** — see "What I did NOT verify".
+
+**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
+(a test that cannot fail is not a test):
+
+| mutant | caught by |
+|---|---|
+| `requests` instead of `limits` | exact-resources assertion |
+| emission dropped entirely | exact-resources assertion |
+| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
+| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
+| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
+| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
+| quantity as the string `"1"` | exact-resources assertion |
+| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
+
+**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
+--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
+else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
+hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
+`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
+from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
+diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
+
+**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
+filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
+`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
+
+**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
+not calling it green.** Run from this worktree in the pinned toolbox
+(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
+read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
+
+| stage | result |
+|---|---|
+| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
+| `decisions-log` | green |
+| `lint` | green |
+| `l0` | green |
+| `render-golden` | green |
+| `l1` | green |
+| `approval-policy` | green (`event='none'`, N/A off a PR event) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
+
+Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
+HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
+coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
+"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
+
+## What I did NOT verify
+
+**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
+it.** A green gate proves the **manifest renders**, nothing more.
+
+- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
+  and Flux reconciles.** The whole case for this change is that the device plugin's
+  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
+  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
+  deliver it here, this change buys nothing and costs schedulability.
+- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
+  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
+  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
+  no matter how the device arrives.
+- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
+  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
+  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
+  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
+  that it had not been observed stable for any length of time.
+- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
+  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
+  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
+  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
+- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
+  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
+- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
+  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
+  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
+- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
+  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
+  floating-image caveat.
+
+### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
+
+1. Merge the PR once the gate reports. Confirm Flux applied it — compare
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
+   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
+2. Confirm Plex **scheduled** (this is the new risk):
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
+   `Pending` ⇒ runbook "Plex will not schedule".
+3. Confirm it consumed the resource:
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
+   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
+4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
+   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
+   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
+   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
+   and uid 0 was one of the two `EPERM`s that killed the hostPath.
+5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
+   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
+   `Transcode (hw)` is the proof; plain `Transcode` is software.
+
+Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
+reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
+re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
+escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.

platform/intel-gpu-plugin/README.md

@@ -0,0 +1,225 @@
+# platform/intel-gpu-plugin
+
+**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
+[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
+pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
+*request* a GPU instead of hoping a `hostPath` is enough.
+
+- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
+  consumer of an Intel iGPU.
+- **Tier:** its **own** Flux Kustomization
+  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
+  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
+  *Placement* below.
+- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
+  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
+  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
+- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
+  to the *local kubelet* over a unix socket.
+
+## What this component owns
+
+| File | Purpose |
+|---|---|
+| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
+| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
+| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
+
+## The problem this fixes
+
+Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
+the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
+
+```
+uid 0,    O_RDWR -> EPERM "Operation not permitted"
+uid 1000, O_RDWR -> EPERM "Operation not permitted"
+```
+
+`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
+
+1. root on the **host** opens the same device with `rc=0`;
+2. the node is unified `cgroup2fs`;
+3. the container's runc `config.json` reads
+   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
+   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
+   device.
+
+A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
+allow rule, because Kubernetes cannot infer one from a volume.
+
+**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
+(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
+per DRM node:
+
+```go
+specs = append(specs, pluginapi.DeviceSpec{
+    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
+})
+```
+
+and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
+cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
+also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
+under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
+2026-10-01). Correctly so.
+
+## Placement — why its own Kustomization
+
+The cheap option was one line in the shared `platform` tier. Neither that file nor a new
+`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
+have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
+already argued the case when Cilium was split out (read the comment in
+`platform/kustomization.yaml`):
+
+* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
+  "the GPU plugin is wedged" from "longhorn is slow".
+* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
+  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
+* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
+  rollout**. This is a 3-node DaemonSet rollout.
+
+The consequence is worse than average here because of what sits downstream: `platform` gates
+`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
+that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
+component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
+cause a *real outage*.
+
+The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
+`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
+path, and authors cannot self-approve. The approval was required either way.
+
+### `wait: true` is a health signal, not a gate
+
+This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
+because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
+on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
+it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
+genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
+answer to "did it come up on all three nodes", which is precisely the question that must be
+answered before the consumer PR may merge.
+
+Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
+`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
+power to stall every app, which is the thing this placement exists to prevent.
+
+## Deliberate non-choices
+
+**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
+controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
+three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
+already declared in git. For three identical nodes and one consumer that is strictly more blast
+radius for no gain.
+
+**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
+what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
+`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
+`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
+`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
+probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
+and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
+
+* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
+  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
+  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
+* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
+  `bypathFound = false`.
+
+**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
+card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
+
+* Plex is the only consumer today and wants one device;
+* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
+  it, so N containers get the same `renderD128` and contend for the same two media engines
+  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
+  "transcodes are mysteriously slow", which is neither;
+* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
+  right number will be known rather than guessed.
+
+**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
+630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
+something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
+rather than relying on the plugin's behaviour on such a node.
+
+**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
+`/dev/dri` char device and adds the `plex` user to its group — measured live as
+`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
+That mechanism is image-side and works identically on a plugin-injected device.
+
+## Resource name
+
+`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
+the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
+is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
+and a consumer's request would stop matching — worth knowing before that kernel arrives.
+
+## Versioning
+
+Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
+vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
+command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
+sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
+`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
+(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
+a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
+diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
+
+Vendored rather than referenced as a remote kustomize base for the same reason as
+`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
+fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
+then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
+which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
+was up.
+
+Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
+depName=...` immediately above the version literal), because `renovate.json` sets
+`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
+plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
+while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
+if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
+`renovate.json`, not a hand-edit.
+
+## Ordering — do not reorder the stack
+
+The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
+`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
+merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
+resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
+`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
+a real outage.
+
+## Post-merge verification
+
+Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
+the operator merges and Flux reconciles. Afterwards:
+
+```bash
+D=./infra/devcontainer/dev.sh
+
+# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
+$D run bash -c 'flux get kustomization intel-gpu-plugin'
+$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
+#    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
+
+# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
+$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
+  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+#    want: three lines, each i915=1   (before this change: ABSENT on all three)
+
+# 3. If a node shows ABSENT, read that node's plugin logs.
+$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
+```
+
+### The decisive test
+
+Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
+the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
+through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
+request the resource:
+
+```bash
+$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
+```
+
+**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
+1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.

platform/intel-gpu-plugin/intel-gpu-plugin.yaml

@@ -0,0 +1,117 @@
+---
+# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
+#   deployments/gpu_plugin/base
+#
+# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
+#   kubectl kustomize \
+#     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
+#     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
+#   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
+#   # and the volume set between minor releases before.
+#
+# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
+#   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
+# The raw upstream source file reviewed to produce it
+# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
+#   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
+# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
+# clean, so passing it through kustomize changed formatting only and no content.
+#
+# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
+# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
+# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
+# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
+# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
+# platform/external-snapshotter documents for the same reason.
+#
+# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
+# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
+# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
+# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
+# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
+# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
+# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
+# fragility; not this component's problem.)
+#
+# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
+# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
+# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
+# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
+# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    app: intel-gpu-plugin
+  name: intel-gpu-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-gpu-plugin
+  template:
+    metadata:
+      labels:
+        app: intel-gpu-plugin
+    spec:
+      containers:
+      - env:
+        - name: NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        - name: HOST_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.hostIP
+        image: intel/intel-gpu-plugin:0.37.0
+        imagePullPolicy: IfNotPresent
+        name: intel-gpu-plugin
+        resources:
+          limits:
+            cpu: 100m
+            memory: 115Mi
+          requests:
+            cpu: 40m
+            memory: 45Mi
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop:
+            - ALL
+          readOnlyRootFilesystem: true
+          seLinuxOptions:
+            type: container_device_plugin_t
+          seccompProfile:
+            type: RuntimeDefault
+        volumeMounts:
+        - mountPath: /dev/dri
+          name: devfs
+          readOnly: true
+        - mountPath: /sys/class/drm
+          name: sysfsdrm
+          readOnly: true
+        - mountPath: /var/lib/kubelet/device-plugins
+          name: kubeletsockets
+        - mountPath: /var/run/cdi
+          name: cdipath
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      volumes:
+      - hostPath:
+          path: /dev/dri
+        name: devfs
+      - hostPath:
+          path: /sys/class/drm
+        name: sysfsdrm
+      - hostPath:
+          path: /var/lib/kubelet/device-plugins
+        name: kubeletsockets
+      - hostPath:
+          path: /var/run/cdi
+          type: DirectoryOrCreate
+        name: cdipath
+  updateStrategy:
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+    type: RollingUpdate

platform/intel-gpu-plugin/kustomization.yaml

@@ -0,0 +1,99 @@
+---
+# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
+#
+# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
+# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
+# Measured in the running pod on k3s-prod-2:
+#     uid 0,    O_RDWR -> EPERM "Operation not permitted"
+#     uid 1000, O_RDWR -> EPERM "Operation not permitted"
+# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
+# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
+# config.json carries one deny-all rule and nothing else:
+#     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
+#     linux.devices: None
+# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
+# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
+# Kubernetes cannot infer one from a volume.
+#
+# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
+# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
+# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
+# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
+# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
+# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
+# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
+#
+# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
+# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
+# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
+# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
+# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
+# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
+# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
+#
+# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
+# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
+# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
+# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
+# stat() demonstrably work today (that is how the Plex probe found the device before failing to
+# open it), so the plugin needs no cgroup grant of its own and no privilege.
+#   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
+#     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
+#     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
+#   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
+#   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
+#
+# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
+# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
+# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
+# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
+# request would stop matching — worth knowing before that kernel arrives.
+#
+# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
+# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
+# share one GPU. Deliberately not raised:
+#   * Plex is the only consumer and wants one device.
+#   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
+#     it — N containers get the same renderD128 and contend for the same two media engines
+#     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
+#     "transcodes are mysteriously slow", which is neither.
+#   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
+#     that point the right number will be known rather than guessed.
+# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
+# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
+# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
+#
+# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
+# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
+# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
+# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
+# or on the consumer would duplicate a thing that already works.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: intel-gpu-plugin
+resources:
+  - namespace.yaml
+  - intel-gpu-plugin.yaml
+images:
+  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
+  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
+  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
+  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
+  # digest pinning is reserved for the conformance runner, where it is a security boundary.
+  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
+  # relies on containerd's docker.io default rather than saying so.
+  #
+  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
+  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
+  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
+  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
+  # `newTag` and not up in this header.
+  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
+  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
+  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
+  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
+  # renovate.json, not a hand-edit of this line.
+  - name: intel/intel-gpu-plugin
+    newName: docker.io/intel/intel-gpu-plugin
+    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
+    newTag: "0.37.0"

platform/intel-gpu-plugin/namespace.yaml

@@ -0,0 +1,34 @@
+---
+# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
+#
+# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
+# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
+# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
+# reading the policies, not assumed:
+#   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
+#   * platform/kyverno/policies/require-egress-class-label.yaml
+#   * platform/kyverno/policies/require-auth-mode-label.yaml
+#   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
+#     app-namespace also gets a default-deny NetworkPolicy minted into it)
+# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
+# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
+# access] live in platform namespaces that do NOT carry the app-namespace label".
+#
+# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
+# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
+# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
+# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
+# in git, and because this component's correctness depends on the answer.
+#
+# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
+# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
+# placement is not a workaround for a policy — there is nothing to work around. It is here because
+# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
+# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
+# for a process that talks only to the local kubelet over a unix socket and never uses the network.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: intel-gpu-plugin
+  labels:
+    homelab.keithjackson.dev/app-namespace: "false"

runbooks/README.md

@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
+  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
+  acceleration when available"**, and no environment variable can do it (the image supports no
+  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
+  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
+  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
+  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
+  fallback, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
   re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
   (the four lines that matter, including the upstream `-` that means the proxy answered a request the

runbooks/kustomization.yaml

@@ -95,6 +95,12 @@ configMapGenerator:
       # because the plex.tv claim window is short and is not a thing to go looking
       # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
       - plex-claim=plex-claim.md
+      # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
+      # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
+      # could evaluate. Served because the feature it completes is INERT until a human ticks a box in
+      # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
+      # feature stays switched off for months.
+      - plex-quicksync=plex-quicksync.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md

runbooks/plex-quicksync.md

@@ -0,0 +1,422 @@
+# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
+
+**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
+not an incident page. It is indexed here because the feature it completes is **inert until a human
+clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
+to prevent.
+**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
+rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
+`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
+**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
+
+## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
+
+That is this feature's **one serious failure mode** and it **stalls the whole app tier**
+(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
+extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
+`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
+start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
+purpose.
+
+**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
+re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
+are in **"Plex will not schedule" → remedy B**.
+
+## What this means here
+
+Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
+moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
+everything else. Two independent things have to be true before that happens, and **only one of them
+is in git**:
+
+| Half | Who does it | State |
+|---|---|---|
+| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+
+**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
+grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
+`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
+Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
+`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
+`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
+and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
+there isn't one, and the search has already been done.
+
+### Why a resource request and not a `/dev/dri` hostPath
+
+Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
+will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
+**2026-10-01**: the device node was plainly **visible** in the container
+(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
+from the outside and are not the same problem. Proven three ways: root on the **host** opened the
+same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
+`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
+deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
+device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
+
+Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
+does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
+`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+
+The device plugin is what produces the allow rule — in the v0.37.0 source,
+`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+into both the device injection and the matching cgroup allow.
+
+**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
+and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
+*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
+radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
+also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
+The one improvement is that the loss is no longer silent.
+
+What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
+hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
+renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+advertises, which is the scheduler's doing and not a selector in this repo.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
+container. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
+picks a current-context **per invocation**, and that default has already changed once. Do not run two
+`dev.sh` invocations at once.
+
+**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
+read of that file below greps a single named key — do not widen one.
+
+## First commands
+
+**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
+
+**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
+hostPath mounted" check, and it is now the first thing that can be wrong.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o json \
+   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+```
+
+Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
+node cannot run Plex. Then the DaemonSet behind it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
+```
+
+Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
+symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
+
+**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
+GitOps half landed:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
+```
+
+A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
+mechanism that does not work. Compare
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
+     -o jsonpath="{.status.lastAppliedRevision}"; echo'
+```
+
+against the merge commit before blaming anything else.
+
+**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
+of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
+```
+
+Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
+`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
+to exec into. Then the open, **as the app user**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+```
+
+`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
+not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
+cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
+visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
+causes" #2.
+
+**4. Is the toggle on?** Named key only — never the whole file:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+```
+
+`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
+and the remaining question is whether it *works* — go to "How to verify".
+
+## The one manual step
+
+Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
+(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
+
+The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
+extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
+recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
+them and not invent them:
+
+| Key | What it is |
+|---|---|
+| `HardwareAcceleratedCodecs` | the toggle this page is about |
+| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
+| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
+| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
+| `TranscoderHEVCEncoding` | HEVC output |
+
+**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
+binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
+`Preferences.xml` and concludes hardware transcoding is broken.
+
+Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
+tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
+
+## How to verify — and why there is no alert
+
+Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
+lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+
+- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
+- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
+
+**That dashboard is the *only* proof available, and nothing else will ever tell you.**
+`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
+to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
+inert, and this repo has already been burned at scale by rules that evaluated against nothing while
+looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
+gap is ever worth closing, the honest order is a real exporter first and a rule second.
+
+Corroborating, not a substitute — the server log on the config PVC:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
+     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
+```
+
+Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
+Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
+of hardware. The dashboard remains the verdict.
+
+## Likely causes, most likely first
+
+When the dashboard still says plain `Transcode`:
+
+1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
+   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
+   investigating anything technical.
+
+2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
+   first-command #3 shows `renderD128` present and correctly owned, and the open returns
+   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
+   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
+   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
+   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
+   the pod really did consume the resource
+
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
+   ```
+
+   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
+   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
+   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
+   that is the mechanism this replaced, and it was measured not to work.
+
+3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
+   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
+   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
+   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
+   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
+   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
+   **Fix the node.**
+
+4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
+   mechanism this feature leans on with no commit in this repo.
+
+5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
+   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
+   a hardware failure.
+
+6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
+   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
+   sessions show `(hw)` and others do not. Not a fault.
+
+## Plex will not schedule
+
+**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
+`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
+2am.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
+```
+
+`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
+advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
+DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
+device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
+
+**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
+nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
+where to fix it. Check its Flux Kustomization reconciled:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
+```
+
+A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
+
+**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
+code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
+revert you):
+
+1. delete these two lines from `catalog/plex.yaml`:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+2. re-render the golden:
+
+   ```
+   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+   ```
+
+   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
+3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
+
+Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
+`/config` PVC is affected either way — do **not** delete the PVC as part of this.
+
+## The floating-image caveat
+
+Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
+user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
+`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
+(`plexmediaserver 1.43.4.10903`).
+
+The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
+therefore change the behaviour this feature depends on **with no commit in this repo**, while the
+manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
+stopped, with no git change to point at.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
+```
+
+If that digest is not the one above, **re-verify the hook before trusting this page**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
+```
+
+`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
+described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
+needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
+hook exists.
+
+## When to escalate or stop
+
+- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
+  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
+  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
+  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
+  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
+  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
+  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
+- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
+  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
+  because root does not bypass the cgroup device gate.
+- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
+  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
+  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
+  grants access; a volume cannot.
+- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
+  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
+  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
+  through a branch, a PR and a green gate.
+- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
+  terminal. It holds `PlexOnlineToken`.
+- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
+  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
+  lose the DB.
+- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
+  confirmed) *under the current mechanism*. The next step is a platform change in
+  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
+
+## Known gaps, deliberately not fixed here
+
+Recorded so they are findable, and so nobody assumes they were overlooked:
+
+- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
+  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
+  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
+  transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
+  need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
+  `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
+  transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
+  is the better one, because it removes the node asymmetry outright instead of working around it.
+- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
+  quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
+  parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
+  cluster has a documented history of that going badly (a prod node hung on a warm reboot on
+  2026-09-03). Not worth coupling to this change.
+
+---
+Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
+The node measurements (all three prod nodes) and the in-container image measurements cited here were
+taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
+(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
+reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
+Hardware transcoding has **still not been observed working**: the open test has not been re-run under
+the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
+toggle is still unset.

schemas/renderer-contract.md

@@ -165,56 +165,115 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
   ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
   over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
+  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
-  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
+  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
-  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
+  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
-  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
+  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
-- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
+  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
-  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
+  second route to the same one.
-  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:
-     render node read-write, so it is deliberately not `readOnly`).
 
-  **Mounting the device is sufficient. That is the whole mapping.**
+      resources:
+        limits:
+          gpu.intel.com/i915: 1
+
+  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
+  no privileged anything.
+  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
+    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
+    requests-only extended resource is rejected as invalid.
+  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
+    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
+    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
+    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
+  - This is the **only** app-container `resources` render path in the file, and it is deliberately
+    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
+    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
+    dict *exactly* so the attempt fails loudly.
+- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
+  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
+  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
+  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
+  **could not be opened**:
+
+  | attempt | result |
+  |---|---|
+  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
+  identical from the outside and are not the same problem. Proven three ways: root on the **host**
+  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
+  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
+  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
+  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
+  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
+  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+  into both the device injection and the matching cgroup allow.
+- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
+  plainly rather than sold:
+
+  | | device/resource unavailable ⇒ |
+  |---|---|
+  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
+  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
+  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
+  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
+  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
+  transcoding working at all.
+  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
+    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
+    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
+    procedure, because it is what someone will need at 2am.
+  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
+    longer silent. There is still no metric and no alert (media apps here render no
+    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
+    "Plex is quietly software-transcoding" never was.
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
+  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
   move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2, 2026-09-30:** the image's own
+  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
   `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
   `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
-  `cont-init.d` completes before `services.d`, so the account database is already correct when the
+  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
-  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
+  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
-  **replaces** the process group list from `/etc/group` rather than appending to it:
+  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
-  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
+  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
-  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
+  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
-  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
+  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
-  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
+  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
+  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
+  own absence in `render.py --selftest`.
   - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
     **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
     No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
+  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
-    sufficient" was established for an image that resolves the device group itself. A FUTURE
+    established for an image that resolves the device group itself. A FUTURE consumer that runs
-    consumer that runs non-root and has no equivalent init hook would still need the group solved
+    non-root and has no equivalent init hook would still need the group solved somehow — and the
-    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
+    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
-    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
+    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
-    first such consumer must re-measure rather than assume this bullet covers it.
+    The first such consumer must re-measure rather than assume this bullet covers it.
-- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
-  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
-  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
-  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
-  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
-  on such a node leaves nothing durable behind.
-  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
-    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
-    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
-    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
-    an operator observing it in the application's own dashboard.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
-  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
+  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
-  this field is about the device, not about where the pod lands); no resource requests/limits (there
+  own justification — this field is about the device, not about where the pod lands; the extended
-  is no app-container resource render path, and inventing one here would be scope creep).
+  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
-- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
+  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
+  `resources` block carries the extended resource and nothing else).
+- **Historical, kept so a reader recognises it rather than reintroduces it:**
+  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
+  what made a GPU-less node degrade to software transcoding instead of failing the mount and
+  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
+  the mechanism it protected never granted access in the first place.
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
+  (no `resources` key, no volume, no mount, no `securityContext`).
   Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
 
@@ -344,11 +403,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing
    `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
    Derived from `exposure` alone; enforced on every gate run by
    `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
-   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
+   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
-   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
+   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
-   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
+   (never a sidecar) and no `securityContext` either. All directions are proven by
-   implementation can be replaced without touching a single ServiceDefinition.
+   `render.py --selftest`, and each assertion is mutation-tested. The resource name
+   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
+   capability's implementation can be replaced without touching a single ServiceDefinition — which
+   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
+   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
+   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
+   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
+   to delete the `hardware:` block and re-render.
 
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02