Merge pull request 'feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)' (#37) from feat/quicksync-via-device-plugin into main

Reviewed-on: #37
Reviewed-by: Keith Jackson <[email protected]>

schemas/renderer-contract.md

@@ -166,56 +166,115 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
   ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
   over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
-  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
-  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
-  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
-- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
-  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
-  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
-     render node read-write, so it is deliberately not `readOnly`).
+  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
+  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
+  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
+  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
+  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
+  second route to the same one.
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:
 
-  **Mounting the device is sufficient. That is the whole mapping.**
+      resources:
+        limits:
+          gpu.intel.com/i915: 1
+
+  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
+  no privileged anything.
+  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
+    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
+    requests-only extended resource is rejected as invalid.
+  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
+    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
+    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
+    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
+  - This is the **only** app-container `resources` render path in the file, and it is deliberately
+    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
+    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
+    dict *exactly* so the attempt fails loudly.
+- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
+  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
+  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
+  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
+  **could not be opened**:
+
+  | attempt | result |
+  |---|---|
+  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
+  identical from the outside and are not the same problem. Proven three ways: root on the **host**
+  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
+  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
+  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
+  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
+  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
+  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+  into both the device injection and the matching cgroup allow.
+- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
+  plainly rather than sold:
+
+  | | device/resource unavailable ⇒ |
+  |---|---|
+  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
+  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
+  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
+  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
+  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
+  transcoding working at all.
+  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
+    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
+    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
+    procedure, because it is what someone will need at 2am.
+  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
+    longer silent. There is still no metric and no alert (media apps here render no
+    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
+    "Plex is quietly software-transcoding" never was.
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
+  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
   move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2, 2026-09-30:** the image's own
+  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
   `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
   `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
-  `cont-init.d` completes before `services.d`, so the account database is already correct when the
-  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
-  **replaces** the process group list from `/etc/group` rather than appending to it:
-  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
-  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
-  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
-  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
+  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
+  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
+  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
+  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
+  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
+  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
+  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
+  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
+  own absence in `render.py --selftest`.
   - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
     **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
     No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
-    sufficient" was established for an image that resolves the device group itself. A FUTURE
-    consumer that runs non-root and has no equivalent init hook would still need the group solved
-    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
-    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
-    first such consumer must re-measure rather than assume this bullet covers it.
-- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
-  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
-  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
-  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
-  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
-  on such a node leaves nothing durable behind.
-  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
-    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
-    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
-    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
-    an operator observing it in the application's own dashboard.
+  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
+    established for an image that resolves the device group itself. A FUTURE consumer that runs
+    non-root and has no equivalent init hook would still need the group solved somehow — and the
+    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
+    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
+    The first such consumer must re-measure rather than assume this bullet covers it.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
-  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
-  this field is about the device, not about where the pod lands); no resource requests/limits (there
-  is no app-container resource render path, and inventing one here would be scope creep).
-- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
+  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
+  own justification — this field is about the device, not about where the pod lands; the extended
+  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
+  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
+  `resources` block carries the extended resource and nothing else).
+- **Historical, kept so a reader recognises it rather than reintroduces it:**
+  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
+  what made a GPU-less node degrade to software transcoding instead of failing the mount and
+  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
+  the mechanism it protected never granted access in the first place.
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
+  (no `resources` key, no volume, no mount, no `securityContext`).
   Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
 
@@ -365,11 +424,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing
    `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
    Derived from `exposure` alone; enforced on every gate run by
    `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
-   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
-   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
-   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
-   implementation can be replaced without touching a single ServiceDefinition.
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
+   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
+   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
+   (never a sidecar) and no `securityContext` either. All directions are proven by
+   `render.py --selftest`, and each assertion is mutation-tested. The resource name
+   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
+   capability's implementation can be replaced without touching a single ServiceDefinition — which
+   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
+   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
+   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
+   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
+   to delete the `hardware:` block and re-render.
 
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02