Merge pull request 'feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)' (#37) from feat/quicksync-via-device-plugin into main

Reviewed-on: #37
Reviewed-by: Keith Jackson <[email protected]>

runbooks/plex-quicksync.md

@@ -5,10 +5,24 @@ not an incident page. It is indexed here because the feature it completes is **i
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
-rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
-mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
+rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
+`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
 
+## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
+
+That is this feature's **one serious failure mode** and it **stalls the whole app tier**
+(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
+extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
+`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
+start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
+purpose.
+
+**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
+re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
+are in **"Plex will not schedule" → remedy B**.
+
 ## What this means here
 
 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
@@ -18,7 +32,7 @@ is in git**:
 
 | Half | Who does it | State |
 |---|---|---|
-| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
+| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
 
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
@@ -30,10 +44,44 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.
 
-What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
-not enable hardware transcoding, it does not prove the device can be *opened* from inside a
-non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
-of the three nodes and all three carry the device.
+### Why a resource request and not a `/dev/dri` hostPath
+
+Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
+will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
+**2026-10-01**: the device node was plainly **visible** in the container
+(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
+from the outside and are not the same problem. Proven three ways: root on the **host** opened the
+same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
+`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
+deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
+device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
+
+Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
+does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
+`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+
+The device plugin is what produces the allow rule — in the v0.37.0 source,
+`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+into both the device injection and the matching cgroup allow.
+
+**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
+and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
+*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
+radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
+also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
+The one improvement is that the loss is no longer silent.
+
+What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
+hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
+renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+advertises, which is the scheduler's doing and not a selector in this repo.
 
 ## Running the commands in this runbook
 
@@ -51,15 +99,46 @@ read of that file below greps a single named key — do not widen one.
 
 ## First commands
 
-**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
+**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
+
+**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
+hostPath mounted" check, and it is now the first thing that can be wrong.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o json \
+   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+```
+
+Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
+node cannot run Plex. Then the DaemonSet behind it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
+```
+
+Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
+symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
+
+**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
+GitOps half landed:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
+     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
 ```
 
-`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
+A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
+mechanism that does not work. Compare
 
 ```
 infra/devcontainer/dev.sh run bash -c \
@@ -69,34 +148,29 @@ infra/devcontainer/dev.sh run bash -c \
 
 against the merge commit before blaming anything else.
 
-**2. Is the device actually visible inside the container?**
+**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
+of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```
 
-Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
-here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
-had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
-transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
-```
-
-**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
+Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
+`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
+to exec into. Then the open, **as the app user**:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```
 
-`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
-in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
-the outside and are not the same problem.
+`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
+not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
+cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
+visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
+causes" #2.
 
 **4. Is the toggle on?** Named key only — never the whole file:
 
@@ -172,25 +246,32 @@ When the dashboard still says plain `Transcode`:
    First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
    investigating anything technical.
 
-2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
-   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
-   character device on this k3s/containerd build **has not been established**; it can only be settled
-   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
-   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
-   correctly owned, step 3 returns `RW-OPEN-DENIED`.
-   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
-   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
-   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
-   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
-   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
-   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
+2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
+   first-command #3 shows `renderD128` present and correctly owned, and the open returns
+   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
+   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
+   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
+   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
+   the pod really did consume the resource
 
-3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
-   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
-   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
-   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
-   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
-   feature that is not load-bearing.
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
+   ```
+
+   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
+   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
+   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
+   that is the mechanism this replaced, and it was measured not to work.
+
+3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
+   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
+   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
+   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
+   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
+   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
+   **Fix the node.**
 
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
    mechanism this feature leans on with no commit in this repo.
@@ -203,6 +284,56 @@ When the dashboard still says plain `Transcode`:
    Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
    sessions show `(hw)` and others do not. Not a fault.
 
+## Plex will not schedule
+
+**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
+`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
+2am.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
+```
+
+`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
+advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
+DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
+device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
+
+**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
+nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
+where to fix it. Check its Flux Kustomization reconciled:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
+```
+
+A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
+
+**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
+code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
+revert you):
+
+1. delete these two lines from `catalog/plex.yaml`:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+2. re-render the golden:
+
+   ```
+   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+   ```
+
+   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
+3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
+
+Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
+`/config` PVC is affected either way — do **not** delete the PVC as part of this.
+
 ## The floating-image caveat
 
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
@@ -243,7 +374,12 @@ hook exists.
   `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
   discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
-  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
+  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
+  because root does not bypass the cgroup device gate.
+- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
+  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
+  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
+  grants access; a volume cannot.
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
   `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
   `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
@@ -253,9 +389,9 @@ hook exists.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
   watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
   lose the DB.
-- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
-  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
-  workload.
+- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
+  confirmed) *under the current mechanism*. The next step is a platform change in
+  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
 
 ## Known gaps, deliberately not fixed here
 
@@ -279,5 +415,8 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
-of writing, and the Plex-side toggle was still unset.
+(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
+reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
+Hardware transcoding has **still not been observed working**: the open test has not been re-run under
+the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
+toggle is still unset.