diff --git a/decisions/2026-10-01-quicksync-via-device-plugin.md b/decisions/2026-10-01-quicksync-via-device-plugin.md new file mode 100644 index 0000000..789cd7a --- /dev/null +++ b/decisions/2026-10-01-quicksync-via-device-plugin.md @@ -0,0 +1,211 @@ +--- +date: 2026-10-01 +branch: feat/quicksync-via-device-plugin +title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits" +--- + +## What changed + +The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field +moved** — that was the whole design bet when this field was made an enum of capability classes +instead of a device path, and this is the change that collects on it. + +**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured +**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible* +in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened: + +| attempt | result | +|---|---| +| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" | +| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" | + +`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the +**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc +`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with +`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem* +operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot +infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so +"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno +`restrict-host-namespaces` is Enforce on app namespaces). + +**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE` +(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on +the **app container only** (never a sidecar): + +```yaml +resources: + limits: + gpu.intel.com/i915: 1 +``` + +and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than +`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and +a requests-only extended resource is invalid. This is the **only** app-container `resources` render +path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented +along the way. + +**Self-tests** — the three existing ones updated plus one new: +- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so + a stray CPU/memory addition through this field fails loudly; +- the retired hostPath volume and `/dev/dri` mount are asserted **absent**; +- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all); +- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`" + assertion is kept verbatim; +- **new:** the i915 limit must never reach the oauth2-proxy sidecar. + +**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten. +The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason* +the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried +and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical. + +**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical. +`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism +("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and +points at the contract, and keeps the measured facts that remain true (device on all three nodes, +`i915` bound, the cont-init hook / `supplementalGroups` explanation). + +**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks +`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the +decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is +recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS +`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape +hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not +settable by any env var) is unchanged. + +**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block +added. Nothing else in `apps/` changed. + +### The one real regression, stated rather than sold + +This trades failure modes, and the new one is **worse in blast radius**: + +| | device/resource unavailable ⇒ | +|---|---| +| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. | +| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. | + +It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only +thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's +schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not +yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at +all. + +**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml` +and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in +the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what +someone will need at 2am. + +## What I verified + +**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone +DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization +(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace +`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every +`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings +themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the +PM, not by me** — see "What I did NOT verify". + +**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught** +(a test that cannot fail is not a test): + +| mutant | caught by | +|---|---| +| `requests` instead of `limits` | exact-resources assertion | +| emission dropped entirely | exact-resources assertion | +| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion | +| emitted unconditionally (no `hardware` key) | no-`hardware` negative | +| `supplementalGroups` synthesised | no-pod-`securityContext` assertion | +| resource renamed `gpu.intel.com/xe` | exact-resources assertion | +| quantity as the string `"1"` | exact-resources assertion | +| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion | + +**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout +--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing +else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six +hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and +`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded +from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious +diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run. + +**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml` +filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives +`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits. + +**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am +not calling it green.** Run from this worktree in the pinned toolbox +(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git` +read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied): + +| stage | result | +|---|---| +| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) | +| `decisions-log` | green | +| `lint` | green | +| `l0` | green | +| `render-golden` | green | +| `l1` | green | +| `approval-policy` | green (`event='none'`, N/A off a PR event) | +| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset | +| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run | + +Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de … +HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03 +coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set +"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting. + +## What I did NOT verify + +**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish +it.** A green gate proves the **manifest renders**, nothing more. + +- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges + and Flux reconciles.** The whole case for this change is that the device plugin's + `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is + read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact + deliver it here, this change buys nothing and costs schedulability. +- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from + `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or + `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes + no matter how the device arrives. +- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's + 3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about + the live platform, and neither is a handoff entry: **re-measure before relying on them** (the + commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough + that it had not been observed stable for any length of time. +- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒ + `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the + Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise + reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers. +- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run + here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped. +- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**: + `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the + colour — and take the newest row per context from `GET /statuses/{sha}`, which appends. +- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the + cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's + floating-image caveat. + +### Post-merge sequence for the operator (all mutating steps are yours, not an agent's) + +1. Merge the PR once the gate reports. Confirm Flux applied it — compare + `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 + -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit. +2. Confirm Plex **scheduled** (this is the new risk): + `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`. + `Pending` ⇒ runbook "Plex will not schedule". +3. Confirm it consumed the resource: + `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex + -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`. +4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:** + `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c + 'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`. + **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000, + and uid 0 was one of the two `EPERM`s that killed the hostPath. +5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when + available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity: + `Transcode (hw)` is the proof; plain `Transcode` is software. + +Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not** +reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not** +re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The +escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.