home-network-k3s-impl/decisions/2026-10-01-quicksync-via-device-plugin.md

date, branch, title

date	branch	title
2026-10-01	feat/quicksync-via-device-plugin	`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits

What changed

The mechanism behind hardware.gpu: intel-quicksync was replaced. No ServiceDefinition field moved — that was the whole design bet when this field was made an enum of capability classes instead of a device path, and this is the change that collects on it.

Why the old mechanism had to go. The /dev/dri hostPath shipped on 2026-09-30 was measured non-functional in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was visible in the container — crw-rw---- root video2 226,128 renderD128 — and could not be opened:

attempt	result
uid 0, O_RDWR	EPERM "Operation not permitted"
uid 1000, O_RDWR	EPERM "Operation not permitted"

EPERM is the device cgroup, not EACCES from file permissions, proven three ways: root on the host opens the same device rc=0; the node is unified cgroup2fs; and the container's runc config.json read linux.resources.devices: [{allow: False, access: 'rwm'}] with linux.devices: None — one deny-all rule, no allow entry for major 226. A hostPath is a filesystem operation: it binds the device node in and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. uid 0 failing matters: root does not bypass the cgroup device gate, so "run it as root" was never going to work either, and privileged: true is unavailable (Kyverno restrict-host-namespaces is Enforce on app namespaces).

catalog/render.py — retired DRI_VOLUME_NAME / DRI_DEVICE_DIR for QUICKSYNC_RESOURCE (gpu.intel.com/i915). The hardware.gpu == intel-quicksync branch now emits exactly one thing on the app container only (never a sidecar):

resources:
  limits:
    gpu.intel.com/i915: 1

and no volume, no volumeMount, no securityContext, no nodeSelector. limits rather than requests is mandatory, not stylistic: Kubernetes mirrors requests from limits automatically and a requests-only extended resource is invalid. This is the only app-container resources render path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented along the way.

Self-tests — the three existing ones updated plus one new:

• the emitted resources dict is asserted exactly ({"limits": {"gpu.intel.com/i915": 1}}), so a stray CPU/memory addition through this field fails loudly;
• the retired hostPath volume and /dev/dri mount are asserted absent;
• the no-hardware negative extended to cover resources (absence still renders nothing at all);
• the podSecurityContext pass-through still holds, and the "no synthesised securityContext" assertion is kept verbatim;
• new: the i915 limit must never reach the oauth2-proxy sidecar.

schemas/renderer-contract.md — the mechanism half of ### hardware and invariant 8 rewritten. The intent half is unchanged and still correct. The EPERM measurement is recorded as the reason the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried and measured. The DirectoryOrCreate rationale is kept, explicitly labelled historical.

catalog/plex.yaml — comment correction only; the hardware: field is byte-identical. git diff on that file contains no non-comment lines. Its old comment restated the mechanism ("exactly two things — a /dev/dri hostPath volume…"), which is now false; it states intent and points at the contract, and keeps the measured facts that remain true (device on all three nodes, i915 bound, the cont-init hook / supplementalGroups explanation).

runbooks/plex-quicksync.md — the diagnostic sequence now checks .status.allocatable["gpu.intel.com/i915"] and the plugin DaemonSet instead of a hostPath; the decisive open test is specified as uid 1000 via s6-setuidgid plex; the cgroup EPERM is recast from a risk to the reason for the design; a new top-of-page "STOP HERE IF PLEX IS Pending" section and a "Plex will not schedule" procedure carry the new failure mode and the escape hatch. The manual-toggle section (HardwareAcceleratedCodecs, Settings → Transcoder, still not settable by any env var) is unchanged.

apps/wave-0/plex.yaml — re-rendered: six hostPath lines removed, a three-line resources block added. Nothing else in apps/ changed.

The one real regression, stated rather than sold

This trades failure modes, and the new one is worse in blast radius:

	device/resource unavailable ⇒
Before	DirectoryOrCreate mounted an empty dir; Plex ran and silently software-transcoded. Invisible, harmless to everything else.
After	Plex is Unschedulable. apps-wave-0 is wait: true and apps-wave-1 dependsOn it, so this stalls the app tier.

It is unavoidable: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's schedulability is now coupled to the intel-gpu-plugin DaemonSet — broken, removed, or merely not yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at all.

Escape hatch — one field, no code change: delete the hardware: block from catalog/plex.yaml and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what someone will need at 2am.

What I verified

The precondition, re-read from the branch rather than assumed: the plugin is a standalone DaemonSet at platform/intel-gpu-plugin/ with its own Flux Kustomization (clusters/base/intel-gpu-plugin.yaml, named intel-gpu-plugin in flux-system), namespace intel-gpu-plugin, DaemonSet intel-gpu-plugin, pinned intel/intel-gpu-plugin:0.37.0. Every kubectl command I wrote into the runbook names those exact objects. The node/DaemonSet readings themselves (i915=1 on all three nodes; 3 desired / 3 ready / 3 available) were measured by the PM, not by me — see "What I did NOT verify".

render.py --selftest: PASS, and every new assertion is mutation-tested — 8/8 mutants caught (a test that cannot fail is not a test):

mutant	caught by
requests instead of limits	exact-resources assertion
emission dropped entirely	exact-resources assertion
/dev/dri hostPath + mount re-added	hostPath-absent assertion
emitted unconditionally (no hardware key)	no-hardware negative
supplementalGroups synthesised	no-pod-securityContext assertion
resource renamed gpu.intel.com/xe	exact-resources assertion
quantity as the string "1"	exact-resources assertion
limit leaked onto the oauth2-proxy sidecar	new app-container-only assertion

Re-render is clean and scoped. python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation then git status --porcelain apps/ ⇒ M apps/wave-0/plex.yaml and nothing else. git diff --stat apps/ ⇒ 1 file changed, 3 insertions(+), 6 deletions(-) — exactly the six hostPath lines out, the three-line resources block in. apps/wave-1/keycloak.yaml and apps/wave-0/vm-demo.yaml are untouched (checked explicitly: they are git-tracked but excluded from the golden comparison by RENDER_GOLDEN_NOT_ONBOARDED in ci/render-golden.sh, so a spurious diff there would land ungated). clusters/ is untouched; --emit-waves was not run.

Invariant 8 holds, mechanically checked rather than eyeballed. git diff -U0 catalog/plex.yaml filtered to non-comment changed lines returns empty, and yaml.safe_load gives hardware == {'gpu': 'intel-quicksync'}. Zero ServiceDefinition field edits.

The gate: rc 3 UNDETERMINED, which is this environment's calibrated best — NOT green, and I am not calling it green. Run from this worktree in the pinned toolbox (DEV_KUBECONFIG=/dev/null, plus an ENGINE shim bind-mounting the main checkout's .git read-only, per CLAUDE.md §3; infra/ansible/.cache/prod.kubeconfig was not copied):

stage	result
spec-immutability	SKIPPED — spec repo not present at /work/../home-network-k3s (R-TEST-06 unverified)
decisions-log	green
lint	green
l0	green
render-golden	green
l1	green
approval-policy	green (event='none', N/A off a PR event)
gitea-settings	SKIPPED — SETTINGS_API_TOKEN unset
conformance	SKIPPED — CLUSTER_AVAILABLE != true, L2/L3 not run

Three skips, the three expected ones; nothing failed. diff base: origin/main @ 8214642d87de … HEAD @ 78bf68493d7d, touched requirements resolved (R-PLAT-02 among them), so the R-TEST-03 coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set "NOT DETERMINABLE (on-base)" and that is not the run I am quoting.

What I did NOT verify

HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish it. A green gate proves the manifest renders, nothing more.

• The open() test has not been re-run under the new mechanism, and cannot be until this merges and Flux reconciles. The whole case for this change is that the device plugin's DeviceSpec{… Permissions: "rw"} produces the cgroup allow rule the hostPath could not. That is read from the v0.37.0 source, not observed on this cluster. If the plugin does not in fact deliver it here, this change buys nothing and costs schedulability.
• The Plex-side toggle is still unset. HardwareAcceleratedCodecs was absent from Preferences.xml as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or postBuild substitution can. It is a manual UI click, and until it happens Plex software-transcodes no matter how the device arrives.
• I did not measure the cluster at all. i915=1 on all three nodes and the DaemonSet's 3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about the live platform, and neither is a handoff entry: re-measure before relying on them (the commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough that it had not been observed stable for any length of time.
• The new failure mode has not been exercised. "Plugin unavailable ⇒ Plex Unschedulable ⇒ apps-wave-0 stalls ⇒ apps-wave-1 blocked" is derived from wait: true and dependsOn in the Flux wiring, not from a drill. The escape hatch (delete hardware:, re-render) is likewise reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
• No L2/L3 conformance. CLUSTER_AVAILABLE is unset, so the gate's runtime layer did not run here and does not run in CI either. spec-immutability and gitea-settings also skipped.
• CI's own tick, when it appears, will not distinguish rc 0 from rc 3: .gitea/workflows/gate.yml:138 is ./ci/gate.sh || [ "$?" -eq 3 ]. Read the stage output, not the colour — and take the newest row per context from GET /statuses/{sha}, which appends.
• imagePullPolicy: Always on plexinc/pms-docker:latest is still a live hazard to the cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's floating-image caveat.

Post-merge sequence for the operator (all mutating steps are yours, not an agent's)

1. Merge the PR once the gate reports. Confirm Flux applied it — compare kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}' against the merge commit.
2. Confirm Plex scheduled (this is the new risk): kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide. Pending ⇒ runbook "Plex will not schedule".
3. Confirm it consumed the resource: kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o jsonpath='{.items[0].spec.containers[0].resources.limits}'.
4. The decisive test — as uid 1000, via s6-setuidgid plex. rc=0 is the pass condition: kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c 'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'. Passing as uid 0 is weaker evidence and must not be accepted as proof — Plex runs as 1000, and uid 0 was one of the two EPERMs that killed the hostPath.
5. Only then the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when available" (under Show Advanced) → Save. Then force a transcode and read Dashboard → Activity: Transcode (hw) is the proof; plain Transcode is software.

Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do not reach for privileged: true (Kyverno Enforce rejects it, and uid 0 failed anyway) and do not re-add a /dev/dri hostPath (that is the mechanism this replaced, measured non-functional). The escape hatch is to delete the hardware: block from catalog/plex.yaml and re-render.