feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)
#37
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#37
Reference in new issue
Block a user
Final step of the
hardware.gpu: intel-quicksyncworkstream: drop the/dev/drihostPath, emit agpu.intel.com/i915resource request instead. The device plugin landed in #36 and is live; this is the renderer change that consumes it.Zero ServiceDefinition field changes.
catalog/plex.yaml'shardware:block is byte-identical —git diff -U0on that file contains no non-comment lines. That was the design bet when the field was made an enum of capability classes rather than a device path (contract §hardware, invariant 8), and this PR collects on it.Why: the hostPath mechanism was measured not to work
In the live Plex pod on k3s-prod-2, 2026-10-01. The device node was visible in the container (
crw-rw---- root video2 226,128 renderD128) and could not be opened:uid 0,O_RDWREPERM"Operation not permitted"uid 1000,O_RDWREPERM"Operation not permitted"EPERMfrom the device cgroup, notEACCESfrom file permissions — proven three ways: root on the host opens the same devicerc=0; the node is unifiedcgroup2fs; and the container's runcconfig.jsonreadlinux.resources.devices: [{allow: False, access: 'rwm'}]withlinux.devices: None— one deny-all rule, no allow entry for major 226. A hostPath is a filesystem operation: it binds the device node in but adds no cgroup allow rule, because Kubernetes cannot infer one from a volume.uid 0failing matters: root does not bypass the cgroup device gate, so "run as root" was never a fix, andprivileged: trueis unavailable (Kyvernorestrict-host-namespacesis Enforce on app namespaces). The device plugin is the fix — v0.37.0'screateDeviceSpecsFromDrmFilesemitsDeviceSpec{…, Permissions: "rw"}, which the kubelet turns into the injection and the matching cgroup allow.What changed
catalog/render.py—DRI_VOLUME_NAME/DRI_DEVICE_DIRretired forQUICKSYNC_RESOURCE. Theintel-quicksyncbranch now emits exactlyresources.limits: {gpu.intel.com/i915: 1}on the app container only (never a sidecar), and no volume, no mount, nosecurityContext, nonodeSelector.limitsnotrequestsis mandatory: Kubernetes mirrors requests from limits, and a requests-only extended resource is invalid. This is the only app-containerresourcesrender path and is deliberately narrow — no general CPU/memory feature was invented along the way./dev/drimount asserted absent; the no-hardwarenegative extended toresources; thepodSecurityContextpass-through and the no-synthesised-securityContextassertion kept; new: the limit must never reach the oauth2-proxy sidecar. Mutation-tested, 8/8 mutants caught —requestsinstead oflimits, emission dropped, hostPath re-added, emitted unconditionally,supplementalGroupssynthesised, resource renamed…/xe, quantity as the string"1", limit leaked onto the sidecar.schemas/renderer-contract.md— mechanism half of §hardwareand invariant 8 rewritten; theEPERMmeasurement recorded as the reason for the design so nobody simplifies it back;DirectoryOrCreatekept but labelled historical.catalog/plex.yaml— comment correction only (the old text restated the dead mechanism). Measured facts that remain true are kept, including the no-supplementalGroupsexplanation.runbooks/plex-quicksync.md— diagnostics now check.status.allocatable["gpu.intel.com/i915"]and the plugin DaemonSet; the decisive open test is specified as uid 1000 vias6-setuidgid plex; new top-of-page "STOP HERE IF PLEX ISPending" plus a "Plex will not schedule" procedure. Manual-toggle section unchanged.apps/wave-0/plex.yaml— re-rendered: six hostPath lines out, a three-lineresourcesblock in.git status --porcelain apps/⇒M apps/wave-0/plex.yamland nothing else.apps/wave-1/keycloak.yamlandapps/wave-0/vm-demo.yamlverified untouched (they are excluded from the golden comparison byRENDER_GOLDEN_NOT_ONBOARDED, so a spurious diff there would land ungated).clusters/untouched;--emit-wavesnot run.The one real regression — stated rather than sold
DirectoryOrCreatemounted an empty dir; Plex ran and silently software-transcoded. Invisible, harmless to everything else.apps-wave-0iswait: trueandapps-wave-1dependsOnit, so this stalls the app tier.Unavoidable: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's schedulability is now coupled to the
intel-gpu-pluginDaemonSet; that is the price of hardware transcoding working at all.Escape hatch — one field, no code change: delete the
hardware:block fromcatalog/plex.yamland re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in the runbook ("Plex will not schedule → remedy B", with commands), and in the handoff.Gate
Pinned toolbox, from a worktree (
DEV_KUBECONFIG=/dev/null+ anENGINEshim bind-mounting the main checkout's.gitread-only, per CLAUDE.md §3;prod.kubeconfigwas not copied). Run on9babe57:rc 3— UNDETERMINED. NOT green, and I am not calling it green. Three skips, the three expected ones; nothing failed.spec-immutabilitydecisions-loglintl0render-goldenl1approval-policyevent='none')gitea-settingsSETTINGS_API_TOKENunsetconformanceCLUSTER_AVAILABLE != truediff base: origin/main @ 8214642d87de … HEAD @ 9babe5768f88; touched requirements resolved (R-PLAT-02among them), so the R-TEST-03 coverage gate actually applied. Note.gitea/workflows/gate.yml:138is./ci/gate.sh || [ "$?" -eq 3 ], so CI's tick cannot distinguish rc 0 from rc 3 — read the stage output.Still NOT verified
Hardware transcoding is still not observed working. A green gate proves the manifest renders.
open()test cannot be re-run until this merges and Flux reconciles. The case for the change rests on the plugin'sDeviceSpec{… Permissions: "rw"}read from v0.37.0 source, not observed on this cluster.HardwareAcceleratedCodecstoggle is still unset, and no env var/ConfigMap/postBuildcan set it.i915=1on all three nodes and the DaemonSet's 3/3/3 come from the brief (2026-10-01, DaemonSet 2m15s old). Per CLAUDE.md §2b, re-measure.wait: true+dependsOn, not drilled. The escape hatch is likewise reasoned, not rehearsed.Post-merge sequence (operator)
apps-wave-0's.status.lastAppliedRevisionis the merge commit.get pod -o wide.Pending⇒ runbook "Plex will not schedule".…containers[0].resources.limits.rc=0is the pass condition:exec deploy/plex -- s6-setuidgid plex sh -c 'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'. Passing asuid 0is weaker evidence and must not be accepted as proof — Plex runs as 1000, and uid 0 was one of the twoEPERMs that killed the hostPath.Transcode (hw)is the proof.🤖 Generated with Claude Code
hardware.gpu: intel-quicksyncwithgpu.intel.com/i915, not a /dev/dri hostPath 78bf68493dThe hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container (`crw-rw---- root video2 226,128 renderD128`) and could not be opened -- `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP, not EACCES from file permissions, proven three ways: root on the HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is a filesystem operation: it binds the device node in but adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. The Intel GPU device plugin produces that allow rule (`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`). It shipped as #36 and is live: all three prod nodes advertise `gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01). ZERO ServiceDefinition field changes, which was the entire point of making the field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s `hardware:` block comes through byte-identical; only its comment changes, because it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather than claimed. - catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for `QUICKSYNC_RESOURCE`; the hardware branch now sets `resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a sidecar) and emits no volume, no mount, no securityContext, no nodeSelector. `limits` not `requests` is mandatory: Kubernetes mirrors requests from limits, and a requests-only extended resource is invalid. This is the only app-container `resources` render path and is deliberately narrow -- no general CPU/memory feature was invented. - selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the no-`hardware` negative extended to `resources`; the podSecurityContext pass-through and the no-synthesised-securityContext assertions kept; one new assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested: 8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource name, quantity as a string, limit leaked onto the sidecar). - apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block added. Only golden touched; keycloak/vm-demo unchanged. THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing -> `DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY software-transcoded (invisible, harmless to everything else). After: resource unavailable (plugin broken, removed, or not yet reconciled) -> Plex is UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it, so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's schedulability is now coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code change: delete the `hardware:` block from catalog/plex.yaml and re-render. Documented in the contract, the runbook (its own top-of-page section and a "Plex will not schedule" procedure) and the handoff. HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test cannot be re-run until this merges and Flux reconciles, and the Plex-side `HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the manifest renders. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>gitea-admin referenced this pull request2026-10-02 04:56:57 +00:00