212 lines
13 KiB
Markdown
212 lines
13 KiB
Markdown
---
|
|
date: 2026-10-01
|
|
branch: feat/quicksync-via-device-plugin
|
|
title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
|
|
---
|
|
|
|
## What changed
|
|
|
|
The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
|
|
moved** — that was the whole design bet when this field was made an enum of capability classes
|
|
instead of a device path, and this is the change that collects on it.
|
|
|
|
**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
|
|
**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
|
|
in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
|
|
|
|
| attempt | result |
|
|
|---|---|
|
|
| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
|
|
| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
|
|
|
|
`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
|
|
**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
|
|
`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
|
|
`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
|
|
operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
|
|
infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
|
|
"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
|
|
`restrict-host-namespaces` is Enforce on app namespaces).
|
|
|
|
**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
|
|
(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
|
|
the **app container only** (never a sidecar):
|
|
|
|
```yaml
|
|
resources:
|
|
limits:
|
|
gpu.intel.com/i915: 1
|
|
```
|
|
|
|
and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
|
|
`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
|
|
a requests-only extended resource is invalid. This is the **only** app-container `resources` render
|
|
path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
|
|
along the way.
|
|
|
|
**Self-tests** — the three existing ones updated plus one new:
|
|
- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
|
|
a stray CPU/memory addition through this field fails loudly;
|
|
- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
|
|
- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
|
|
- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
|
|
assertion is kept verbatim;
|
|
- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
|
|
|
|
**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
|
|
The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
|
|
the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
|
|
and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
|
|
|
|
**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
|
|
`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
|
|
("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
|
|
points at the contract, and keeps the measured facts that remain true (device on all three nodes,
|
|
`i915` bound, the cont-init hook / `supplementalGroups` explanation).
|
|
|
|
**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
|
|
`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
|
|
decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
|
|
recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
|
|
`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
|
|
hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
|
|
settable by any env var) is unchanged.
|
|
|
|
**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
|
|
added. Nothing else in `apps/` changed.
|
|
|
|
### The one real regression, stated rather than sold
|
|
|
|
This trades failure modes, and the new one is **worse in blast radius**:
|
|
|
|
| | device/resource unavailable ⇒ |
|
|
|---|---|
|
|
| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
|
|
| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
|
|
|
|
It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
|
|
thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
|
|
schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
|
|
yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
|
|
all.
|
|
|
|
**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
|
|
and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
|
|
the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
|
|
someone will need at 2am.
|
|
|
|
## What I verified
|
|
|
|
**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
|
|
DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
|
|
(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
|
|
`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
|
|
`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
|
|
themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
|
|
PM, not by me** — see "What I did NOT verify".
|
|
|
|
**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
|
|
(a test that cannot fail is not a test):
|
|
|
|
| mutant | caught by |
|
|
|---|---|
|
|
| `requests` instead of `limits` | exact-resources assertion |
|
|
| emission dropped entirely | exact-resources assertion |
|
|
| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
|
|
| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
|
|
| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
|
|
| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
|
|
| quantity as the string `"1"` | exact-resources assertion |
|
|
| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
|
|
|
|
**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
|
|
--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
|
|
else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
|
|
hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
|
|
`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
|
|
from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
|
|
diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
|
|
|
|
**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
|
|
filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
|
|
`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
|
|
|
|
**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
|
|
not calling it green.** Run from this worktree in the pinned toolbox
|
|
(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
|
|
read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
|
|
|
|
| stage | result |
|
|
|---|---|
|
|
| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
|
|
| `decisions-log` | green |
|
|
| `lint` | green |
|
|
| `l0` | green |
|
|
| `render-golden` | green |
|
|
| `l1` | green |
|
|
| `approval-policy` | green (`event='none'`, N/A off a PR event) |
|
|
| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
|
|
| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
|
|
|
|
Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
|
|
HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
|
|
coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
|
|
"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
|
|
|
|
## What I did NOT verify
|
|
|
|
**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
|
|
it.** A green gate proves the **manifest renders**, nothing more.
|
|
|
|
- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
|
|
and Flux reconciles.** The whole case for this change is that the device plugin's
|
|
`DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
|
|
read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
|
|
deliver it here, this change buys nothing and costs schedulability.
|
|
- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
|
|
`Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
|
|
`postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
|
|
no matter how the device arrives.
|
|
- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
|
|
3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
|
|
the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
|
|
commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
|
|
that it had not been observed stable for any length of time.
|
|
- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
|
|
`apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
|
|
Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
|
|
reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
|
|
- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
|
|
here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
|
|
- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
|
|
`.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
|
|
colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
|
|
- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
|
|
cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
|
|
floating-image caveat.
|
|
|
|
### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
|
|
|
|
1. Merge the PR once the gate reports. Confirm Flux applied it — compare
|
|
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
|
|
-o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
|
|
2. Confirm Plex **scheduled** (this is the new risk):
|
|
`kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
|
|
`Pending` ⇒ runbook "Plex will not schedule".
|
|
3. Confirm it consumed the resource:
|
|
`kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
|
|
-o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
|
|
4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
|
|
`kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
|
|
'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
|
|
**Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
|
|
and uid 0 was one of the two `EPERM`s that killed the hostPath.
|
|
5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
|
|
available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
|
|
`Transcode (hw)` is the proof; plain `Transcode` is software.
|
|
|
|
Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
|
|
reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
|
|
re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
|
|
escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.
|