feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync

A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.

Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).

The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.

When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:

  - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
  - a read-write volumeMount of it at /dev/dri (VA-API opens the render node
    read-write, so deliberately not readOnly).

Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:

  PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
  pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000

The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.

`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.

Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).

No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.

Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-01 20:54:35 -05:00
1 parent 08deb208a1
commit fa4765bb3e
5 files changed
+391 -2

No files matched your search

@@ -0,0 +1,184 @@
---
date: 2026-09-30
branch: feat/sd-hardware-capability
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
---
## What changed
A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
This PR delivers the capability; consuming it (Plex) is separate work on top.
- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
`minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
emission in `r_deployment`.
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
invariant **8** in that file's numbered list.
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
ballooned.
**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
sidecar), and it is exactly two things:
1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
`readOnly` — VA-API opens the render node read-write).
**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
(placement is a separate concern with its own justification), **no** resource requests (there is no
app-container resource render path), and — the one that is worth spelling out — **no
`podSecurityContext.supplementalGroups`**.
**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
and no GID is pinned anywhere in this repo.
**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
says *what class of host device the workload needs*; the hostPath and the mount exist only in
`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
`description` and the contract section say this explicitly, because the obvious "simplification" —
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
contract and make that migration a catalog-wide edit.
**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
## What I verified
Measured, in this worktree, on this branch:
- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
negative), and a pass-through case (an SD declaring both `hardware` and its own
`podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
- **Re-render is a zero diff.**
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
`git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
rewritten byte-identically and needed no revert). The change is inert until a service opts in.
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
**12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
(`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
`volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
`{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
`{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
was never written into `catalog/`.
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
result each time, most recently against tree `c132893045b1` — the final commit differs from that
run only by this paragraph. Per stage:
| stage | result |
|---|---|
| spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
| decisions-log | pass |
| lint | pass |
| l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
| render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
| l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
| approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
| conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
not calling this green.
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
`apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
the zero diff above demonstrates rather than asserts.
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
`infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
under `apps/` did not change).
## What I did NOT verify
- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
device. Still unverified, and it is the single largest technical risk in this workstream.**
containerd installs a per-container **device cgroup** allow-list, and whether
`/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
seeing it in the application's own dashboard.
- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
I have not verified that it does. The claim was measured for an image that resolves the device
group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
equivalent hook would still need the group solved — and the fix for that case is a **renderer**
change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
`renderer-contract.md` rather than letting it read as general; the first such consumer must
re-measure.
- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
change the behaviour this feature depends on, with no commit anywhere in this repo to show for
it** — and the failure would be the silent software-transcode described above. I have not fixed
that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
be forgotten.
- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
`INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
risk is retired, not overlooked.
- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
`gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
sensitive path and would drag in a human reviewer for a change unrelated to this one).
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
`vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.