home-network-k3s-impl/decisions/2026-09-30-sd-hardware-capability.md

---
date: 2026-09-30
branch: feat/sd-hardware-capability
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
---

## What changed

A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
This PR delivers the capability; consuming it (Plex) is separate work on top.

- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
  `minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
  1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
  emission in `r_deployment`.
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
  invariant **8** in that file's numbered list.
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
  Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
  bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
  ballooned.

**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
sidecar), and it is exactly two things:

1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
   `readOnly` — VA-API opens the render node read-write).

**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
(placement is a separate concern with its own justification), **no** resource requests (there is no
app-container resource render path), and — the one that is worth spelling out — **no
`podSecurityContext.supplementalGroups`**.

**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:

    PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
    pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000

`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.

Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
and no GID is pinned anywhere in this repo.

**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
says *what class of host device the workload needs*; the hostPath and the mount exist only in
`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
`description` and the contract section say this explicitly, because the obvious "simplification" —
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
contract and make that migration a catalog-wide edit.

**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.

### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign

Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
PR with a new head.

**One conflict, in `catalog/render.py`, purely textual.** PR #28
(`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
(`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
`DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
`supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
survive verbatim.

Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
**zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.

## What I verified

Measured, in this worktree, on this branch:

- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
  existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
  privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
  volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
  negative), and a pass-through case (an SD declaring both `hardware` and its own
  `podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
  True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
  dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
- **Re-render is a zero diff.**
  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
  `git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
  not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
  rewritten byte-identically and needed no revert). The change is inert until a service opts in.
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
  **12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
  intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
  (`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
  volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
  `volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
  `{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
  `{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
  was never written into `catalog/`.
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
  both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
  main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
  requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
  result each time, most recently against tree `c132893045b1` — the final commit differs from that
  run only by this paragraph. Per stage:

  | stage | result |
  |---|---|
  | spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
  | decisions-log | pass |
  | lint | pass |
  | l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
  | render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
  | l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
  | approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
  | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
  | conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |

  The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
  failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
  not calling this green.
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
  `apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
  the zero diff above demonstrates rather than asserts.
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
  `infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
  are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
  under `apps/` did not change).

## What I did NOT verify

- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
  device. Still unverified, and it is the single largest technical risk in this workstream.**
  containerd installs a per-container **device cgroup** allow-list, and whether
  `/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
  k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
  operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
  but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
  hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
  plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
  ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
  justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
  of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
  apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
  metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
  is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
  where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
  seeing it in the application's own dashboard.
- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
  I have not verified that it does. The claim was measured for an image that resolves the device
  group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
  equivalent hook would still need the group solved — and the fix for that case is a **renderer**
  change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
  `renderer-contract.md` rather than letting it read as general; the first such consumer must
  re-measure.
- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
  above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
  deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
  change the behaviour this feature depends on, with no commit anywhere in this repo to show for
  it** — and the failure would be the silent software-transcode described above. I have not fixed
  that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
  be forgotten.
- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
  `INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
  re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
  risk is retired, not overlooked.
- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
  node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
  `gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
  posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
  restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
  me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
  no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
  I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
  Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
  sensitive path and would drag in a human reviewer for a change unrelated to this one).
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
  `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.