From 2279d14ec5fc91b1eb3bdddce8704a8eb97d005c Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 10:48:00 -0500 Subject: [PATCH] feat(schema): declare a host hardware capability, `hardware.gpu: intel-quicksync` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ServiceDefinition could not say it needs the host's Intel QuickSync render device. The schema is `additionalProperties: false`, so there was no authorable form for it at all, and `apps/` is generated — a hand-edit would be reverted by the next render. Adds optional top-level `hardware`, with one optional member `gpu`, an enum whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0). The enum is the point. The SD declares WHAT CLASS of device it needs; the hostPath, the mount and the GID live only in catalog/render.py. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then a renderer change with zero catalog edits. Letting an SD name `/dev/dri` directly would look simpler and make that migration a catalog-wide edit, so both the schema description and the renderer contract say so explicitly. When `hardware.gpu == intel-quicksync`, r_deployment emits, on the app container / pod spec only: - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate; - a read-write volumeMount of it at /dev/dri (VA-API opens the render node read-write, so deliberately not readOnly); - INTEL_RENDER_GID MERGED into podSecurityContext.supplementalGroups, never clobbering what the SD already declares. `DirectoryOrCreate` is load-bearing. A node that comes back without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding, not fail the mount: apps-wave-0 is `wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so the empty directory the kubelet creates leaves nothing durable behind. Its cost — a missing device is silent, and no metric exists to alert on — is recorded in the contract rather than papered over with an alert that could not fire. Deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host- namespaces is Enforce and it is not needed), no nodeSelector (placement is a separate concern with its own justification), no resource requests (there is no app-container resource render path). INTEL_RENDER_GID = 992 is a single named constant carrying its provenance: measured on all three prod nodes on 2026-09-30, and a *systemd dynamic* allocation rather than a static GID, so a node rebuild can move it. No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero diff, which is the proof the change is inert until opted into. Three self-test cases in the existing check(...) style: the positive, the negative (an SD without `hardware` gains no volume, no mount, no securityContext — mirroring the podSecurityContext negative) and the merge case ([44] + 992 -> [44, 992], fsGroup preserved). Note that NO gate stage runs `render.py --selftest`; they were run by hand and the gap is recorded in the handoff. Co-Authored-By: Claude Opus 5 (1M context) --- catalog/render.py | 136 +++++++++++++++- .../2026-09-30-sd-hardware-capability.md | 147 ++++++++++++++++++ schemas/README.md | 5 +- schemas/renderer-contract.md | 44 ++++++ schemas/servicedefinition.schema.json | 15 +- 5 files changed, 343 insertions(+), 4 deletions(-) create mode 100644 decisions/2026-09-30-sd-hardware-capability.md diff --git a/catalog/render.py b/catalog/render.py index cc6c9a5..9fefba1 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -133,6 +133,24 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1" OAUTH2_PROXY_PORT = 4180 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab" +# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------ +# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json +# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is +# deliberately confined to this file. Swapping to the Intel GPU device plugin +# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and +# nowhere else -- no catalog entry moves. +DRI_VOLUME_NAME = "dri" +DRI_DEVICE_DIR = "/dev/dri" +# GID of the host `render` group, which owns /dev/dri/renderD128 (mode 0660, root:render). +# MEASURED 2026-09-30 on ALL THREE prod nodes (HP EliteDesk 800 G5 Mini, i7-9700T, UHD +# Graphics 630, i915 bound -- not xe): GID 992, group name `render`, identical on each. +# This is a *systemd dynamic* allocation (systemd-udev creates `render` out of the dynamic +# system range at first boot), NOT a Debian/LSB static GID, so it is NOT guaranteed stable: +# a node rebuild, or a reinstall that creates system groups in a different order, can move +# it. If hardware transcode stops working after a node rebuild, re-measure +# `getent group render` on every node before assuming anything else. +INTEL_RENDER_GID = 992 + # ============================================================================= # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping. @@ -483,6 +501,30 @@ def r_deployment(s: dict) -> dict: "persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"}, } ) + # hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this + # workload needs the host iGPU's render node for hardware video transcode. The SD says + # only that; the wiring below is the platform's current mechanism for satisfying it. + # + # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to + # `Directory`. A node that comes back without the render device (rebuild, BIOS change, + # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding, + # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a + # failed hostPath mount stalls the entire app tier for the sake of a feature that is not + # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in + # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty + # directory the kubelet creates on such a node leaves nothing durable behind. + # + # The mount is deliberately NOT readOnly: VA-API opens the render node read-write. + hw_gpu = (s.get("hardware") or {}).get("gpu") + if hw_gpu == "intel-quicksync": + volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR}) + volumes.append( + { + "name": DRI_VOLUME_NAME, + "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"}, + } + ) + if volume_mounts: container["volumeMounts"] = volume_mounts @@ -505,8 +547,19 @@ def r_deployment(s: dict) -> dict: # pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a # non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC). # Absent => the key is never set, so apps that declare none render byte-identical. - if s.get("podSecurityContext"): - pod_spec["securityContext"] = s["podSecurityContext"] + # + # `hardware.gpu: intel-quicksync` additionally needs the host `render` GID in + # supplementalGroups, because /dev/dri/renderD128 is root:render 0660 and the container + # does not run as root. It is MERGED into whatever the SD already declares rather than + # replacing it -- an SD that sets its own supplementalGroups keeps every one of them. + psc = dict(s.get("podSecurityContext") or {}) + if hw_gpu == "intel-quicksync": + groups = list(psc.get("supplementalGroups") or []) + if INTEL_RENDER_GID not in groups: + groups.append(INTEL_RENDER_GID) + psc["supplementalGroups"] = groups + if psc: + pod_spec["securityContext"] = psc if volumes: pod_spec["volumes"] = volumes @@ -2440,6 +2493,85 @@ def _selftest() -> int: "an SD without podSecurityContext must not gain a pod securityContext", ) + # 6b) hardware.gpu: intel-quicksync -> /dev/dri hostPath volume + RW mount + the host + # `render` GID. The SD declares the CAPABILITY; these assertions pin the MECHANISM + # that currently satisfies it, so a change to it is a deliberate, visible edit. + hw_docs = render( + { + "name": "hw", + "source": {"image": "a:1"}, + "hardware": {"gpu": "intel-quicksync"}, + } + ) + hw_pod = next( + d[1] for d in hw_docs if d[1]["kind"] == "Deployment" + )["spec"]["template"]["spec"] + check( + { + "name": "dri", + "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"}, + } + in hw_pod.get("volumes", []), + "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume " + "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)", + ) + hw_mounts = hw_pod["containers"][0].get("volumeMounts", []) + check( + {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts, + "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container", + ) + check( + hw_pod.get("securityContext", {}).get("supplementalGroups") + == [INTEL_RENDER_GID], + "hardware.gpu=intel-quicksync must add the host render GID to supplementalGroups", + ) + # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce) + # rejects `privileged: true` in an app namespace, and placement is a separate concern + # with its own justification -- this field is about the DEVICE only. + check( + "privileged" not in hw_pod.get("securityContext", {}) + and "securityContext" not in hw_pod["containers"][0] + and "nodeSelector" not in hw_pod, + "hardware.gpu must emit no privileged securityContext and no nodeSelector", + ) + # The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext + # negative above -- an SD that does not opt in must be byte-identical to before. + nohw_pod = next( + d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}}) + if d[1]["kind"] == "Deployment" + )["spec"]["template"]["spec"] + check( + "volumes" not in nohw_pod + and "volumeMounts" not in nohw_pod["containers"][0] + and "securityContext" not in nohw_pod, + "an SD without `hardware` must gain no dri volume, no mount and no securityContext", + ) + # Merge, not clobber: an SD that declares its own supplementalGroups keeps them AND + # gains the render GID, and its other podSecurityContext keys survive untouched. + hwm_pod = next( + d[1] + for d in render( + { + "name": "hwm", + "source": {"image": "a:1"}, + "hardware": {"gpu": "intel-quicksync"}, + "podSecurityContext": { + "fsGroup": 1001, + "supplementalGroups": [44], + }, + } + ) + if d[1]["kind"] == "Deployment" + )["spec"]["template"]["spec"] + check( + hwm_pod["securityContext"] == { + "fsGroup": 1001, + "supplementalGroups": [44, INTEL_RENDER_GID], + }, + "hardware.gpu must MERGE the render GID into the SD's own supplementalGroups " + "(both GIDs, fsGroup preserved), never clobber them", + ) + # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that # fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged. fa_docs = render( diff --git a/decisions/2026-09-30-sd-hardware-capability.md b/decisions/2026-09-30-sd-hardware-capability.md new file mode 100644 index 0000000..5295de4 --- /dev/null +++ b/decisions/2026-09-30-sd-hardware-capability.md @@ -0,0 +1,147 @@ +--- +date: 2026-09-30 +branch: feat/sd-hardware-capability +title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` renders the /dev/dri wiring, and the enum is what keeps the mechanism swappable +--- + +## What changed + +A new optional top-level ServiceDefinition property, `hardware`, with one optional member, +`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.** +This PR delivers the capability; consuming it (Plex) is separate work on top. + +- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`, + `minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 → + 1.8.0**, changelog entry prepended to the schema `description` per that file's own convention. +- `catalog/render.py` — three module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`, + `INTEL_RENDER_GID = 992`) and the emission in `r_deployment`. +- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field + invariant **8** in that file's numbered list. +- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0. + Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one + bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing + ballooned. + +**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container / pod spec +only (never a sidecar): + +1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`; +2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not + `readOnly` — VA-API opens the render node read-write); +3. `INTEL_RENDER_GID` **merged into** `podSecurityContext.supplementalGroups`, never clobbering + what the SD already declares. + +It emits **no** `privileged: true` (Kyverno `restrict-host-namespaces` is Enforce on app namespaces +and it is not needed), **no** `nodeSelector` (placement is a separate concern with its own +justification) and **no** resource requests (there is no app-container resource render path). + +**The design decision worth defending: the field is an enum of intent, not a device path.** The SD +says *what class of host device the workload needs*; the hostPath, the mount and the GID exist only +in `catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource +requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema +`description` and the contract section say this explicitly, because the obvious "simplification" — +letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing +contract and make that migration a catalog-wide edit. + +**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild, +BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding, +not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed +hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing. +`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable +behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site. + +## What I verified + +Measured, in this worktree, on this branch: + +- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** Three new cases + were added in the existing `check(...)` style: the positive (volume + mount + GID, and *no* + privileged/nodeSelector), the negative (an SD without `hardware` gains no volume, no volumeMount + and no `securityContext`), and the merge case (`supplementalGroups: [44]` + the field → + `[44, 992]`, with `fsGroup` preserved). +- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if + True:`) in a scratch copy and re-ran: rc 1 with three named failures, including "an SD without + `hardware` must gain no dri volume, no mount and no securityContext". A test that cannot fail is + not a test; this one can. +- **Re-render is a zero diff.** + `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then + `git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two + not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were + rewritten byte-identically and needed no revert). The change is inert until a service opts in. +- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all + **12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu: + intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected + (`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`). +- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn + volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result: + `volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `securityContext` = + `{runAsUser: 1000, supplementalGroups: [44, 992]}`; `volumes` = the PVC then + `{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`. That toy SD was never written + into `catalog/`. +- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with + both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the + main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f` … `HEAD @ + 9f0db1495cfd`; touched requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Per stage: + + | stage | result | + |---|---| + | spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. | + | decisions-log | pass | + | lint | pass | + | l0 | pass | + | render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* | + | l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded | + | approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) | + | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. | + | conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. | + + The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage + failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am + not calling this green. +- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so + `apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what + the zero diff above demonstrates rather than asserts. +- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per + `infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**` + are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files + under `apps/` did not change). + +## What I did NOT verify + +- **Whether a non-privileged pod can actually `open()` this hostPath char device at all. This is + the largest technical risk in the workstream and nobody has tested it.** `supplementalGroups` gets + the DAC permission bits right, but containerd installs a **device cgroup** allow-list per + container, and whether `/dev/dri/renderD128` reaching the container as a bind-mounted device node + is also permitted by that cgroup on this k3s/containerd version is **unmeasured**. If it is not, + the mount will succeed, the group will be present, and the `open()` will still fail `EPERM` — and + the design would need the device plugin (which sets the cgroup entry) rather than a hostPath. The + first real consumer proves or disproves this; nothing in this PR does. +- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost + of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media + apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no + metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire + is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`, + where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator + seeing it in the application's own dashboard. +- **Whether GID 992 survives a node rebuild.** It was measured on all three prod nodes on + 2026-09-30 and was identical on each, but `render` is a *systemd dynamic* group allocation, not a + Debian/LSB static GID. A reinstall that creates system groups in a different order can move it, + and nothing in this repo would notice. Recorded in the constant's comment; re-measure `getent + group render` on every node after any rebuild. +- **Whether `supplementalGroups` is the load-bearing mechanism or belt-and-braces.** A concurrent + agent was checking whether the pms-docker image's `45-plex-hw-transcode-and-connected-tuner` init + hook resolves the render group dynamically at startup. **No finding reached me**, so the constant's + comment describes the GID and its provenance without claiming which mechanism does the work. + Update that comment if the answer arrives. +- **Anything on the live cluster.** I ran no `kubectl`. The node facts above (device present, + `root:render` 0660, GID 992, `i915` bound, no device plugin, no `gpu.intel.com/*` advertised) and + the Kyverno posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups` + restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by + me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried. +- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing — + no gate stage invokes the renderer's self-test. They are the executable spec and house style, and + I ran them by hand, but a future regression in this code path **will not be caught by the gate**. + Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a + sensitive path and would drag in a human reviewer for a change unrelated to this one). +- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only; + `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI. diff --git a/schemas/README.md b/schemas/README.md index 6847b69..20bddc9 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f | `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. | | `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. | -**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change: +**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change: propose on the task board, update dependent renderers/tests in the same change, never silently break a consumer (coordination-protocol §2). @@ -35,6 +35,9 @@ consumer (coordination-protocol §2). - `ingressRoutes[].host` must match `