jakman85andClaude Opus 5
78bf68493d
The hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on
k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container
(`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
`uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE
CGROUP, not EACCES from file permissions, proven three ways: root on the HOST
opens the same device rc=0; the node is unified cgroup2fs; and the container's
runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]`
with `linux.devices: None` -- one deny-all rule, no allow entry for major 226.
A hostPath is a filesystem operation: it binds the device node in but adds no
cgroup allow rule, because Kubernetes cannot infer one from a volume.
The Intel GPU device plugin produces that allow rule
(`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`).
It shipped as #36 and is live: all three prod nodes advertise
`gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01).
ZERO ServiceDefinition field changes, which was the entire point of making the
field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s
`hardware:` block comes through byte-identical; only its comment changes, because
it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather
than claimed.
- catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for
`QUICKSYNC_RESOURCE`; the hardware branch now sets
`resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a
sidecar) and emits no volume, no mount, no securityContext, no nodeSelector.
`limits` not `requests` is mandatory: Kubernetes mirrors requests from limits,
and a requests-only extended resource is invalid. This is the only app-container
`resources` render path and is deliberately narrow -- no general CPU/memory
feature was invented.
- selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the
no-`hardware` negative extended to `resources`; the podSecurityContext
pass-through and the no-synthesised-securityContext assertions kept; one new
assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested:
8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath
re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource
name, quantity as a string, limit leaked onto the sidecar).
- apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block
added. Only golden touched; keycloak/vm-demo unchanged.
THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing ->
`DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY
software-transcoded (invisible, harmless to everything else). After: resource
unavailable (plugin broken, removed, or not yet reconciled) -> Plex is
UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it,
so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard
scheduling constraint, and it is also the only thing that produces the cgroup
allow rule. There is no graceful-degradation variant. Plex's schedulability is now
coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code
change: delete the `hardware:` block from catalog/plex.yaml and re-render.
Documented in the contract, the runbook (its own top-of-page section and a
"Plex will not schedule" procedure) and the handoff.
HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test
cannot be re-run until this merges and Flux reconciles, and the Plex-side
`HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the
manifest renders.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
apps
App-tier manifests, rendered from catalog/*.yaml ServiceDefinitions by catalog/render.py
and split into dependency-ordered waves (wave-<N>/).
- Each
wave-<N>/is reconciled by its own Flux Kustomizationapps-wave-<N>(clusters/base/apps-wave-<N>.yaml), whichdependsOnwave N-1 (wave 0 dependsOnplatform-config) and health-gates (wait: true) ONLY that wave. A crashlooping workload in a later wave can no longer stall an earlier wave or block new revisions. wave-<N>/kustomization.yamlandclusters/base/apps-wave-<N>.yamlare generated — runpython3 catalog/render.py --emit-wavesafter onboarding a service; do not hand-edit them.- Wave assignment = longest dependency-path depth over the catalog
dependencies:graph; a co-located CNPG database<svc>-dbis a leaf and always lands strictly below its consumer.
See ../DESIGN-waves.md for the wave table, Flux topology, and the
cutover runbook.