diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 2928eaa..3962837 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -137,14 +137,15 @@ spec: secretKeyRef: name: plex-claim key: PLEX_CLAIM + resources: + limits: + gpu.intel.com/i915: 1 volumeMounts: - name: config mountPath: /config - name: media mountPath: /media readOnly: true - - name: dri - mountPath: /dev/dri volumes: - name: config persistentVolumeClaim: @@ -152,10 +153,6 @@ spec: - name: media persistentVolumeClaim: claimName: plex-media - - name: dri - hostPath: - path: /dev/dri - type: DirectoryOrCreate --- # Service apiVersion: v1 diff --git a/catalog/plex.yaml b/catalog/plex.yaml index 0b14b29..85b697e 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -81,46 +81,49 @@ env: # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443". # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02). -# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a -# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath -# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container — -# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md -# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin -# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not -# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum. +# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an +# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource. +# How the platform hands the device over is `catalog/render.py`'s business and is documented in +# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment. +# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath +# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change +# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the +# whole point of the enum. # # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`, # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines -# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but -# that is scheduler history rather than a requirement — this field deliberately renders no -# nodeSelector, because placement is a separate concern with its own justification. +# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all +# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no +# nodeSelector; placement is a separate concern with its own justification. +# +# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost, +# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is +# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because +# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER. +# The previous hostPath mechanism degraded silently to software transcoding instead — but it never +# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep. +# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to +# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md` +# carries that procedure for whoever needs it at 2am. # # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this. # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push -# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING -# CONTAINER, 2026-09-30: the image's own -# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under -# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and +# the host `render` GID in as a supplemental group. It cannot work here, and this survived the +# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the +# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device +# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host -# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via -# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather -# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which -# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan -# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the -# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod -# securityContext for this field precisely so nobody adds an inert one back later. +# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way +# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with +# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job. +# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from +# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold — +# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever +# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the +# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later. # -# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the -# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex -# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because -# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app -# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is -# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode -# metric, and no alert that could ever fire. The only proof is an operator reading the Plex -# dashboard. -# -# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does +# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`, @@ -128,8 +131,7 @@ env: # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`, # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI. -# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the -# device-cgroup failure mode to expect first. +# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took. hardware: gpu: intel-quicksync exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET) diff --git a/catalog/render.py b/catalog/render.py index 3d2db72..1cc339f 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -166,11 +166,14 @@ OAUTH2_PROXY_METRICS_PORT = 44180 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------ # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is -# deliberately confined to this file. Swapping to the Intel GPU device plugin -# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and -# nowhere else -- no catalog entry moves. -DRI_VOLUME_NAME = "dri" -DRI_DEVICE_DIR = "/dev/dri" +# deliberately confined to this file. This file is the ONLY place the mechanism appears, +# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel +# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits. +# +# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet +# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three +# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`. +QUICKSYNC_RESOURCE = "gpu.intel.com/i915" # ============================================================================= @@ -526,41 +529,59 @@ def r_deployment(s: dict) -> dict: # workload needs the host iGPU's render node for hardware video transcode. The SD says # only that; the wiring below is the platform's current mechanism for satisfying it. # - # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to - # `Directory`. A node that comes back without the render device (rebuild, BIOS change, - # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding, - # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a - # failed hostPath mount stalls the entire app tier for the sake of a feature that is not - # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in - # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty - # directory the kubelet creates on such a node leaves nothing durable behind. + # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT + # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the + # original implementation (merged 2026-09-30) and it was proven broken in the live Plex + # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container + # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened -- + # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP, + # not EACCES from file permissions, proven three ways: root on the HOST opens the same + # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json` + # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with + # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is + # a filesystem operation: it binds the device node in but adds NO cgroup allow rule, + # because Kubernetes cannot infer one from a volume. # - # The mount is deliberately NOT readOnly: VA-API opens the render node read-write. + # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source: + # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the + # kubelet turns into both the device injection and the matching cgroup allow. + # + # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be + # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a + # requests-only extended resource is invalid. App container only, never a sidecar -- the + # oauth2-proxy sidecar does no transcoding. + # + # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and + # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir, + # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else). + # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the + # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1` + # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an + # extended resource is a hard scheduling constraint, and it is also the only thing that + # produces the cgroup allow rule. Plex's schedulability is now coupled to the + # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all. + # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from + # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier + # unblocks. runbooks/plex-quicksync.md carries it. # # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it - # would be a regression, not a hardening. MEASURED in the running container on prod-2, - # 2026-09-30: the image's own cont-init hook + # would be a regression, not a hardening. MEASURED in the running container on prod-2 + # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the # container when none exists, and `usermod -a -G` the app user into it -- every start, - # nothing hardcoded. cont-init.d completes before services.d, so the account database - # s6 reads at exec time already carries the group. The service is then started with - # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from - # /etc/group rather than appending to it: `Plex Media Server` runs with - # `Uid: 1000 Gid: 1000 Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does - # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is - # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the - # group handling is the image's job and it does it correctly. A supplementalGroups entry - # here would be inert AND would mislead whoever debugs this next. + # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which + # is why this swap needs no group work either. cont-init.d completes before services.d, + # so the account database s6 reads at exec time already carries the group: measured live, + # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render` + # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid` + # REPLACES the process group list from /etc/group rather than appending to it, so + # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded + # before the app ever starts. A supplementalGroups entry here would be inert AND would + # mislead whoever debugs this next. hw_gpu = (s.get("hardware") or {}).get("gpu") if hw_gpu == "intel-quicksync": - volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR}) - volumes.append( - { - "name": DRI_VOLUME_NAME, - "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"}, - } - ) + container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}} if volume_mounts: container["volumeMounts"] = volume_mounts @@ -2579,10 +2600,13 @@ def _selftest() -> int: "an SD without podSecurityContext must not gain a pod securityContext", ) - # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount - # on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these - # assertions pin the MECHANISM that currently satisfies it -- including what it - # deliberately does NOT emit -- so a change to it is a deliberate, visible edit. + # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container + # and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector. + # The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently + # satisfies it -- including what it deliberately does NOT emit -- so a change to it + # is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is + # asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see + # the emission site), and the absence assertion is what stops it coming back. hw_docs = render( { "name": "hw", @@ -2593,24 +2617,36 @@ def _selftest() -> int: hw_pod = next( d[1] for d in hw_docs if d[1]["kind"] == "Deployment" )["spec"]["template"]["spec"] + hw_app = hw_pod["containers"][0] + # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set + # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended + # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a + # CPU/memory resources feature through this field (that would be scope creep, and this + # field is about the DEVICE only). check( - { - "name": "dri", - "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"}, - } - in hw_pod.get("volumes", []), - "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume " - "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)", + hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}}, + "hardware.gpu=intel-quicksync must render resources.limits " + "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources", ) - hw_mounts = hw_pod["containers"][0].get("volumeMounts", []) + # The retired mechanism must stay retired. A hostPath binds the device node in but adds + # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible. check( - {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts, - "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container", + not any( + "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", []) + ) + and not any( + m.get("mountPath") == "/dev/dri" or m.get("name") == "dri" + for m in hw_app.get("volumeMounts", []) + ), + "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no " + "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup " + "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)", ) - # The device mount is the WHOLE feature: no pod securityContext is invented. The image's - # own cont-init hook adds the app user to the device's group inside the container, and - # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the - # emission site). Asserting the absence is what stops someone adding an inert one back. + # The resource is the WHOLE feature: no pod securityContext is invented. The image's own + # cont-init hook adds the app user to the (plugin-injected) device's group inside the + # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway + # (see the emission site). Asserting the absence is what stops someone adding an inert + # one back. check( "securityContext" not in hw_pod, "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert " @@ -2618,10 +2654,12 @@ def _selftest() -> int: ) # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce) # rejects `privileged: true` in an app namespace, and placement is a separate concern - # with its own justification -- this field is about the DEVICE only. + # with its own justification -- this field is about the DEVICE only. (The extended + # resource now constrains placement to nodes the plugin advertises, which is the + # scheduler's own doing, not a nodeSelector this renderer writes.) check( "privileged" not in hw_pod.get("securityContext", {}) - and "securityContext" not in hw_pod["containers"][0] + and "securityContext" not in hw_app and "nodeSelector" not in hw_pod, "hardware.gpu must emit no privileged securityContext and no nodeSelector", ) @@ -2634,8 +2672,10 @@ def _selftest() -> int: check( "volumes" not in nohw_pod and "volumeMounts" not in nohw_pod["containers"][0] + and "resources" not in nohw_pod["containers"][0] and "securityContext" not in nohw_pod, - "an SD without `hardware` must gain no dri volume, no mount and no securityContext", + "an SD without `hardware` must gain no resources block, no volume, no mount and no " + "securityContext", ) # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the # hardware field neither adds to it nor rewrites it. @@ -2652,9 +2692,35 @@ def _selftest() -> int: if d[1]["kind"] == "Deployment" )["spec"]["template"]["spec"] check( - hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}, + hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]} + and hwm_pod["containers"][0].get("resources") + == {"limits": {"gpu.intel.com/i915": 1}}, "hardware.gpu must leave the SD's own podSecurityContext untouched", ) + # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy + # sidecar, which does no transcoding and must not consume one of the node's i915 slots. + hws_containers = next( + d[1] + for d in render( + { + "name": "hws", + "source": {"image": "a:1"}, + "hardware": {"gpu": "intel-quicksync"}, + "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}], + "auth": {"mode": "forward-auth", "groups": ["media-users"]}, + } + ) + if d[1]["kind"] == "Deployment" + )["spec"]["template"]["spec"]["containers"] + check( + len(hws_containers) == 2 + and hws_containers[0].get("resources") + == {"limits": {"gpu.intel.com/i915": 1}} + and "gpu.intel.com/i915" + not in hws_containers[1].get("resources", {}).get("limits", {}), + "hardware.gpu must put the i915 limit on the APP container only, never on the " + "oauth2-proxy sidecar", + ) # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that # fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged. diff --git a/runbooks/plex-quicksync.md b/runbooks/plex-quicksync.md index 4ccd4d5..8c59a03 100644 --- a/runbooks/plex-quicksync.md +++ b/runbooks/plex-quicksync.md @@ -5,10 +5,24 @@ not an incident page. It is indexed here because the feature it completes is **i clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists to prevent. **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the -rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the -mechanism), `schemas/renderer-contract.md` `### hardware` (the contract). +rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the +mechanism), `schemas/renderer-contract.md` `### hardware` (the contract), +`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource). **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile. +## STOP HERE IF PLEX IS `Pending` / `Unschedulable` + +That is this feature's **one serious failure mode** and it **stalls the whole app tier** +(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the +extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the +`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not +start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on +purpose. + +**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`, +re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands +are in **"Plex will not schedule" → remedy B**. + ## What this means here Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync @@ -18,7 +32,7 @@ is in git**: | Half | Who does it | State | |---|---|---| -| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. | +| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. | | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. | **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded @@ -30,10 +44,44 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this; there isn't one, and the search has already been done. -What the device mount does **not** do, so nobody reads more into a green gate than is there: it does -not enable hardware transcoding, it does not prove the device can be *opened* from inside a -non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any -of the three nodes and all three carry the device. +### Why a resource request and not a `/dev/dri` hostPath + +Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that +will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on +**2026-10-01**: the device node was plainly **visible** in the container +(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** — + +| attempt | result | +|---|---| +| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" | +| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" | + +`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical +from the outside and are not the same problem. Proven three ways: root on the **host** opened the +same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read +`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one +deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the +device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume. + +Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root +does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno +`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it. + +The device plugin is what produces the allow rule — in the v0.37.0 source, +`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns +into both the device injection and the matching cgroup allow. + +**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran +and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing +*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast +radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is +also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. +The one improvement is that the loss is no longer silent. + +What this does **not** do, so nobody reads more into a green gate than is there: it does not enable +hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest +renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin +advertises, which is the scheduler's doing and not a selector in this repo. ## Running the commands in this runbook @@ -51,15 +99,46 @@ read of that file below greps a single named key — do not widen one. ## First commands -**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.) +**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide' +``` + +`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on. + +**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the +hostPath mounted" check, and it is now the first thing that can be wrong. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -o json \ + | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""' +``` + +Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that +node cannot run Plex. Then the DaemonSet behind it: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin' +``` + +Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a +symptom — fix it there, in `platform/intel-gpu-plugin/`, not here. + +**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the +GitOps half landed: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ - -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo' + -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo' ``` -`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare +A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the +mechanism that does not work. Compare ``` infra/devcontainer/dev.sh run bash -c \ @@ -69,34 +148,29 @@ infra/devcontainer/dev.sh run bash -c \ against the merge commit before blaming anything else. -**2. Is the device actually visible inside the container?** +**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test +of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri' ``` -Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory** -here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node -had none. That is the designed-safe outcome — the app tier still reconciles — but it means software -transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU: - -``` -infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide' -``` - -**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2. +Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any +`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running +to exec into. Then the open, **as the app user**: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ - bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"' + s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"' ``` -`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible -in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from -the outside and are not the same problem. +`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must +not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a +cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly +visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely +causes" #2. **4. Is the toggle on?** Named key only — never the whole file: @@ -172,25 +246,32 @@ When the dashboard still says plain `Transcode`: First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before** investigating anything technical. -2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining - technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted - character device on this k3s/containerd build **has not been established**; it can only be settled - by deploying and trying. The container inits as uid 0, which settles *file* permissions, but - **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and - correctly owned, step 3 returns `RW-OPEN-DENIED`. - **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a - resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than - a device path, that migration is a **`catalog/render.py` change plus a re-render — with no - ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today. - **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in - **Enforce** mode and rejects it in app namespaces. Do not reach for it. +2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom: + first-command #3 shows `renderD128` present and correctly owned, and the open returns + `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not + `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the + design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath"). + If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm + the pod really did consume the resource -3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod - nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a - node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app - tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:** - `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a - feature that is not load-bearing. + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ + -o jsonpath="{.items[0].spec.containers[0].resources}"; echo' + ``` + + and then check the plugin pod on Plex's node (first-command #1). **Do not reach for + `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0 + failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**; + that is the mechanism this replaced, and it was measured not to work. + +3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not** + present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex + either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not + as silent software transcoding: see "Plex will not schedule". All three prod nodes carried + `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on + 2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression. + **Fix the node.** 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the mechanism this feature leans on with no commit in this repo. @@ -203,6 +284,56 @@ When the dashboard still says plain `Transcode`: Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some sessions show `(hw)` and others do not. Not a fault. +## Plex will not schedule + +**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds +`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at +2am. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20' +``` + +`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is +advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin` +DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the +device-cgroup allow rule, so there is no variant of this feature that degrades gracefully. + +**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the +nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is +where to fix it. Check its Flux Kustomization reconciled: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin' +``` + +A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not. + +**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no +code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would +revert you): + +1. delete these two lines from `catalog/plex.yaml`: + + ```yaml + hardware: + gpu: intel-quicksync + ``` + +2. re-render the golden: + + ``` + python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation + ``` + + The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block. +3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks. + +Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the +`/config` PVC is affected either way — do **not** delete the PVC as part of this. + ## The floating-image caveat Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app @@ -243,7 +374,12 @@ hook exists. `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is discarded before Plex starts. `render.py --selftest` asserts that absence deliberately. - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will - reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2). + reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement, + because root does not bypass the cgroup device gate. +- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That + was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM` + with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what + grants access; a volume cannot. - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change @@ -253,9 +389,9 @@ hook exists. - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to lose the DB. -- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The - next step is a platform change — the Intel device plugin — with its own PR, not more poking at this - workload. +- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2 + confirmed) *under the current mechanism*. The next step is a platform change in + `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload. ## Known gaps, deliberately not fixed here @@ -279,5 +415,8 @@ Recorded so they are findable, and so nobody assumes they were overlooked: Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/` The node measurements (all three prod nodes) and the in-container image measurements cited here were taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…` -(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time -of writing, and the Plex-side toggle was still unset. +(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000` +reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**. +Hardware transcoding has **still not been observed working**: the open test has not been re-run under +the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side +toggle is still unset. diff --git a/schemas/renderer-contract.md b/schemas/renderer-contract.md index a560c96..d503b36 100644 --- a/schemas/renderer-contract.md +++ b/schemas/renderer-contract.md @@ -165,56 +165,115 @@ platform must render, and cites the requirement each mapping satisfies. Changing (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The ServiceDefinition says *what class of host device this workload needs*; how the platform hands it over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw - `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the - Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero** - catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely - different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one. -- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things: - 1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**; - 2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the - render node read-write, so it is deliberately not `readOnly`). + `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was + swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource — + as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of + per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical. + Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a + second route to the same one. +- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**: - **Mounting the device is sufficient. That is the whole mapping.** + resources: + limits: + gpu.intel.com/i915: 1 + + **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`, + no privileged anything. + - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be + expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a + requests-only extended resource is rejected as invalid. + - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`, + its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned + `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report + `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available. + - This is the **only** app-container `resources` render path in the file, and it is deliberately + narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that + is a separate field with a separate justification, and `render.py --selftest` asserts the emitted + dict *exactly* so the attempt fails loudly. +- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first + implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write + mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The + device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and + **could not be opened**: + + | attempt | result | + |---|---| + | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" | + | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" | + + `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look + identical from the outside and are not the same problem. Proven three ways: root on the **host** + opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc + `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with + `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a + *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because + Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass + the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`: + Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it. + The device plugin is what produces the allow rule — confirmed in the v0.37.0 source, + `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns + into both the device injection and the matching cgroup allow. +- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated + plainly rather than sold: + + | | device/resource unavailable ⇒ | + |---|---| + | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. | + | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. | + + There is **no graceful-degradation variant**: an extended resource is a hard scheduling + constraint, and it is also the only thing that produces the cgroup allow rule. The workload's + schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken, + removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware + transcoding working at all. + - **Escape hatch, one field and no code change:** delete the `hardware:` block from the + ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software + transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class + procedure, because it is what someone will need at 2am. + - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no + longer silent. There is still no metric and no alert (media apps here render no + `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas + "Plex is quietly software-transcoding" never was. - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a - regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious + regression rather than a hardening.** This survives the mechanism swap unchanged, and it was + re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious move is to push the host `render` GID in — and it does not work. **Measured in the running - container on prod-2, 2026-09-30:** the image's own + container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when - none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. - `cont-init.d` completes before `services.d`, so the account database is already correct when the - service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid` - **replaces** the process group list from `/etc/group` rather than appending to it: - `Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which - PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches - `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead - whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`. + none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does + exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this + swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is + already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and + `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it — + measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host + `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A + Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever + starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its + own absence in `render.py --selftest`. - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing. No GID is pinned anywhere in this repo. - - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is - sufficient" was established for an image that resolves the device group itself. A FUTURE - consumer that runs non-root and has no equivalent init hook would still need the group solved - somehow — and the answer for that case is a **renderer** change (per-image handling, or the - device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The - first such consumer must re-measure rather than assume this bullet covers it. -- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back - without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the - workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and - `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake - of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates - on such a node leaves nothing durable behind. - - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload - software-transcodes and nothing reports it. There is no metric to alert on — media apps here - render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse - than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is - an operator observing it in the application's own dashboard. + - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was + established for an image that resolves the device group itself. A FUTURE consumer that runs + non-root and has no equivalent init hook would still need the group solved somehow — and the + answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition + field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works. + The first such consumer must re-measure rather than assume this bullet covers it. - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno - `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node - does not need it); no `nodeSelector` (placement is a separate concern with its own justification — - this field is about the device, not about where the pod lands); no resource requests/limits (there - is no app-container resource render path, and inventing one here would be scope creep). -- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical. + `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have + helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its + own justification — this field is about the device, not about where the pod lands; the extended + resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's** + doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the + `resources` block carries the extended resource and nothing else). +- **Historical, kept so a reader recognises it rather than reintroduces it:** + `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is + what made a GPU-less node degrade to software transcoding instead of failing the mount and + stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since + the mechanism it protected never granted access in the first place. +- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical + (no `resources` key, no volume, no mount, no `securityContext`). Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`. - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need). @@ -344,11 +403,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it. Derived from `exposure` alone; enforced on every gate run by `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03). -8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no - `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext` - either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the - `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's - implementation can be replaced without touching a single ServiceDefinition. +8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED, + not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no + volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only** + (never a sidecar) and no `securityContext` either. All directions are proven by + `render.py --selftest`, and each assertion is mutation-tested. The resource name + `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the + capability's implementation can be replaced without touching a single ServiceDefinition — which + is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath + (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with + **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now + **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is + to delete the `hardware:` block and re-render. ## Requirement traceability (contract-level) Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02