--- date: 2026-09-30 branch: feat/node-health-render-device title: The Intel render device is now self-re-measuring, not a fact someone checked once --- ## What changed ### Why this probe has value at all **Nothing else in the platform would notice the device going missing.** Plex exposes no ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node *without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU. This check is the only thing in the system that would surface it, which is the actual justification for the role carrying it. ### The change `roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working Intel render device (measured by hand on all three on 2026-09-30) and the point of this change is that the measurement now **re-runs forever** instead of decaying silently in a handoff note. The failure this exists to catch is silent by construction. Plex hardware-transcodes through `/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls back to software transcoding on its own and the only symptom is a hot CPU. Four files: - `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged** pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`, `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the existing tool probe; both are optional and neither is depended on. - `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device` (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack` (INFO/SKIP). - `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the check makes is data rather than task logic. - `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and `roles/node_health/README.md` gained a section explaining the design and two new "known limits". Three design rulings worth stating, because each is the kind of thing a later edit would get wrong: 1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**, mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats the device and joins whichever group owns it, so drift genuinely is not a fault. The observed group name and number appear in the detail line, which makes drift *visible* without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall. 2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on the bus and there is still no render node" is a **FAIL**. 3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality* only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver inside its container. Grading either would put a permanent WARN on a chosen configuration, or invite someone to `apt install vainfo` on a node to clear a health check, and this role installs nothing. `xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is bound to nothing. ## What I verified ### The real prod run (read-only, all three nodes) `--list-hosts` first, every time. It named exactly three hosts and nothing else: play #1 (k3s_prod): Inspect node hardware health (read-only) pattern: ['k3s_prod'] hosts (3): k3s-prod-3 k3s-prod-2 k3s-prod-1 Then `--syntax-check` (rc 0), then the run: ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 skipped=1 k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 PLAY rc=0 (The `changed=` counts are the *controller-side* artifact writes under `infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed — the collector task is `changed_when: false` and the role touches nothing.) All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall: k3s-prod-1 VERDICT 0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO k3s-prod-2 VERDICT 0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO k3s-prod-3 VERDICT 0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*) [PASS] graphics.drm_driver — DRM driver bound to the display device (expected i915) [INFO] graphics.media_stack — Media engines, GuC/HuC firmware and host VAAPI userspace (informational) (The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not mine; none of them moved.) The new report section, **byte-identical on all three nodes**: -------------------------------------------------------------------------------- GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) -------------------------------------------------------------------------------- pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02) driver i915 drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0 dev card0 character special mode 0660 root:video gid 44 dev renderD128 character special mode 0660 root:render gid 992 GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not readable without root (i915 module parameters are mode 0400) no i915/xe parameter on the kernel cmdline modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides) vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128` character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode `0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)` CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`. **One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in `node_baseline_packages` and depending on an undeclared package would be wrong — but the comments and README that asserted it was missing were factually wrong and are now corrected. Its presence is why the `pci` line above carries a readable device name rather than bare IDs. ### The gate DEV_KUBECONFIG=/dev/null ENGINE= ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' GATE rc=3 **rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly three stages did not run, all three the known local ceiling from a worktree, and **no fourth skip**: [skip] spec-immutability spec repo not present at /work/../home-network-k3s (set SPEC_REPO) [skip] gitea-settings SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule [skip] conformance no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN [ ok ] decisions-log: pass [ ok ] lint: pass [ ok ] l0: pass [ ok ] render-golden: pass [ ok ] l1: pass [ ok ] approval-policy: pass Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null` (the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I know the shim took effect rather than the git stages silently degrading. `ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped `ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not claiming to have matched a lint convention that does not exist. The gate's `lint` stage passed with one `line too long` **warning** on `roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well above my insertion point, and warnings do not fail the stage. ### Offline work (before the lock was released) - `python3 -m py_compile` on the collector: clean. - `collect_graphics()` executed for real against this workstation's hardware (an unrelated machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole structure with no exception, correctly reporting two cards, two render nodes, the bound driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable without root* rather than as a fabricated `0`. - That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports `NOT READ — ` and `/proc/cmdline` is read as the world-readable second source. An earlier draft would have silently reported a null. - Two other bugs found and fixed by running it rather than reasoning about it: `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be excluded from the card scan. - All three checks and the new template section were rendered through a standalone Jinja harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real workstation; GPU present with the render node removed; no display device at all; `xe` bound in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`, `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches Ansible's `trim_blocks=True`. **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated dictionaries, not machines. They are not hardware tests and must not be written up as if they were. ## What I did NOT verify - **The FAIL path has never fired on a real node.** Every prod node has a working render device, and this role is read-only, so a device cannot be taken away to prove the check catches its absence. The failure branches were exercised against **synthetic fact sets only** (see above). I am not claiming the failure path is tested against hardware. - **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of `/sys/class/drm/card*` under it is something I have assumed matches i915's (same `device/driver` symlink, same `engine/*` directories). That assumption is unverified. - **The media-engine sysfs path is confirmed on these nodes but not proven portable.** `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it is measured here. I did **not** confirm against kernel source that no other kernel version or driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO line, not a false verdict, which is why this failure mode is tolerable. - **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes; I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to guess it. Reading it would need root, and it is INFO either way, so I left it alone. - **Not a transcode test.** These checks prove the render *device* exists, is a character device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a node-level check that installs nothing. - **The gate did not establish greenness.** It returned **rc 3**, and three stages did not run (`spec-immutability`, `gitea-settings`, `conformance`). So: the offline L0/L1 subset passed, the live Gitea branch-protection rule is **UNVERIFIED** from this run, and L2/L3 conformance did not execute. I am not describing this PR as green. - **No claim about the live platform beyond what the run printed.** Per CLAUDE.md §2b this repo is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. I did not query Gitea's branch protection or runner state, so I assert nothing about either. - **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer exists. That is real, it is out of this change's scope, and it is unclaimed work.