--- date: 2026-09-30 branch: feat/node-health-render-device title: The Intel render device is now self-re-measuring, not a fact someone checked once --- ## What changed ### Why this probe has value at all **Nothing else in the platform would notice the device going missing.** Plex exposes no ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node *without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU. This check is the only thing in the system that would surface it, which is the actual justification for the role carrying it. ### The change `roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working Intel render device (measured by hand on all three on 2026-09-30) and the point of this change is that the measurement now **re-runs forever** instead of decaying silently in a handoff note. The failure this exists to catch is silent by construction. Plex hardware-transcodes through `/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls back to software transcoding on its own and the only symptom is a hot CPU. Four files: - `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged** pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`, `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the existing tool probe; both are optional and neither is depended on. - `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device` (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack` (INFO/SKIP). - `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the check makes is data rather than task logic. - `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and `roles/node_health/README.md` gained a section explaining the design and two new "known limits". Three design rulings worth stating, because each is the kind of thing a later edit would get wrong: 1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**, mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats the device and joins whichever group owns it, so drift genuinely is not a fault. The observed group name and number appear in the detail line, which makes drift *visible* without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall. 2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on the bus and there is still no render node" is a **FAIL**. 3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality* only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver inside its container. Grading either would put a permanent WARN on a chosen configuration, or invite someone to `apt install vainfo` on a node to clear a health check, and this role installs nothing. `xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is bound to nothing. ## What I verified ### The real prod run (read-only, all three nodes) `--list-hosts` first, every time. It named exactly three hosts and nothing else: play #1 (k3s_prod): Inspect node hardware health (read-only) pattern: ['k3s_prod'] hosts (3): k3s-prod-3 k3s-prod-2 k3s-prod-1 Then `--syntax-check` (rc 0), then the run: ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 skipped=1 k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 PLAY rc=0 (The `changed=` counts are the *controller-side* artifact writes under `infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed — the collector task is `changed_when: false` and the role touches nothing.) All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall: k3s-prod-1 VERDICT 0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO k3s-prod-2 VERDICT 0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO k3s-prod-3 VERDICT 0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*) [PASS] graphics.drm_driver — DRM driver bound to the display device (expected i915) [INFO] graphics.media_stack — Media engines, GuC/HuC firmware and host VAAPI userspace (informational) (The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not mine; none of them moved.) The new report section, **byte-identical on all three nodes**: -------------------------------------------------------------------------------- GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) -------------------------------------------------------------------------------- pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02) driver i915 drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0 dev card0 character special mode 0660 root:video gid 44 dev renderD128 character special mode 0660 root:render gid 992 GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not readable without root (i915 module parameters are mode 0400) no i915/xe parameter on the kernel cmdline modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides) vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128` character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode `0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)` CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`. **One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in `node_baseline_packages` and depending on an undeclared package would be wrong — but the comments and README that asserted it was missing were factually wrong and are now corrected. Its presence is why the `pci` line above carries a readable device name rather than bare IDs. ### The gate DEV_KUBECONFIG=/dev/null ENGINE= ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' GATE rc=3 **rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly three stages did not run, all three the known local ceiling from a worktree, and **no fourth skip**: [skip] spec-immutability spec repo not present at /work/../home-network-k3s (set SPEC_REPO) [skip] gitea-settings SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule [skip] conformance no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN [ ok ] decisions-log: pass [ ok ] lint: pass [ ok ] l0: pass [ ok ] render-golden: pass [ ok ] l1: pass [ ok ] approval-policy: pass Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null` (the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I know the shim took effect rather than the git stages silently degrading. `ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped `ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not claiming to have matched a lint convention that does not exist. ### The CI gate on the PR — a green tick that is exit 3, read from the log The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"** on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with "Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s. **But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is `run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this Gitea, the web route serves it) and read the gate's own headline: [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. [fail] Stages that did not run: spec-immutability **So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea check is the tolerance swallowing that 3, not a rc 0. **CI verified two things my local run could not, and both came back green:** [ ok ] stage gitea-settings: pass [ ok ] enable_push = false enable_status_check = true [ ok ] required_approvals = 1 block_admin_merge_override = true [ ok ] block_on_rejected_reviews = true dismiss_stale_approvals = true [ ok ] runners online: 1 (>= 1) [ ok ] Gitea settings match the declaration. [ ok ] stage approval-policy: pass [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver. My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give `approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip there would have failed the job. That is measured confirmation that the live Gitea matches `infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too (`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well. Corroborated independently: `GET /repos/.../branches/main` → `protected=true user_can_push=false enable_status_check=true required_approvals=1`, contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind status=online`. And note the commit's **combined** state is `pending`, not success, because `gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is not in the required list, so it does not block, but the PR is not "all green" either. The gate's `lint` stage passed with one `line too long` **warning** on `roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well above my insertion point, and warnings do not fail the stage. ### Offline work (before the lock was released) - `python3 -m py_compile` on the collector: clean. - `collect_graphics()` executed for real against this workstation's hardware (an unrelated machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole structure with no exception, correctly reporting two cards, two render nodes, the bound driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable without root* rather than as a fabricated `0`. - That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports `NOT READ — ` and `/proc/cmdline` is read as the world-readable second source. An earlier draft would have silently reported a null. - Two other bugs found and fixed by running it rather than reasoning about it: `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be excluded from the card scan. - All three checks and the new template section were rendered through a standalone Jinja harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real workstation; GPU present with the render node removed; no display device at all; `xe` bound in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`, `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches Ansible's `trim_blocks=True`. **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated dictionaries, not machines. They are not hardware tests and must not be written up as if they were. ## What I did NOT verify - **The FAIL path has never fired on a real node.** Every prod node has a working render device, and this role is read-only, so a device cannot be taken away to prove the check catches its absence. The failure branches were exercised against **synthetic fact sets only** (see above). I am not claiming the failure path is tested against hardware. - **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of `/sys/class/drm/card*` under it is something I have assumed matches i915's (same `device/driver` symlink, same `engine/*` directories). That assumption is unverified. - **The media-engine sysfs path is confirmed on these nodes but not proven portable.** `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it is measured here. I did **not** confirm against kernel source that no other kernel version or driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO line, not a false verdict, which is why this failure mode is tolerable. - **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes; I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to guess it. Reading it would need root, and it is INFO either way, so I left it alone. - **Not a transcode test.** These checks prove the render *device* exists, is a character device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a node-level check that installs nothing. - **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not describing this PR as green**, and the green Gitea tick must not be read as one: the workflow line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3. - **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed. - **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my worktree nor the CI workspace, so nothing confirms this change does not drift from the spec. It is the expected, tolerated skip — but it is still a skip, not a pass. - **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. Branch protection and runner state I did query directly, and the CI `gitea-settings` stage passed — so those two I assert on measurement, not on this file's prose. - **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer exists. That is real, it is out of this change's scope, and it is unclaimed work.