home-network-k3s-impl/decisions/2026-09-30-node-health-render-device.md

date, branch, title

date	branch	title
2026-09-30	feat/node-health-render-device	The Intel render device is now self-re-measuring, not a fact someone checked once

What changed

Why this probe has value at all

Nothing else in the platform would notice the device going missing. Plex exposes no ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the manifest side mounts the device with hostPath type: DirectoryOrCreate, which means a node without the device mounts an empty directory — Plex starts, runs, and degrades quietly rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU. This check is the only thing in the system that would surface it, which is the actual justification for the role carrying it.

The change

roles/node_health grew a graphics / render-device probe. Nothing was installed, no module was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working Intel render device (measured by hand on all three on 2026-09-30) and the point of this change is that the measurement now re-runs forever instead of decaying silently in a handoff note.

The failure this exists to catch is silent by construction. Plex hardware-transcodes through /dev/dri/renderD128; if that node stops existing — node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls back to software transcoding on its own and the only symptom is a hot CPU.

Four files:

• files/node_health_collect.py — new collect_graphics(), wired into the unprivileged pass (main()). Everything it reads is world-readable: lstat() on /dev/dri/*, grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver symlink target for the bound driver, /sys/bus/pci/devices/*/class for display-class (PCI base class 0x03) devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the vcs*/vecs* media engines, /proc/modules, /proc/cmdline, and /sys/module/{i915,xe}/parameters/enable_guc. lspci and vainfo were added to the existing tool probe; both are optional and neither is depended on.
• tasks/assess.yml — one shared derive task plus three checks: graphics.render_device (PASS/FAIL/SKIP), graphics.drm_driver (PASS/WARN/FAIL/SKIP), graphics.media_stack (INFO/SKIP).
• defaults/main.yml — node_health_graphics_expected_driver: i915, so the one comparison the check makes is data rather than task logic.
• templates/summary.txt.j2 — a GRAPHICS / RENDER DEVICE section, and roles/node_health/README.md gained a section explaining the design and two new "known limits".

Three design rulings worth stating, because each is the kind of thing a later edit would get wrong:

1. The render node's GID is reported, never asserted. It read root:render, gid 992, mode 0660 on all three prod nodes. The check does not compare against 992. render is a systemd dynamic allocation (systemd-sysusers), not a Debian static gid from base-passwd, so a rebuild can legitimately move it — and Plex's container entrypoint stats the device and joins whichever group owns it, so drift genuinely is not a fault. The observed group name and number appear in the detail line, which makes drift visible without making it an error. A hard != 992 would be a false alarm waiting for a reinstall.
2. The PCI scan is deliberately separate from the DRM scan. /sys/class/drm/card* exists only once a driver has bound, so a GPU with nothing attached has no card0 and would otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately is what lets "no display device at all" be an honest SKIP (a VM guest — the same convention thermal.max already uses for "no sensor on this hardware") while "a GPU is on the bus and there is still no render node" is a FAIL.
3. GuC/HuC and host VAAPI userspace are INFO and must stay INFO. GuC/HuC off (i915.enable_guc=0) is an accepted state here and costs low-power H.264 encode quality only. No host vainfo is not a defect at all — Plex ships its own libva and media driver inside its container. Grading either would put a permanent WARN on a chosen configuration, or invite someone to apt install vainfo on a node to clear a health check, and this role installs nothing.

xe bound instead of i915 is a WARN, not a FAIL: the device still works, but the two drivers do not share a userspace media stack. On these nodes xe.ko.xz exists on disk and is bound to nothing.

What I verified

The real prod run (read-only, all three nodes)

--list-hosts first, every time. It named exactly three hosts and nothing else:

play #1 (k3s_prod): Inspect node hardware health (read-only)
    pattern: ['k3s_prod']
    hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1

Then --syntax-check (rc 0), then the run:

ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod

k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
PLAY rc=0

(The changed= counts are the controller-side artifact writes under infra/ansible/_artifacts/node-health/, delegated to localhost. Nothing on any node changed — the collector task is changed_when: false and the role touches nothing.)

All three nodes graded PASS / PASS / INFO on the new checks, 0 FAIL overall:

k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO

[PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
[PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
[INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)

(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not mine; none of them moved.)

The new report section, byte-identical on all three nodes:

--------------------------------------------------------------------------------
 GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
--------------------------------------------------------------------------------
 pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
              driver i915
 drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
 dev  card0        character special  mode 0660  root:video gid 44
 dev  renderD128   character special  mode 0660  root:render gid 992
 GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
              readable without root (i915 module parameters are mode 0400)
             no i915/xe parameter on the kernel cmdline
 modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
 vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container

That reproduces every item of the 2026-09-30 hand measurement independently: renderD128 character special, root:render, gid 992, mode 0660; card0 root:video gid 44 mode 0660; i915 bound; xe not loaded and bound to nothing; [8086:3e98] (rev 02) CoffeeLake-S GT2 UHD Graphics 630 at 00:02.0; vcs0 and vecs0 live; no host vainfo.

One measured correction to my own phase-1 assumption: lspci is present on all three prod nodes (/usr/bin/lspci — Debian ships pciutils by default). I had written that it was absent. It is still treated as optional, because pciutils is genuinely not declared in node_baseline_packages and depending on an undeclared package would be wrong — but the comments and README that asserted it was missing were factually wrong and are now corrected. Its presence is why the pci line above carries a readable device name rather than bare IDs.

The gate

DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
GATE rc=3

rc 3 — UNDETERMINED. NOT green. Nothing failed, and nothing is certified either. Exactly three stages did not run, all three the known local ceiling from a worktree, and no fourth skip:

[skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
[skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
[skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN

[ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
[ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass

Both CLAUDE.md §3 worktree workarounds were needed and both worked: DEV_KUBECONFIG=/dev/null (the credential in the main checkout was not copied) and an ENGINE shim that bind-mounts the main checkout's .git read-only. No tracked_files() traceback appeared, which is how I know the shim took effect rather than the git stages silently degrading.

ansible-lint was not run, because ci/ does not invoke it anywhere — I grepped ci/ and .gitea/ and the lint stage is yamllint -c ci/yamllint.yaml only. I am not claiming to have matched a lint convention that does not exist.

The CI gate on the PR — a green tick that is exit 3, read from the log

The required context gate / gate (pull_request) reported success, "Successful in 1m40s" on commit a78f010. That description matters: a skipped Gitea job reports success with "Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.

But the green tick does NOT mean gate.sh returned 0. .gitea/workflows/gate.yml:138 is run: ./ci/gate.sh || [ "$?" -eq 3 ] — the job tolerates rc 3 by design. So I fetched the job log (/actions/runs/173/jobs/0/logs; the /api/v1/.../tasks/{id}/logs endpoints 404 on this Gitea, the web route serves it) and read the gate's own headline:

[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
[fail]   Stages that did not run: spec-immutability

So CI is also rc 3, with exactly ONE stage skipped — spec-immutability. The green Gitea check is the tolerance swallowing that 3, not a rc 0.

CI verified two things my local run could not, and both came back green:

[ ok ] stage gitea-settings: pass
[ ok ]   enable_push = false          enable_status_check = true
[ ok ]   required_approvals = 1       block_admin_merge_override = true
[ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
[ ok ]   runners online: 1 (>= 1)
[ ok ] Gitea settings match the declaration.
[ ok ] stage approval-policy: pass
[ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.

My local run skipped gitea-settings (no SETTINGS_API_TOKEN) and could not give approval-policy a PR number. CI has the secret and REQUIRE_GITEA_SETTINGS: "true", so a skip there would have failed the job. That is measured confirmation that the live Gitea matches infra/gitea/expected-branch-protection.json, and that infra/ansible/** needs no human reviewer. lint passed over 405 files. conformance SKIPPED in CI too (CLUSTER_AVAILABLE != true), so L2/L3 remains unverified there as well.

Corroborated independently: GET /repos/.../branches/main → protected=true user_can_push=false enable_status_check=true required_approvals=1, contexts ["gate / gate (pull_request)"]; GET /admin/actions/runners → incluster-dind status=online. And note the commit's combined state is pending, not success, because gate / conformance (pull_request) sits at "Blocked by required conditions" — that context is not in the required list, so it does not block, but the PR is not "all green" either.

The gate's lint stage passed with one line too long warning on roles/node_health/tasks/assess.yml:189 — that is pre-existing, in the SMART block, well above my insertion point, and warnings do not fail the stage.

Offline work (before the lock was released)

• python3 -m py_compile on the collector: clean.
• collect_graphics() executed for real against this workstation's hardware (an unrelated machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole structure with no exception, correctly reporting two cards, two render nodes, the bound driver per card, the vcs0/vcs1/vecs0 media engines, and enable_guc as not readable without root rather than as a fabricated 0.
• That last point is a real finding and changed the design: /sys/module/i915/parameters/* is mode 0400, so an unprivileged pass cannot read enable_guc. It now reports NOT READ — <reason> and /proc/cmdline is read as the world-readable second source. An earlier draft would have silently reported a null.
• Two other bugs found and fixed by running it rather than reasoning about it: /dev/dri/by-path is a directory and was being lstat()ed as a device; and /sys/class/drm carries one entry per connector (card0-HDMI-A-1), which had to be excluded from the card scan.
• All three checks and the new template section were rendered through a standalone Jinja harness (Ansible is not on the host — CLAUDE.md §5) against seven fact sets: the real workstation; GPU present with the render node removed; no display device at all; xe bound in place of i915; a display device with no driver bound; renderD128 present but not a character device; and a shape with neither lspci nor vainfo. Zero template errors, and the statuses came out as designed — FAIL/PASS/INFO, SKIP/SKIP/SKIP, PASS/WARN/INFO, FAIL/FAIL/INFO, FAIL/PASS/INFO respectively. The harness supplies stand-ins for the Ansible-only filters (bool, dict2items, flatten, zip, regex_replace) and matches Ansible's trim_blocks=True. Read that for exactly what it is: evidence that the LOGIC is right, and no evidence whatsoever that the HARDWARE DETECTION is. Six of those seven fact sets were hand-mutated dictionaries, not machines. They are not hardware tests and must not be written up as if they were.

What I did NOT verify

• The FAIL path has never fired on a real node. Every prod node has a working render device, and this role is read-only, so a device cannot be taken away to prove the check catches its absence. The failure branches were exercised against synthetic fact sets only (see above). I am not claiming the failure path is tested against hardware.
• xe detection is likewise untested against reality. No node here has xe bound — the module exists on disk as xe.ko.xz and is bound to nothing — so the WARN branch was only ever driven by a mutated fact set. If xe ever does bind, the shape of /sys/class/drm/card* under it is something I have assumed matches i915's (same device/driver symlink, same engine/* directories). That assumption is unverified.
• The media-engine sysfs path is confirmed on these nodes but not proven portable. vcs0/vecs0 were read from /sys/class/drm/card0/engine/* on all three prod nodes, so it is measured here. I did not confirm against kernel source that no other kernel version or driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO line, not a false verdict, which is why this failure mode is tolerable.
• enable_guc is still unread on prod. The report says NOT READ — mode 0400, which is honest but means I have not confirmed by measurement that GuC/HuC is off on these nodes; I only know the kernel cmdline carries no i915.enable_guc= override. The brief's ground truth says i915.enable_guc=0; my check does not corroborate that value, it declines to guess it. Reading it would need root, and it is INFO either way, so I left it alone.
• Not a transcode test. These checks prove the render device exists, is a character device, and which driver owns it. They do not prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a node-level check that installs nothing.
• Greenness was never established, locally or in CI. Both runs returned rc 3. My local run skipped three stages (spec-immutability, gitea-settings, conformance); CI skipped one (spec-immutability) and its own headline says MERGE GATE: UNDETERMINED. I am not describing this PR as green, and the green Gitea tick must not be read as one: the workflow line is ./ci/gate.sh || [ "$?" -eq 3 ], so it is a tolerated 3.
• L2/L3 conformance did not run anywhere — CLUSTER_AVAILABLE != true in both. Whatever runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
• spec-immutability ran in neither. The sibling spec repo is checked out in neither my worktree nor the CI workspace, so nothing confirms this change does not drift from the spec. It is the expected, tolerated skip — but it is still a skip, not a pass.
• No claim about the live platform beyond what was printed. Per CLAUDE.md §2b this repo is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. Branch protection and runner state I did query directly, and the CI gitea-settings stage passed — so those two I assert on measurement, not on this file's prose.
• The stale dev inventory was left alone deliberately. hosts.ini still defines the k3s-dev-* groups as the default target of most playbooks and the dev cluster no longer exists. That is real, it is out of this change's scope, and it is unclaimed work.