home-network-k3s-impl/decisions/2026-09-30-node-health-render-device.md

---
date: 2026-09-30
branch: feat/node-health-render-device
title: The Intel render device is now self-re-measuring, not a fact someone checked once
---

## What changed

### Why this probe has value at all

**Nothing else in the platform would notice the device going missing.** Plex exposes no
ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
This check is the only thing in the system that would surface it, which is the actual
justification for the role carrying it.

### The change

`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.

The failure this exists to catch is silent by construction. Plex hardware-transcodes through
`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
back to software transcoding on its own and the only symptom is a hot CPU.

Four files:

- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
  existing tool probe; both are optional and neither is depended on.
- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
  (INFO/SKIP).
- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
  check makes is data rather than task logic.
- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
  `roles/node_health/README.md` gained a section explaining the design and two new "known
  limits".

Three design rulings worth stating, because each is the kind of thing a later edit would get
wrong:

1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
   the device and joins whichever group owns it, so drift genuinely is not a fault. The
   observed group name and number appear in the detail line, which makes drift *visible*
   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
   the bus and there is still no render node" is a **FAIL**.
3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
   inside its container. Grading either would put a permanent WARN on a chosen configuration,
   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
   installs nothing.

`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
bound to nothing.

## What I verified

### The real prod run (read-only, all three nodes)

`--list-hosts` first, every time. It named exactly three hosts and nothing else:

    play #1 (k3s_prod): Inspect node hardware health (read-only)
        pattern: ['k3s_prod']
        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1

Then `--syntax-check` (rc 0), then the run:

    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod

    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
    PLAY rc=0

(The `changed=` counts are the *controller-side* artifact writes under
`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
the collector task is `changed_when: false` and the role touches nothing.)

All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:

    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO

    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)

(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
mine; none of them moved.)

The new report section, **byte-identical on all three nodes**:

    --------------------------------------------------------------------------------
     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
    --------------------------------------------------------------------------------
     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
                  driver i915
     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
     dev  card0        character special  mode 0660  root:video gid 44
     dev  renderD128   character special  mode 0660  root:render gid 992
     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
                  readable without root (i915 module parameters are mode 0400)
                 no i915/xe parameter on the kernel cmdline
     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container

That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.

**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
`node_baseline_packages` and depending on an undeclared package would be wrong — but the
comments and README that asserted it was missing were factually wrong and are now corrected.
Its presence is why the `pci` line above carries a readable device name rather than bare IDs.

### The gate

    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
    GATE rc=3

**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
three stages did not run, all three the known local ceiling from a worktree, and **no fourth
skip**:

    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN

    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass

Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
know the shim took effect rather than the git stages silently degrading.

`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
claiming to have matched a lint convention that does not exist.

### The CI gate on the PR — a green tick that is exit 3, read from the log

The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.

**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
Gitea, the web route serves it) and read the gate's own headline:

    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
    [fail]   Stages that did not run: spec-immutability

**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
check is the tolerance swallowing that 3, not a rc 0.

**CI verified two things my local run could not, and both came back green:**

    [ ok ] stage gitea-settings: pass
    [ ok ]   enable_push = false          enable_status_check = true
    [ ok ]   required_approvals = 1       block_admin_merge_override = true
    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
    [ ok ]   runners online: 1 (>= 1)
    [ ok ] Gitea settings match the declaration.
    [ ok ] stage approval-policy: pass
    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.

My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
there would have failed the job. That is measured confirmation that the live Gitea matches
`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.

Corroborated independently: `GET /repos/.../branches/main` →
`protected=true user_can_push=false enable_status_check=true required_approvals=1`,
contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
status=online`. And note the commit's **combined** state is `pending`, not success, because
`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
not in the required list, so it does not block, but the PR is not "all green" either.

The gate's `lint` stage passed with one `line too long` **warning** on
`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
above my insertion point, and warnings do not fail the stage.

### Offline work (before the lock was released)

- `python3 -m py_compile` on the collector: clean.
- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
  structure with no exception, correctly reporting two cards, two render nodes, the bound
  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
  without root* rather than as a fabricated `0`.
- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
  earlier draft would have silently reported a null.
- Two other bugs found and fixed by running it rather than reasoning about it:
  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
  excluded from the card scan.
- All three checks and the new template section were rendered through a standalone Jinja
  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
  workstation; GPU present with the render node removed; no display device at all; `xe` bound
  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
  Ansible's `trim_blocks=True`.
  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
  dictionaries, not machines. They are not hardware tests and must not be written up as if
  they were.

## What I did NOT verify

- **The FAIL path has never fired on a real node.** Every prod node has a working render
  device, and this role is read-only, so a device cannot be taken away to prove the check
  catches its absence. The failure branches were exercised against **synthetic fact sets
  only** (see above). I am not claiming the failure path is tested against hardware.
- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
  is measured here. I did **not** confirm against kernel source that no other kernel version or
  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
  line, not a false verdict, which is why this failure mode is tolerable.
- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
- **Not a transcode test.** These checks prove the render *device* exists, is a character
  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
  node-level check that installs nothing.
- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
  It is the expected, tolerated skip — but it is still a skip, not a pass.
- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
  passed — so those two I assert on measurement, not on this file's prose.
- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
  exists. That is real, it is out of this change's scope, and it is unclaimed work.