feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
0fbf544d5e
commit
a78f010368
6 files changed
+785
-4
No files matched your search
@@ -0,0 +1,234 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: feat/node-health-render-device
|
||||
title: The Intel render device is now self-re-measuring, not a fact someone checked once
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
### Why this probe has value at all
|
||||
|
||||
**Nothing else in the platform would notice the device going missing.** Plex exposes no
|
||||
ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
|
||||
manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
|
||||
*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
|
||||
rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
|
||||
This check is the only thing in the system that would surface it, which is the actual
|
||||
justification for the role carrying it.
|
||||
|
||||
### The change
|
||||
|
||||
`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
|
||||
was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
|
||||
Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
|
||||
is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
|
||||
|
||||
The failure this exists to catch is silent by construction. Plex hardware-transcodes through
|
||||
`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
|
||||
the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
|
||||
back to software transcoding on its own and the only symptom is a hot CPU.
|
||||
|
||||
Four files:
|
||||
|
||||
- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
|
||||
pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
|
||||
`grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
|
||||
target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
|
||||
`0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
|
||||
`vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
|
||||
`/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
|
||||
existing tool probe; both are optional and neither is depended on.
|
||||
- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
|
||||
(PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
|
||||
(INFO/SKIP).
|
||||
- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
|
||||
check makes is data rather than task logic.
|
||||
- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
|
||||
`roles/node_health/README.md` gained a section explaining the design and two new "known
|
||||
limits".
|
||||
|
||||
Three design rulings worth stating, because each is the kind of thing a later edit would get
|
||||
wrong:
|
||||
|
||||
1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
|
||||
mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
|
||||
a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
|
||||
`base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
|
||||
the device and joins whichever group owns it, so drift genuinely is not a fault. The
|
||||
observed group name and number appear in the detail line, which makes drift *visible*
|
||||
without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
|
||||
2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
|
||||
only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
|
||||
otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
|
||||
is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
|
||||
convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
|
||||
the bus and there is still no render node" is a **FAIL**.
|
||||
3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
|
||||
(`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
|
||||
only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
|
||||
inside its container. Grading either would put a permanent WARN on a chosen configuration,
|
||||
or invite someone to `apt install vainfo` on a node to clear a health check, and this role
|
||||
installs nothing.
|
||||
|
||||
`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
|
||||
drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
|
||||
bound to nothing.
|
||||
|
||||
## What I verified
|
||||
|
||||
### The real prod run (read-only, all three nodes)
|
||||
|
||||
`--list-hosts` first, every time. It named exactly three hosts and nothing else:
|
||||
|
||||
play #1 (k3s_prod): Inspect node hardware health (read-only)
|
||||
pattern: ['k3s_prod']
|
||||
hosts (3): k3s-prod-3 k3s-prod-2 k3s-prod-1
|
||||
|
||||
Then `--syntax-check` (rc 0), then the run:
|
||||
|
||||
ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
|
||||
|
||||
k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 skipped=1
|
||||
k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 skipped=1
|
||||
k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 skipped=1
|
||||
PLAY rc=0
|
||||
|
||||
(The `changed=` counts are the *controller-side* artifact writes under
|
||||
`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
|
||||
the collector task is `changed_when: false` and the role touches nothing.)
|
||||
|
||||
All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
|
||||
|
||||
k3s-prod-1 VERDICT 0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
|
||||
k3s-prod-2 VERDICT 0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
|
||||
k3s-prod-3 VERDICT 0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
|
||||
|
||||
[PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
|
||||
[PASS] graphics.drm_driver — DRM driver bound to the display device (expected i915)
|
||||
[INFO] graphics.media_stack — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
|
||||
|
||||
(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
|
||||
mine; none of them moved.)
|
||||
|
||||
The new report section, **byte-identical on all three nodes**:
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
|
||||
--------------------------------------------------------------------------------
|
||||
pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
|
||||
driver i915
|
||||
drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0
|
||||
dev card0 character special mode 0660 root:video gid 44
|
||||
dev renderD128 character special mode 0660 root:render gid 992
|
||||
GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not
|
||||
readable without root (i915 module parameters are mode 0400)
|
||||
no i915/xe parameter on the kernel cmdline
|
||||
modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
|
||||
vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
|
||||
|
||||
That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
|
||||
character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
|
||||
`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
|
||||
CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
|
||||
|
||||
**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
|
||||
prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
|
||||
absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
|
||||
`node_baseline_packages` and depending on an undeclared package would be wrong — but the
|
||||
comments and README that asserted it was missing were factually wrong and are now corrected.
|
||||
Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
|
||||
|
||||
### The gate
|
||||
|
||||
DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
GATE rc=3
|
||||
|
||||
**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
|
||||
three stages did not run, all three the known local ceiling from a worktree, and **no fourth
|
||||
skip**:
|
||||
|
||||
[skip] spec-immutability spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
|
||||
[skip] gitea-settings SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
|
||||
[skip] conformance no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
|
||||
|
||||
[ ok ] decisions-log: pass [ ok ] lint: pass [ ok ] l0: pass
|
||||
[ ok ] render-golden: pass [ ok ] l1: pass [ ok ] approval-policy: pass
|
||||
|
||||
Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
|
||||
(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
|
||||
the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
|
||||
know the shim took effect rather than the git stages silently degrading.
|
||||
|
||||
`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
|
||||
`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
|
||||
claiming to have matched a lint convention that does not exist.
|
||||
|
||||
The gate's `lint` stage passed with one `line too long` **warning** on
|
||||
`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
|
||||
above my insertion point, and warnings do not fail the stage.
|
||||
|
||||
### Offline work (before the lock was released)
|
||||
|
||||
- `python3 -m py_compile` on the collector: clean.
|
||||
- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
|
||||
machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
|
||||
structure with no exception, correctly reporting two cards, two render nodes, the bound
|
||||
driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
|
||||
without root* rather than as a fabricated `0`.
|
||||
- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
|
||||
**mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
|
||||
`NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
|
||||
earlier draft would have silently reported a null.
|
||||
- Two other bugs found and fixed by running it rather than reasoning about it:
|
||||
`/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
|
||||
`/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
|
||||
excluded from the card scan.
|
||||
- All three checks and the new template section were rendered through a standalone Jinja
|
||||
harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
|
||||
workstation; GPU present with the render node removed; no display device at all; `xe` bound
|
||||
in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
|
||||
character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
|
||||
the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
|
||||
`FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
|
||||
Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
|
||||
Ansible's `trim_blocks=True`.
|
||||
**Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
|
||||
whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
|
||||
dictionaries, not machines. They are not hardware tests and must not be written up as if
|
||||
they were.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The FAIL path has never fired on a real node.** Every prod node has a working render
|
||||
device, and this role is read-only, so a device cannot be taken away to prove the check
|
||||
catches its absence. The failure branches were exercised against **synthetic fact sets
|
||||
only** (see above). I am not claiming the failure path is tested against hardware.
|
||||
- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
|
||||
module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
|
||||
ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
|
||||
`/sys/class/drm/card*` under it is something I have assumed matches i915's (same
|
||||
`device/driver` symlink, same `engine/*` directories). That assumption is unverified.
|
||||
- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
|
||||
`vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
|
||||
is measured here. I did **not** confirm against kernel source that no other kernel version or
|
||||
driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
|
||||
line, not a false verdict, which is why this failure mode is tolerable.
|
||||
- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
|
||||
honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
|
||||
I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
|
||||
truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
|
||||
guess it. Reading it would need root, and it is INFO either way, so I left it alone.
|
||||
- **Not a transcode test.** These checks prove the render *device* exists, is a character
|
||||
device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
|
||||
VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
|
||||
node-level check that installs nothing.
|
||||
- **The gate did not establish greenness.** It returned **rc 3**, and three stages did not run
|
||||
(`spec-immutability`, `gitea-settings`, `conformance`). So: the offline L0/L1 subset passed,
|
||||
the live Gitea branch-protection rule is **UNVERIFIED** from this run, and L2/L3 conformance
|
||||
did not execute. I am not describing this PR as green.
|
||||
- **No claim about the live platform beyond what the run printed.** Per CLAUDE.md §2b this repo
|
||||
is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
|
||||
I did not query Gitea's branch protection or runner state, so I assert nothing about either.
|
||||
- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
|
||||
`k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
|
||||
exists. That is real, it is out of this change's scope, and it is unclaimed work.
|
||||
Reference in new issue
Block a user