Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(node-health): probe the Intel render device, read-only #31

Merged
jakman85 merged 2 commits from feat/node-health-render-device into main 2026-10-02 01:42:00 +00:00
Conversation 1 Commits 2 Files Changed 6
+834 -4
gitea-admin commented 2026-09-30 18:07:20 +00:00
Owner
Copy link
Copy source

The prod nodes already have a working Intel render device — measured by hand on all three on 2026-09-30. This PR installs nothing, loads no module, changes no group and reboots nothing. It makes that fact self-re-measuring, so a node rebuild, BIOS change or driver regression surfaces in the health report instead of silently degrading Plex to software transcoding.

Why this is worth a check rather than a runbook note

Nothing else in the platform would notice the device going missing. Plex exposes no ServiceMonitor, so nothing alerts on a silent fall back to software transcoding — and the manifest side mounts the device with hostPath type: DirectoryOrCreate, so a node without the device mounts an empty directory and Plex starts, runs and degrades quietly rather than failing loudly. No crash loop, no failed mount, no alert: just a hot CPU and stuttering clients. This check is the only thing in the system that would surface it.

What it adds

roles/node_health gains a graphics / render-device probe, entirely in the unprivileged collector pass: lstat() on /dev/dri/*, grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the vcs*/vecs* media engines, plus /proc/modules and /proc/cmdline.

Three checks in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:

id grading
graphics.render_device the outcome — a usable character-special renderD* node. FAIL only when a display device is on the PCI bus and there is still no render node; SKIP when the machine has no display device at all (a VM guest)
graphics.drm_driver what is bound, against the new node_health_graphics_expected_driver default (i915). xe bound instead is a WARN, not a FAIL — the device works, but the userspace media stack differs
graphics.media_stack INFO only, deliberately

Three design rulings worth reviewing:

  1. The render node's GID is reported, never asserted. It read root:render gid 992 on all three nodes; the check does not compare against 992. render is a systemd dynamic allocation, not a Debian static gid, so a rebuild can legitimately move it — and Plex's container entrypoint stats the device and joins whichever group owns it. Drift stays visible without being a false alarm.
  2. The PCI scan is separate from the DRM scan. /sys/class/drm/card* exists only once a driver has bound, so a GPU with nothing attached has no card0 and would otherwise look identical to a machine with no GPU. Separating them is what makes "no GPU here" an honest SKIP and "GPU present, no render node" a FAIL.
  3. GuC/HuC and host VAAPI userspace are INFO and must stay INFO. GuC/HuC off is an accepted state costing low-power H.264 encode quality only; no host vainfo is not a defect at all, since Plex ships its own libva inside its container. Grading either would invite someone to apt install vainfo on a node to clear a health check.

Verification

Read-only prod run, all three nodes. --list-hosts first (exactly 3 hosts, nothing else), then --syntax-check (rc 0), then:

ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0
k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0
k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0

(changed= is the controller-side artifact write, delegated to localhost. Nothing on any node changed.)

All three graded PASS / PASS / INFO, 0 FAIL, with a byte-identical report section:

 pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
              driver i915
 drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
 dev  card0        character special  mode 0660  root:video gid 44
 dev  renderD128   character special  mode 0660  root:render gid 992
 GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
              readable without root (i915 module parameters are mode 0400)
             no i915/xe parameter on the kernel cmdline
 modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false
 vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container

That reproduces every item of the hand measurement independently.

Gate: rc 3 — UNDETERMINED, NOT green. Nothing failed; three stages did not run, all three the known local ceiling from a worktree, and no fourth skip:

[skip] spec-immutability  spec repo not present (set SPEC_REPO)
[skip] gitea-settings     SETTINGS_API_TOKEN is not set
[skip] conformance        CLUSTER_AVAILABLE != true — L2/L3 NOT RUN
[ ok ] decisions-log · lint · l0 · render-golden · l1 · approval-policy

Re-run on the final amended tree, same shape. Both CLAUDE.md §3 worktree workarounds were needed and both worked (DEV_KUBECONFIG=/dev/null, and an ENGINE shim bind-mounting the main checkout's .git — no tracked_files() traceback, so the shim took effect). ansible-lint was not run because ci/ does not invoke it anywhere; the lint stage is yamllint only.

What is NOT verified

  • The FAIL path has never fired on real hardware. Every prod node has a working render device and this role is read-only, so one cannot be taken away to prove the check catches its absence. The failure branches were exercised against synthetic fact sets only — good evidence the logic is right, and no evidence the hardware detection is.
  • xe binding is untested for the same reason. xe.ko.xz exists on disk here and is bound to nothing, so the WARN branch was only ever driven by a mutated fact set.
  • enable_guc is still unread on prod (mode 0400). The check reports NOT READ with the reason rather than guessing a value, so it does not corroborate the enable_guc=0 ground truth — it declines to.
  • Not a transcode test. This proves the render device exists and which driver owns it, not that a transcode succeeds.

One measured correction to my own earlier assumption: lspci is present on all three prod nodes (Debian ships pciutils). It is still treated as optional, because pciutils is genuinely not declared in node_baseline_packages; the comments that asserted it was absent are corrected.

infra/ansible/** is not a sensitive path in infra/gitea/approval-policy.json, and the approval-policy stage passed. Nothing under conformance/** or clusters/** is touched.

🤖 Generated with Claude Code

The prod nodes already have a working Intel render device — measured by hand on all three on 2026-09-30. **This PR installs nothing, loads no module, changes no group and reboots nothing.** It makes that fact *self-re-measuring*, so a node rebuild, BIOS change or driver regression surfaces in the health report instead of silently degrading Plex to software transcoding. ## Why this is worth a check rather than a runbook note **Nothing else in the platform would notice the device going missing.** Plex exposes no ServiceMonitor, so nothing alerts on a silent fall back to software transcoding — and the manifest side mounts the device with `hostPath type: DirectoryOrCreate`, so a node *without* the device mounts an **empty directory** and Plex starts, runs and degrades quietly rather than failing loudly. No crash loop, no failed mount, no alert: just a hot CPU and stuttering clients. This check is the only thing in the system that would surface it. ## What it adds `roles/node_health` gains a graphics / render-device probe, entirely in the **unprivileged** collector pass: `lstat()` on `/dev/dri/*`, `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink for the bound driver, `/sys/bus/pci/devices/*/class` for display-class devices *whether or not a driver is bound*, `/sys/class/drm/card*/engine/*` for the `vcs*`/`vecs*` media engines, plus `/proc/modules` and `/proc/cmdline`. Three checks in the role's existing `PASS/WARN/FAIL/SKIP/INFO` shape: | id | grading | |---|---| | `graphics.render_device` | the outcome — a usable character-special `renderD*` node. **FAIL** only when a display device *is* on the PCI bus and there is still no render node; **SKIP** when the machine has no display device at all (a VM guest) | | `graphics.drm_driver` | what is bound, against the new `node_health_graphics_expected_driver` default (`i915`). `xe` bound instead is a **WARN**, not a FAIL — the device works, but the userspace media stack differs | | `graphics.media_stack` | **INFO only, deliberately** | Three design rulings worth reviewing: 1. **The render node's GID is reported, never asserted.** It read `root:render` gid **992** on all three nodes; the check does *not* compare against `992`. `render` is a **systemd dynamic** allocation, not a Debian static gid, so a rebuild can legitimately move it — and Plex's container entrypoint stats the device and joins whichever group owns it. Drift stays visible without being a false alarm. 2. **The PCI scan is separate from the DRM scan.** `/sys/class/drm/card*` exists only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would otherwise look identical to a machine with no GPU. Separating them is what makes "no GPU here" an honest SKIP and "GPU present, no render node" a FAIL. 3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off is an accepted state costing low-power H.264 encode *quality* only; no host `vainfo` is not a defect at all, since Plex ships its own libva inside its container. Grading either would invite someone to `apt install vainfo` on a node to clear a health check. ## Verification **Read-only prod run, all three nodes.** `--list-hosts` first (exactly 3 hosts, nothing else), then `--syntax-check` (rc 0), then: ``` ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 ``` (`changed=` is the controller-side artifact write, delegated to localhost. Nothing on any node changed.) All three graded **PASS / PASS / INFO**, 0 FAIL, with a byte-identical report section: ``` pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02) driver i915 drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0 dev card0 character special mode 0660 root:video gid 44 dev renderD128 character special mode 0660 root:render gid 992 GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not readable without root (i915 module parameters are mode 0400) no i915/xe parameter on the kernel cmdline modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container ``` That reproduces every item of the hand measurement independently. **Gate: `rc 3` — UNDETERMINED, NOT green.** Nothing failed; three stages did not run, all three the known local ceiling from a worktree, and **no fourth skip**: ``` [skip] spec-immutability spec repo not present (set SPEC_REPO) [skip] gitea-settings SETTINGS_API_TOKEN is not set [skip] conformance CLUSTER_AVAILABLE != true — L2/L3 NOT RUN [ ok ] decisions-log · lint · l0 · render-golden · l1 · approval-policy ``` Re-run on the final amended tree, same shape. Both CLAUDE.md §3 worktree workarounds were needed and both worked (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the main checkout's `.git` — no `tracked_files()` traceback, so the shim took effect). `ansible-lint` was not run because **`ci/` does not invoke it anywhere**; the lint stage is `yamllint` only. ## What is NOT verified - **The FAIL path has never fired on real hardware.** Every prod node has a working render device and this role is read-only, so one cannot be taken away to prove the check catches its absence. The failure branches were exercised against **synthetic fact sets only** — good evidence the *logic* is right, and no evidence the *hardware detection* is. - **`xe` binding is untested for the same reason.** `xe.ko.xz` exists on disk here and is bound to nothing, so the WARN branch was only ever driven by a mutated fact set. - **`enable_guc` is still unread on prod** (mode 0400). The check reports `NOT READ` with the reason rather than guessing a value, so it does not corroborate the `enable_guc=0` ground truth — it declines to. - **Not a transcode test.** This proves the render *device* exists and which driver owns it, not that a transcode succeeds. One measured correction to my own earlier assumption: `lspci` **is** present on all three prod nodes (Debian ships `pciutils`). It is still treated as optional, because `pciutils` is genuinely not declared in `node_baseline_packages`; the comments that asserted it was absent are corrected. `infra/ansible/**` is not a sensitive path in `infra/gitea/approval-policy.json`, and the `approval-policy` stage passed. Nothing under `conformance/**` or `clusters/**` is touched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-09-30 18:07:21 +00:00
feat(node-health): probe the Intel render device, read-only
gate / gate (pull_request) Successful in 1m40s
Details
gate / conformance (pull_request) Has been skipped
Details
a78f010368 
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.

The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.

All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.

Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:

  graphics.render_device  the outcome — a usable character-special renderD* node.
                          FAIL only when a display device IS on the PCI bus and
                          there is still no render node. SKIP when the machine has
                          no display device at all (a VM guest), which is why the
                          PCI scan is separate from the DRM scan: /sys/class/drm/
                          card* exists only once a driver has BOUND.
  graphics.drm_driver     what is bound, against the new
                          node_health_graphics_expected_driver default (i915), so
                          the comparison is data and not task logic. `xe` bound in
                          place of i915 is a WARN, not a FAIL: the device works but
                          the userspace media stack differs.
  graphics.media_stack    INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
                          is an accepted state that costs low-power H.264 encode
                          quality only, and no host vainfo is not a defect at all
                          because Plex ships its own libva inside its container.

The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
gitea-admin added 1 commit 2026-09-30 21:14:35 +00:00
docs(decisions): record what the CI gate actually reported, not the tick
gate / gate (pull_request) Successful in 1m39s
Details
gate / conformance (pull_request) Has been skipped
Details
7df72db487 
The PR's required check went green as "Successful in 1m40s" — a real run, not
"Has been skipped". But .gitea/workflows/gate.yml:138 is
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot
distinguish them. Fetched the job log and read the gate's own headline:
MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability).

So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that
plainly rather than leaving "CI green" to be read as rc 0.

Also corrects a claim that had gone stale within the same entry: CI ran two
stages my worktree run could not, and both passed — gitea-settings (the live
Gitea matches infra/gitea/expected-branch-protection.json; runners online 1)
and approval-policy with real PR context (no sensitive paths touched, so
infra/ansible/** needs no human reviewer). My local run skipped both, and the
handoff said branch protection was UNVERIFIED. It is now measured, so saying
otherwise would have left a known-false line in the log.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85 approved these changes 2026-10-02 01:41:54 +00:00
jakman85 merged commit b763f11e19 into main 2026-10-02 01:42:00 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
jakman85
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#31