From a78f0103684c37f367887614da6cd415ea3d83f6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 12:56:55 -0500 Subject: [PATCH 1/2] feat(node-health): probe the Intel render device, read-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prod nodes already have a working Intel render device — measured by hand on all three on 2026-09-30. Nothing here installs, loads or changes anything; the point is that the measurement now re-runs on every health run instead of decaying silently in a handoff note. The failure being hunted is silent by construction: Plex hardware-transcodes through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver) Plex falls back to software transcoding on its own and the only symptom is a hot CPU. All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*, grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are probed but never depended on — neither is in node_baseline's package floor. Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape: graphics.render_device the outcome — a usable character-special renderD* node. FAIL only when a display device IS on the PCI bus and there is still no render node. SKIP when the machine has no display device at all (a VM guest), which is why the PCI scan is separate from the DRM scan: /sys/class/drm/ card* exists only once a driver has BOUND. graphics.drm_driver what is bound, against the new node_health_graphics_expected_driver default (i915), so the comparison is data and not task logic. `xe` bound in place of i915 is a WARN, not a FAIL: the device works but the userspace media stack differs. graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0) is an accepted state that costs low-power H.264 encode quality only, and no host vainfo is not a defect at all because Plex ships its own libva inside its container. The render node's owning group and GID are REPORTED, never asserted: `render` is a systemd dynamic allocation rather than a Debian static gid, so a rebuild can legitimately move it off the 992 measured today, and Plex's container entrypoint stats the device and joins whichever group owns it. Drift stays visible without being an error. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-09-30-node-health-render-device.md | 234 ++++++++++++++++++ infra/ansible/roles/node_health/README.md | 88 ++++++- .../roles/node_health/defaults/main.yml | 18 ++ .../node_health/files/node_health_collect.py | 220 +++++++++++++++- .../roles/node_health/tasks/assess.yml | 198 +++++++++++++++ .../node_health/templates/summary.txt.j2 | 31 +++ 6 files changed, 785 insertions(+), 4 deletions(-) create mode 100644 decisions/2026-09-30-node-health-render-device.md diff --git a/decisions/2026-09-30-node-health-render-device.md b/decisions/2026-09-30-node-health-render-device.md new file mode 100644 index 0000000..c6d3cd9 --- /dev/null +++ b/decisions/2026-09-30-node-health-render-device.md @@ -0,0 +1,234 @@ +--- +date: 2026-09-30 +branch: feat/node-health-render-device +title: The Intel render device is now self-re-measuring, not a fact someone checked once +--- + +## What changed + +### Why this probe has value at all + +**Nothing else in the platform would notice the device going missing.** Plex exposes no +ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the +manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node +*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly +rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU. +This check is the only thing in the system that would surface it, which is the actual +justification for the role carrying it. + +### The change + +`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module +was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working +Intel render device (measured by hand on all three on 2026-09-30) and the point of this change +is that the measurement now **re-runs forever** instead of decaying silently in a handoff note. + +The failure this exists to catch is silent by construction. Plex hardware-transcodes through +`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables +the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls +back to software transcoding on its own and the only symptom is a hot CPU. + +Four files: + +- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged** + pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`, + `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink + target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class + `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the + `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and + `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the + existing tool probe; both are optional and neither is depended on. +- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device` + (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack` + (INFO/SKIP). +- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the + check makes is data rather than task logic. +- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and + `roles/node_health/README.md` gained a section explaining the design and two new "known + limits". + +Three design rulings worth stating, because each is the kind of thing a later edit would get +wrong: + +1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**, + mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is + a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from + `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats + the device and joins whichever group owns it, so drift genuinely is not a fault. The + observed group name and number appear in the detail line, which makes drift *visible* + without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall. +2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists + only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would + otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately + is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same + convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on + the bus and there is still no render node" is a **FAIL**. +3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off + (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality* + only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver + inside its container. Grading either would put a permanent WARN on a chosen configuration, + or invite someone to `apt install vainfo` on a node to clear a health check, and this role + installs nothing. + +`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two +drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is +bound to nothing. + +## What I verified + +### The real prod run (read-only, all three nodes) + +`--list-hosts` first, every time. It named exactly three hosts and nothing else: + + play #1 (k3s_prod): Inspect node hardware health (read-only) + pattern: ['k3s_prod'] + hosts (3): k3s-prod-3 k3s-prod-2 k3s-prod-1 + +Then `--syntax-check` (rc 0), then the run: + + ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod + + k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 skipped=1 + k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 + k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 + PLAY rc=0 + +(The `changed=` counts are the *controller-side* artifact writes under +`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed — +the collector task is `changed_when: false` and the role touches nothing.) + +All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall: + + k3s-prod-1 VERDICT 0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO + k3s-prod-2 VERDICT 0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO + k3s-prod-3 VERDICT 0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO + + [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*) + [PASS] graphics.drm_driver — DRM driver bound to the display device (expected i915) + [INFO] graphics.media_stack — Media engines, GuC/HuC firmware and host VAAPI userspace (informational) + +(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not +mine; none of them moved.) + +The new report section, **byte-identical on all three nodes**: + + -------------------------------------------------------------------------------- + GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) + -------------------------------------------------------------------------------- + pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02) + driver i915 + drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0 + dev card0 character special mode 0660 root:video gid 44 + dev renderD128 character special mode 0660 root:render gid 992 + GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not + readable without root (i915 module parameters are mode 0400) + no i915/xe parameter on the kernel cmdline + modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides) + vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container + +That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128` +character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode +`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)` +CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`. + +**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three +prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was +absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in +`node_baseline_packages` and depending on an undeclared package would be wrong — but the +comments and README that asserted it was missing were factually wrong and are now corrected. +Its presence is why the `pci` line above carries a readable device name rather than bare IDs. + +### The gate + + DEV_KUBECONFIG=/dev/null ENGINE= ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' + GATE rc=3 + +**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly +three stages did not run, all three the known local ceiling from a worktree, and **no fourth +skip**: + + [skip] spec-immutability spec repo not present at /work/../home-network-k3s (set SPEC_REPO) + [skip] gitea-settings SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule + [skip] conformance no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN + + [ ok ] decisions-log: pass [ ok ] lint: pass [ ok ] l0: pass + [ ok ] render-golden: pass [ ok ] l1: pass [ ok ] approval-policy: pass + +Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null` +(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts +the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I +know the shim took effect rather than the git stages silently degrading. + +`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped +`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not +claiming to have matched a lint convention that does not exist. + +The gate's `lint` stage passed with one `line too long` **warning** on +`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well +above my insertion point, and warnings do not fail the stage. + +### Offline work (before the lock was released) + +- `python3 -m py_compile` on the collector: clean. +- `collect_graphics()` executed for real against this workstation's hardware (an unrelated + machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole + structure with no exception, correctly reporting two cards, two render nodes, the bound + driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable + without root* rather than as a fabricated `0`. +- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is + **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports + `NOT READ — ` and `/proc/cmdline` is read as the world-readable second source. An + earlier draft would have silently reported a null. +- Two other bugs found and fixed by running it rather than reasoning about it: + `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and + `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be + excluded from the card scan. +- All three checks and the new template section were rendered through a standalone Jinja + harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real + workstation; GPU present with the render node removed; no display device at all; `xe` bound + in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a + character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and + the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`, + `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the + Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches + Ansible's `trim_blocks=True`. + **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence + whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated + dictionaries, not machines. They are not hardware tests and must not be written up as if + they were. + +## What I did NOT verify + +- **The FAIL path has never fired on a real node.** Every prod node has a working render + device, and this role is read-only, so a device cannot be taken away to prove the check + catches its absence. The failure branches were exercised against **synthetic fact sets + only** (see above). I am not claiming the failure path is tested against hardware. +- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the + module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only + ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of + `/sys/class/drm/card*` under it is something I have assumed matches i915's (same + `device/driver` symlink, same `engine/*` directories). That assumption is unverified. +- **The media-engine sysfs path is confirmed on these nodes but not proven portable.** + `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it + is measured here. I did **not** confirm against kernel source that no other kernel version or + driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO + line, not a false verdict, which is why this failure mode is tolerable. +- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is + honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes; + I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground + truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to + guess it. Reading it would need root, and it is INFO either way, so I left it alone. +- **Not a transcode test.** These checks prove the render *device* exists, is a character + device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a + VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a + node-level check that installs nothing. +- **The gate did not establish greenness.** It returned **rc 3**, and three stages did not run + (`spec-immutability`, `gitea-settings`, `conformance`). So: the offline L0/L1 subset passed, + the live Gitea branch-protection rule is **UNVERIFIED** from this run, and L2/L3 conformance + did not execute. I am not describing this PR as green. +- **No claim about the live platform beyond what the run printed.** Per CLAUDE.md §2b this repo + is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. + I did not query Gitea's branch protection or runner state, so I assert nothing about either. +- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the + `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer + exists. That is real, it is out of this change's scope, and it is unclaimed work. diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index 95f7dd6..88fcb97 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 ## Contract -- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`, - and — only when it can escalate — `smartctl`. It writes nothing to the node and - **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is +- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu` + (and `lspci` only if it happens to be installed), and — only when it can escalate — + `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is `node_baseline`'s job, never a health check's, never a manual `apt-get`). @@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters | | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | +| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) | +| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) | +| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | @@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t back to re-run the role that had already run. +## The render-device check + +Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing, +nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom +is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard +IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly +that silent degradation. + +The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A +fact measured once decays silently**, so it is re-measured on every health run instead of +living in a runbook. Nothing is installed, no module is loaded, no group is changed — the +role's read-only contract is unchanged. + +| fact | where it comes from | root? | +|---|---|---| +| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no | +| owning group **name** for the render node's gid | `grp.getgrgid()` | no | +| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no | +| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no | +| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no | +| module load state | `/proc/modules` | no | +| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** | + +### The PCI scan is separate from the DRM scan on purpose + +`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on +the bus with nothing attached has no `card0` at all — and that is precisely the failure being +hunted. So the display device is looked for on the PCI bus independently, from sysfs, which +is always there. That is what lets `graphics.render_device` distinguish + +- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention + `thermal.max` uses for "no sensor on this hardware"), from +- *a GPU is on the bus and there is still no render node* → **FAIL**. + +`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as +optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing +in this repo guarantees it. It did turn out to be present on all three prod nodes +(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending +on that would mean depending on an undeclared package. Absent, `description` is `null`, the +numeric IDs are printed instead, and no verdict changes. + +### The render node's GID is reported, never asserted + +The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on +2026-09-30. The check does **not** compare against `992`, and must not be changed to. +`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid +from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own +container entrypoint stats the device and joins whichever group owns it, so drift genuinely +is not a fault. Recording the observed group name and gid in the detail line makes drift +**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a +reinstall. + +What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver` +(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a +**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace +media stack and a libva/iHD build that transcodes correctly against one is not automatically +right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing. + +### Two things that are informational and must stay that way + +- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It + costs low-power H.264 encode *quality* only; decode and encode both still run on the media + engines. Grading it would put a permanent WARN on a chosen configuration. +- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and + media driver inside its container, so the host needs neither. Flagging it would invite + someone to `apt install vainfo` on a node to clear a health check — and this role installs + nothing. + +`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually +cannot read it. It reports as an honest `NOT READ — ` rather than a fabricated `0`, +with `/proc/cmdline` as the world-readable second source for an explicit boot override. + ## The EFI memattr check `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. @@ -273,3 +348,10 @@ stays `null` — no claim is made either way. but the wear/endurance context SKIPs rather than guessing. - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs `dmesg`, which needs root. +- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod + node has a working render device, and the role is read-only so one cannot be taken away to + prove it. The failure branches were exercised against synthetic fact sets only. The same + applies to the `xe`-bound WARN: no node here has `xe` bound. +- `graphics.*` proves the render **device** exists and which driver owns it. It does **not** + prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container, + with Plex's own libva, which is out of scope for a node-level check that installs nothing. diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index eae98c9..9b02dbc 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -35,6 +35,24 @@ node_health_try_privileged: true # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene. node_health_smartctl_args: "--json=c --all --nocheck=standby" +# --- Graphics / render device --------------------------------------------------------- +# The DRM driver this site EXPECTS to be bound to its display device. Measured on all three +# prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP +# EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to +# nothing, and that difference matters enough to surface: the two drivers do not share a +# userspace media stack (a libva/iHD build that works against i915 is not automatically the +# right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD +# or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic. +# +# WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning +# group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static +# one, so a rebuilt node may legitimately land on a different number — it was 992 on all +# three prod nodes on 2026-09-30, and an assertion against that literal would be a false +# alarm waiting to happen. Plex's own container entrypoint stats the device and joins +# whichever group owns it, so drift genuinely is not a fault. The observed group name and GID +# are therefore REPORTED in the check detail and never graded. +node_health_graphics_expected_driver: i915 + # --- Thresholds --------------------------------------------------------------------- # All verdicts come from here, so a site can retune without touching task logic. node_health_thresholds: diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index 8eb79e7..d6d6e1a 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -35,10 +35,13 @@ # ============================================================================= import glob +import grp import json import os +import pwd import re import shutil +import stat import subprocess import sys import time @@ -756,6 +759,212 @@ def collect_pressure(): return out +# --- Graphics / DRM render device ----------------------------------------------------- +# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that +# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that +# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software +# transcoding and the only symptom is a hot CPU. So the render device is measured on every +# health run rather than being a fact someone established once by hand. +# +# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are +# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so +# are the module parameters under /sys/module. Nothing here needs root, so nothing here +# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need +# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is +# what actually decides whether the firmware is loaded at all.) +# +# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists +# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no +# card0 at all, and that is precisely the failure this is meant to catch — so the display +# device is looked for on the PCI bus independently, from sysfs, which is always there. +# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as +# optional because it IS optional: pciutils is not declared in node_baseline_packages, so +# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes +# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on +# that would make the collector depend on an undeclared package. A missing lspci leaves +# `description` null, the numeric IDs are reported instead, and no verdict changes. +DISPLAY_PCI_CLASS_PREFIX = "0x03" # PCI base class 03 = display controller (VGA/3D/other) +# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative +# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is +# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact. +DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia") + + +def _uid_name(uid): + try: + return pwd.getpwuid(uid).pw_name + except (KeyError, OverflowError, TypeError, OSError): + return None + + +def _gid_name(gid): + """Group NAME for a numeric gid, or None when nothing resolves it. + + Reported as an OBSERVED value, never asserted against a literal. The `render` group is + a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's + static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own + container init stats the device and joins whatever group owns it, so a moved gid is + drift to notice, not a fault to fail on.""" + try: + return grp.getgrgid(gid).gr_name + except (KeyError, OverflowError, TypeError, OSError): + return None + + +def _module_param(mod, param): + """One /sys/module//parameters/, with WHY it is missing when it is. + + Three outcomes that must not be conflated: a real value; "the module exposes no such + parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no + enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915 + module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an + unprivileged pass legitimately cannot read them. Returning null with the reason keeps + that an honest UNKNOWN instead of a fabricated 0. read_text() also records the + permission denial in UNREADABLE, so the summary lists it with everything else root + would add.""" + path = "/sys/module/%s/parameters/%s" % (mod, param) + if not os.path.exists(path): + return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, " + "built in, or no such knob on this driver)" % (mod, param)} + value = read_text(path) + if value is None: + return {"value": None, "reason": "%s is not readable without root (i915 module " + "parameters are mode 0400)" % path} + return {"value": value, "reason": None} + + +def _dev_node(path): + """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is.""" + try: + st = os.lstat(path) + except OSError as exc: + UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc}) + return None + return { + "path": path, + "name": os.path.basename(path), + "char_device": stat.S_ISCHR(st.st_mode), + "mode": "%04o" % stat.S_IMODE(st.st_mode), + "uid": st.st_uid, + "gid": st.st_gid, + "owner": _uid_name(st.st_uid), + "group": _gid_name(st.st_gid), + } + + +def collect_graphics(): + dri_dir = "/dev/dri" + dri_present = os.path.isdir(dri_dir) + devices = [] + if dri_present: + for path in sorted(glob.glob(dri_dir + "/*")): + # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping + # directories rather than filtering to char devices is deliberate: anything + # in here that is NOT a character special file is an anomaly worth seeing, and + # `char_device: false` is how it becomes visible. + if os.path.isdir(path): + continue + entry = _dev_node(path) + if entry is not None: + devices.append(entry) + render_nodes = [d for d in devices if d["name"].startswith("renderD")] + + # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR + # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped. + cards = [] + for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")): + name = os.path.basename(cdir) + if "-" in name: + continue + driver = None + drvlink = cdir + "/device/driver" + if os.path.islink(drvlink): + driver = os.path.basename(os.path.realpath(drvlink)) + address = None + devlink = cdir + "/device" + if os.path.islink(devlink): + address = os.path.basename(os.path.realpath(devlink)) + # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC) + # engine and vecs* the video-enhancement engine: those two are what a transcode + # actually runs on, so their presence is the difference between "a GPU is bound" + # and "this GPU can do media work". + engines = sorted( + { + os.path.basename(p) + for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*")) + if os.path.isdir(p) + } + ) + cards.append( + { + "card": name, + "driver": driver, + "pci_address": address, + "vendor_id": read_text(cdir + "/device/vendor"), + "device_id": read_text(cdir + "/device/device"), + "engines": engines, + "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))], + } + ) + + # Display-class devices on the PCI bus, driver bound or not. + pci_display = [] + for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")): + class_id = read_text(pdir + "/class") or "" + if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX): + continue + address = os.path.basename(pdir) + driver = None + if os.path.islink(pdir + "/driver"): + driver = os.path.basename(os.path.realpath(pdir + "/driver")) + description = None + rc, out, _ = run(["lspci", "-nn", "-s", address]) + if rc == 0 and out.strip(): + description = out.strip().splitlines()[0] + pci_display.append( + { + "address": address, + "class_id": class_id or None, + "vendor_id": read_text(pdir + "/vendor"), + "device_id": read_text(pdir + "/device"), + "revision": read_text(pdir + "/revision"), + "driver": driver, + "description": description, + } + ) + + loaded = set() + for line in (read_text("/proc/modules") or "").splitlines(): + fields = line.split() + if fields: + loaded.add(fields[0]) + modules = {m: (m in loaded) for m in DRM_MODULES} + + # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC + # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the + # one place the state is visible without root AND without debugfs — but the sysfs file + # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel + # command line is the world-readable second source: an explicit `i915.enable_guc=` there + # is authoritative about what was ASKED for, even when the parameter file cannot be read. + uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")} + cmdline_params = [ + token + for token in (read_text("/proc/cmdline") or "").split() + if token.startswith(tuple("%s." % m for m in DRM_MODULES)) + ] + + return { + "dri_present": dri_present, + "devices": devices, + "render_nodes": render_nodes, + "drm_cards": cards, + "pci_display": pci_display, + "modules": modules, + "uc": uc, + "cmdline_params": cmdline_params, + } + + # --- Storage ------------------------------------------------------------------------ PSEUDO_FS = { "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore", @@ -1095,7 +1304,15 @@ def collect_os(): # we only DETECT, and the role skips the SMART checks with an accurate reason. def collect_tools(): tools = {} - for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"): + # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in + # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod + # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and + # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo + # is not needed on the host at all, because Plex ships its own libva and media driver + # inside its container. They are probed so the report can state their presence or absence + # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted + # to install them. This role installs nothing. + for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"): tools[tool] = which(tool) return tools @@ -1127,6 +1344,7 @@ def main(): "os": collect_os(), "dmi": collect_dmi(), "cpu": collect_cpu(), + "graphics": collect_graphics(), "memory": collect_memory(), "thermal": collect_thermal(), "pressure": collect_pressure(), diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 1e34098..20dcdb5 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -652,6 +652,204 @@ ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '') ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }} +# --- Graphics / render device ---------------------------------------------------------- +# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through +# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back +# to software transcoding on its own and the only symptom is a hot CPU and stuttering +# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that +# binds a different DRM driver all produce exactly that silent degradation. It was measured +# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once +# decays silently, so it is re-measured on every health run instead. +# +# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the +# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink, +# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs +# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role +# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400), +# and it reports as an honest UNKNOWN with the reason rather than as a 0. +# +# THE THREE VERDICTS ARE SPLIT ON PURPOSE: +# graphics.render_device the OUTCOME — can anything hardware-transcode on this node. +# graphics.drm_driver the CAUSE side — what is bound, and is it what this site expects. +# graphics.media_stack INFO only. GuC/HuC and host VAAPI userspace both belong here and +# NEITHER is ever a failure; see the task's own comment for why. +- name: Assess — locate the display device and its bound driver (once, shared by three checks) + ansible.builtin.set_fact: + node_health_render: + # Display-class PCI devices (base class 0x03), driver bound or not. This is the + # question "is there a GPU in this machine at all", and it is answered from the PCI + # bus rather than from /sys/class/drm because a card* entry only exists once a driver + # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look + # identical to a machine with no GPU. + display_devices: "{{ g.pci_display }}" + bound_drivers: >- + {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }} + # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD* + # is an anomaly, not a device, and must not be counted as one. + render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}" + # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the + # difference between "a GPU is bound" and "this GPU can do media work". + media_engines: >- + {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }} + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + +- name: Assess — DRM render node for hardware transcoding + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.render_device', + 'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)', + 'status': render_status, + 'detail': render_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is + # then nothing to grade, and the same convention thermal.max already uses for "no sensor + # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex — + # a GPU IS on the bus and there is still no render node. + render_status: >- + {{ 'SKIP' if (r.display_devices | length) == 0 + else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }} + dri_listing: >- + {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0 + else 'nothing' }} + render_detail: >- + {%- if (r.display_devices | length) == 0 -%} + this machine exposes no display-class PCI device (base class 0x03) at all, so there is + no GPU here for a render node to belong to and hardware transcoding is not available on + this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices, + not assumed. + {%- elif (r.render_chardevs | length) > 0 -%} + {{ r.render_chardevs | length }} render node(s): + {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }}, + owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }} + gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}. + /dev/dri holds: {{ dri_listing }}. + THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal: + `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt + node may legitimately land on a different number (all three prod nodes read gid 992 on + 2026-09-30), and Plex's container entrypoint stats the device and joins whichever group + owns it. Drift is therefore visible in this line without being a failure. + {%- else -%} + HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the + PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is + no usable character-special render node: /dev/dri + {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}. + {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', ')) + ~ ' — present, but NOT character special, which is an anomaly in its own right. ') + if (g.render_nodes | length) > 0 else '' }} + Nothing fails loudly in this state — Plex silently falls back to software transcoding + and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM + driver bound at all, then whether the onboard IGD is still enabled in BIOS. + {%- endif -%} + +- name: Assess — DRM driver bound to the display device + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.drm_driver', + 'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')', + 'status': driver_status, + 'detail': driver_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + expected: "{{ node_health_graphics_expected_driver }}" + # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device + # still works, but the userspace media stack is not the same one, so it is a change the + # operator must see before a transcode regression surfaces as a user complaint. FAIL is + # for nothing bound at all, which is the cause of a missing render node. + driver_status: >- + {{ 'SKIP' if (r.display_devices | length) == 0 + else ('FAIL' if (r.bound_drivers | length) == 0 + else ('PASS' if expected in r.bound_drivers else 'WARN')) }} + driver_detail: >- + {%- if (r.display_devices | length) == 0 -%} + no display-class PCI device on this machine, so no DRM driver can be bound to one. + Measured from /sys/bus/pci/devices. + {%- else -%} + {% for p in r.display_devices %}{{ p.address }} + {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }} + -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}. + {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver')) + | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0 + else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }} + Module load state (/proc/modules, corroboration only — a built-in driver never appears + there, the bound-driver symlink above is what decides): + {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string')) + | map('join', '=') | join(', ') }}. + {%- if (r.bound_drivers | length) == 0 %} + NOTHING IS BOUND to the display device. That is why there is no render node: the DRM + subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the + module exists for this kernel and whether it was blacklisted or failed to probe. + {%- elif expected in r.bound_drivers %} + This is the expected driver for this site (node_health_graphics_expected_driver). + {%- else %} + EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still + works, so this is not an outage — but the two DRM drivers do not share a userspace + media stack, and a libva/iHD build that transcodes correctly against one is not + automatically right for the other. Confirm Plex still hardware-transcodes before + treating this as benign. + {%- endif %} + {%- endif -%} + +# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT: +# +# GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It +# costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work +# through the media engines. Grading it would put a permanent WARN on a configuration the +# operator has chosen. +# No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media +# driver inside its container, so the host needs neither. Flagging it would invite someone +# to `apt install vainfo` on a node to clear a health check, and this role installs nothing. +# +# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs* +# (video-enhancement) are what a transcode actually runs on. +- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational) + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.media_stack', + 'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)', + 'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'), + 'detail': media_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site + # that set the expected driver to amdgpu must get a clean "no such parameters" line + # instead of an undefined-key traceback. + guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}" + guc_text: >- + {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none) + else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true)) + if guc is mapping + else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }} + media_detail: >- + {%- if (r.display_devices | length) == 0 -%} + no display-class PCI device on this machine, so there is no media engine, no GuC/HuC + firmware and nothing to transcode with. + {%- else -%} + media engines: + {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0 + else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }} + (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}). + GuC/HuC: {{ guc_text }}. + {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0 + else 'No i915/xe parameter on the kernel cmdline' }}. + GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs + low-power H.264 encode quality only, and both decode and encode still run on the + engines above. + Host VAAPI userspace: vainfo + {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo + else 'NOT installed on this host' }} + — also never graded, because Plex ships its own libva and media driver inside its + container, so the host needs none. This role installs nothing. + {%- endif -%} + # --- Thermals ----------------------------------------------------------------------- - name: Assess — temperatures ansible.builtin.set_fact: diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 41ebb69..7e9f5ab 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -136,6 +136,37 @@ {% endfor %} {% endfor %} +-------------------------------------------------------------------------------- + GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) +-------------------------------------------------------------------------------- +{% set gfx = node_health_facts.graphics %} +{% if gfx.pci_display %} +{% for p in gfx.pci_display %} + pci {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }} + driver {{ p.driver | default('NONE BOUND', true) }} +{% endfor %} +{% else %} + pci no display-class device (PCI base class 0x03) on this machine — no GPU here +{% endif %} +{% for c in gfx.drm_cards %} + drm {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }} engines {{ c.engines | join(' ') | default('none exposed', true) }} +{% endfor %} +{% if gfx.dri_present %} +{% for d in gfx.devices %} + dev {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }} {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }} +{% endfor %} +{% else %} + dev /dev/dri DOES NOT EXIST — no DRM device node on this node at all +{% endif %} +{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may + legitimately move it and Plex's container init joins whichever group owns the device. See + defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #} +{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %} + GuC/HuC {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }} + {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }} + modules {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides) + vainfo {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }} + -------------------------------------------------------------------------------- PRESSURE STALL (/proc/pressure — % of wall time stalled) -------------------------------------------------------------------------------- -- 2.54.0 From 7df72db4877da0c983609ef6330e6d9831efac34 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 16:12:56 -0500 Subject: [PATCH 2/2] docs(decisions): record what the CI gate actually reported, not the tick MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The PR's required check went green as "Successful in 1m40s" — a real run, not "Has been skipped". But .gitea/workflows/gate.yml:138 is `./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot distinguish them. Fetched the job log and read the gate's own headline: MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability). So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that plainly rather than leaving "CI green" to be read as rc 0. Also corrects a claim that had gone stale within the same entry: CI ran two stages my worktree run could not, and both passed — gitea-settings (the live Gitea matches infra/gitea/expected-branch-protection.json; runners online 1) and approval-policy with real PR context (no sensitive paths touched, so infra/ansible/** needs no human reviewer). My local run skipped both, and the handoff said branch protection was UNVERIFIED. It is now measured, so saying otherwise would have left a known-false line in the log. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-09-30-node-health-render-device.md | 63 ++++++++++++++++--- 1 file changed, 56 insertions(+), 7 deletions(-) diff --git a/decisions/2026-09-30-node-health-render-device.md b/decisions/2026-09-30-node-health-render-device.md index c6d3cd9..d5438d9 100644 --- a/decisions/2026-09-30-node-health-render-device.md +++ b/decisions/2026-09-30-node-health-render-device.md @@ -163,6 +163,48 @@ know the shim took effect rather than the git stages silently degrading. `ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not claiming to have matched a lint convention that does not exist. +### The CI gate on the PR — a green tick that is exit 3, read from the log + +The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"** +on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with +"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s. + +**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is +`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job +log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this +Gitea, the web route serves it) and read the gate's own headline: + + [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. + [fail] Stages that did not run: spec-immutability + +**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea +check is the tolerance swallowing that 3, not a rc 0. + +**CI verified two things my local run could not, and both came back green:** + + [ ok ] stage gitea-settings: pass + [ ok ] enable_push = false enable_status_check = true + [ ok ] required_approvals = 1 block_admin_merge_override = true + [ ok ] block_on_rejected_reviews = true dismiss_stale_approvals = true + [ ok ] runners online: 1 (>= 1) + [ ok ] Gitea settings match the declaration. + [ ok ] stage approval-policy: pass + [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver. + +My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give +`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip +there would have failed the job. That is measured confirmation that the live Gitea matches +`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human +reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too +(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well. + +Corroborated independently: `GET /repos/.../branches/main` → +`protected=true user_can_push=false enable_status_check=true required_approvals=1`, +contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind +status=online`. And note the commit's **combined** state is `pending`, not success, because +`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is +not in the required list, so it does not block, but the PR is not "all green" either. + The gate's `lint` stage passed with one `line too long` **warning** on `roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well above my insertion point, and warnings do not fail the stage. @@ -222,13 +264,20 @@ above my insertion point, and warnings do not fail the stage. device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a node-level check that installs nothing. -- **The gate did not establish greenness.** It returned **rc 3**, and three stages did not run - (`spec-immutability`, `gitea-settings`, `conformance`). So: the offline L0/L1 subset passed, - the live Gitea branch-protection rule is **UNVERIFIED** from this run, and L2/L3 conformance - did not execute. I am not describing this PR as green. -- **No claim about the live platform beyond what the run printed.** Per CLAUDE.md §2b this repo - is not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. - I did not query Gitea's branch protection or runner state, so I assert nothing about either. +- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local + run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped + one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not + describing this PR as green**, and the green Gitea tick must not be read as one: the workflow + line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3. +- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever + runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed. +- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my + worktree nor the CI workspace, so nothing confirms this change does not drift from the spec. + It is the expected, tolerated skip — but it is still a skip, not a pass. +- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is + not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. + Branch protection and runner state I did query directly, and the CI `gitea-settings` stage + passed — so those two I assert on measurement, not on this file's prose. - **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer exists. That is real, it is out of this change's scope, and it is unclaimed work. -- 2.54.0