feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
0fbf544d5e
commit
a78f010368
6 files changed
+785
-4
No files matched your search
@@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
|
||||
## Contract
|
||||
|
||||
- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
|
||||
and — only when it can escalate — `smartctl`. It writes nothing to the node and
|
||||
**installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
|
||||
- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`
|
||||
(and `lspci` only if it happens to be installed), and — only when it can escalate —
|
||||
`smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
|
||||
absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
|
||||
package installation is `node_baseline`'s job, never a health check's, never a manual
|
||||
`apt-get`).
|
||||
@@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
|
||||
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
|
||||
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
|
||||
| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) |
|
||||
| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) |
|
||||
| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why |
|
||||
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
|
||||
| `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
|
||||
| `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
|
||||
@@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t
|
||||
back to re-run the role that had already run.
|
||||
|
||||
|
||||
## The render-device check
|
||||
|
||||
Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing,
|
||||
nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom
|
||||
is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard
|
||||
IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly
|
||||
that silent degradation.
|
||||
|
||||
The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A
|
||||
fact measured once decays silently**, so it is re-measured on every health run instead of
|
||||
living in a runbook. Nothing is installed, no module is loaded, no group is changed — the
|
||||
role's read-only contract is unchanged.
|
||||
|
||||
| fact | where it comes from | root? |
|
||||
|---|---|---|
|
||||
| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no |
|
||||
| owning group **name** for the render node's gid | `grp.getgrgid()` | no |
|
||||
| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no |
|
||||
| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no |
|
||||
| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no |
|
||||
| module load state | `/proc/modules` | no |
|
||||
| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** |
|
||||
|
||||
### The PCI scan is separate from the DRM scan on purpose
|
||||
|
||||
`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on
|
||||
the bus with nothing attached has no `card0` at all — and that is precisely the failure being
|
||||
hunted. So the display device is looked for on the PCI bus independently, from sysfs, which
|
||||
is always there. That is what lets `graphics.render_device` distinguish
|
||||
|
||||
- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention
|
||||
`thermal.max` uses for "no sensor on this hardware"), from
|
||||
- *a GPU is on the bus and there is still no render node* → **FAIL**.
|
||||
|
||||
`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as
|
||||
optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing
|
||||
in this repo guarantees it. It did turn out to be present on all three prod nodes
|
||||
(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending
|
||||
on that would mean depending on an undeclared package. Absent, `description` is `null`, the
|
||||
numeric IDs are printed instead, and no verdict changes.
|
||||
|
||||
### The render node's GID is reported, never asserted
|
||||
|
||||
The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on
|
||||
2026-09-30. The check does **not** compare against `992`, and must not be changed to.
|
||||
`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid
|
||||
from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own
|
||||
container entrypoint stats the device and joins whichever group owns it, so drift genuinely
|
||||
is not a fault. Recording the observed group name and gid in the detail line makes drift
|
||||
**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a
|
||||
reinstall.
|
||||
|
||||
What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver`
|
||||
(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a
|
||||
**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace
|
||||
media stack and a libva/iHD build that transcodes correctly against one is not automatically
|
||||
right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing.
|
||||
|
||||
### Two things that are informational and must stay that way
|
||||
|
||||
- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It
|
||||
costs low-power H.264 encode *quality* only; decode and encode both still run on the media
|
||||
engines. Grading it would put a permanent WARN on a chosen configuration.
|
||||
- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and
|
||||
media driver inside its container, so the host needs neither. Flagging it would invite
|
||||
someone to `apt install vainfo` on a node to clear a health check — and this role installs
|
||||
nothing.
|
||||
|
||||
`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually
|
||||
cannot read it. It reports as an honest `NOT READ — <reason>` rather than a fabricated `0`,
|
||||
with `/proc/cmdline` as the world-readable second source for an explicit boot override.
|
||||
|
||||
## The EFI memattr check
|
||||
|
||||
`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
|
||||
@@ -273,3 +348,10 @@ stays `null` — no claim is made either way.
|
||||
but the wear/endurance context SKIPs rather than guessing.
|
||||
- `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
|
||||
unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
|
||||
- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod
|
||||
node has a working render device, and the role is read-only so one cannot be taken away to
|
||||
prove it. The failure branches were exercised against synthetic fact sets only. The same
|
||||
applies to the `xe`-bound WARN: no node here has `xe` bound.
|
||||
- `graphics.*` proves the render **device** exists and which driver owns it. It does **not**
|
||||
prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container,
|
||||
with Plex's own libva, which is out of scope for a node-level check that installs nothing.
|
||||
Reference in new issue
Block a user