feat(node-health): probe the Intel render device, read-only
gate / gate (pull_request) Successful in 1m40s
gate / conformance (pull_request) Has been skipped

The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.

The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.

All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.

Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:

  graphics.render_device  the outcome — a usable character-special renderD* node.
                          FAIL only when a display device IS on the PCI bus and
                          there is still no render node. SKIP when the machine has
                          no display device at all (a VM guest), which is why the
                          PCI scan is separate from the DRM scan: /sys/class/drm/
                          card* exists only once a driver has BOUND.
  graphics.drm_driver     what is bound, against the new
                          node_health_graphics_expected_driver default (i915), so
                          the comparison is data and not task logic. `xe` bound in
                          place of i915 is a WARN, not a FAIL: the device works but
                          the userspace media stack differs.
  graphics.media_stack    INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
                          is an accepted state that costs low-power H.264 encode
                          quality only, and no host vainfo is not a defect at all
                          because Plex ships its own libva inside its container.

The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-30 13:04:43 -05:00
1 parent 0fbf544d5e
commit a78f010368
6 files changed
+785 -4

No files matched your search

@@ -35,10 +35,13 @@
# =============================================================================
import glob
import grp
import json
import os
import pwd
import re
import shutil
import stat
import subprocess
import sys
import time
@@ -756,6 +759,212 @@ def collect_pressure():
return out
# --- Graphics / DRM render device -----------------------------------------------------
# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that
# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that
# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software
# transcoding and the only symptom is a hot CPU. So the render device is measured on every
# health run rather than being a fact someone established once by hand.
#
# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are
# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so
# are the module parameters under /sys/module. Nothing here needs root, so nothing here
# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need
# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is
# what actually decides whether the firmware is loaded at all.)
#
# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists
# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no
# card0 at all, and that is precisely the failure this is meant to catch — so the display
# device is looked for on the PCI bus independently, from sysfs, which is always there.
# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as
# optional because it IS optional: pciutils is not declared in node_baseline_packages, so
# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes
# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on
# that would make the collector depend on an undeclared package. A missing lspci leaves
# `description` null, the numeric IDs are reported instead, and no verdict changes.
DISPLAY_PCI_CLASS_PREFIX = "0x03" # PCI base class 03 = display controller (VGA/3D/other)
# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative
# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is
# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact.
DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia")
def _uid_name(uid):
try:
return pwd.getpwuid(uid).pw_name
except (KeyError, OverflowError, TypeError, OSError):
return None
def _gid_name(gid):
"""Group NAME for a numeric gid, or None when nothing resolves it.
Reported as an OBSERVED value, never asserted against a literal. The `render` group is
a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's
static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own
container init stats the device and joins whatever group owns it, so a moved gid is
drift to notice, not a fault to fail on."""
try:
return grp.getgrgid(gid).gr_name
except (KeyError, OverflowError, TypeError, OSError):
return None
def _module_param(mod, param):
"""One /sys/module/<mod>/parameters/<param>, with WHY it is missing when it is.
Three outcomes that must not be conflated: a real value; "the module exposes no such
parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no
enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915
module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an
unprivileged pass legitimately cannot read them. Returning null with the reason keeps
that an honest UNKNOWN instead of a fabricated 0. read_text() also records the
permission denial in UNREADABLE, so the summary lists it with everything else root
would add."""
path = "/sys/module/%s/parameters/%s" % (mod, param)
if not os.path.exists(path):
return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, "
"built in, or no such knob on this driver)" % (mod, param)}
value = read_text(path)
if value is None:
return {"value": None, "reason": "%s is not readable without root (i915 module "
"parameters are mode 0400)" % path}
return {"value": value, "reason": None}
def _dev_node(path):
"""stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is."""
try:
st = os.lstat(path)
except OSError as exc:
UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc})
return None
return {
"path": path,
"name": os.path.basename(path),
"char_device": stat.S_ISCHR(st.st_mode),
"mode": "%04o" % stat.S_IMODE(st.st_mode),
"uid": st.st_uid,
"gid": st.st_gid,
"owner": _uid_name(st.st_uid),
"group": _gid_name(st.st_gid),
}
def collect_graphics():
dri_dir = "/dev/dri"
dri_present = os.path.isdir(dri_dir)
devices = []
if dri_present:
for path in sorted(glob.glob(dri_dir + "/*")):
# /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping
# directories rather than filtering to char devices is deliberate: anything
# in here that is NOT a character special file is an anomaly worth seeing, and
# `char_device: false` is how it becomes visible.
if os.path.isdir(path):
continue
entry = _dev_node(path)
if entry is not None:
devices.append(entry)
render_nodes = [d for d in devices if d["name"].startswith("renderD")]
# Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR
# (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped.
cards = []
for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")):
name = os.path.basename(cdir)
if "-" in name:
continue
driver = None
drvlink = cdir + "/device/driver"
if os.path.islink(drvlink):
driver = os.path.basename(os.path.realpath(drvlink))
address = None
devlink = cdir + "/device"
if os.path.islink(devlink):
address = os.path.basename(os.path.realpath(devlink))
# i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC)
# engine and vecs* the video-enhancement engine: those two are what a transcode
# actually runs on, so their presence is the difference between "a GPU is bound"
# and "this GPU can do media work".
engines = sorted(
{
os.path.basename(p)
for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*"))
if os.path.isdir(p)
}
)
cards.append(
{
"card": name,
"driver": driver,
"pci_address": address,
"vendor_id": read_text(cdir + "/device/vendor"),
"device_id": read_text(cdir + "/device/device"),
"engines": engines,
"media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))],
}
)
# Display-class devices on the PCI bus, driver bound or not.
pci_display = []
for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")):
class_id = read_text(pdir + "/class") or ""
if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX):
continue
address = os.path.basename(pdir)
driver = None
if os.path.islink(pdir + "/driver"):
driver = os.path.basename(os.path.realpath(pdir + "/driver"))
description = None
rc, out, _ = run(["lspci", "-nn", "-s", address])
if rc == 0 and out.strip():
description = out.strip().splitlines()[0]
pci_display.append(
{
"address": address,
"class_id": class_id or None,
"vendor_id": read_text(pdir + "/vendor"),
"device_id": read_text(pdir + "/device"),
"revision": read_text(pdir + "/revision"),
"driver": driver,
"description": description,
}
)
loaded = set()
for line in (read_text("/proc/modules") or "").splitlines():
fields = line.split()
if fields:
loaded.add(fields[0])
modules = {m: (m in loaded) for m in DRM_MODULES}
# GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC
# firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the
# one place the state is visible without root AND without debugfs — but the sysfs file
# is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel
# command line is the world-readable second source: an explicit `i915.enable_guc=` there
# is authoritative about what was ASKED for, even when the parameter file cannot be read.
uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")}
cmdline_params = [
token
for token in (read_text("/proc/cmdline") or "").split()
if token.startswith(tuple("%s." % m for m in DRM_MODULES))
]
return {
"dri_present": dri_present,
"devices": devices,
"render_nodes": render_nodes,
"drm_cards": cards,
"pci_display": pci_display,
"modules": modules,
"uc": uc,
"cmdline_params": cmdline_params,
}
# --- Storage ------------------------------------------------------------------------
PSEUDO_FS = {
"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore",
@@ -1095,7 +1304,15 @@ def collect_os():
# we only DETECT, and the role skips the SMART checks with an accurate reason.
def collect_tools():
tools = {}
for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
# lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in
# node_baseline_packages, so nothing in this repo guarantees either. Measured on prod
# 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and
# vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo
# is not needed on the host at all, because Plex ships its own libva and media driver
# inside its container. They are probed so the report can state their presence or absence
# as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted
# to install them. This role installs nothing.
for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"):
tools[tool] = which(tool)
return tools
@@ -1127,6 +1344,7 @@ def main():
"os": collect_os(),
"dmi": collect_dmi(),
"cpu": collect_cpu(),
"graphics": collect_graphics(),
"memory": collect_memory(),
"thermal": collect_thermal(),
"pressure": collect_pressure(),