diff --git a/decisions/2026-09-30-node-health-render-device.md b/decisions/2026-09-30-node-health-render-device.md new file mode 100644 index 0000000..d5438d9 --- /dev/null +++ b/decisions/2026-09-30-node-health-render-device.md @@ -0,0 +1,283 @@ +--- +date: 2026-09-30 +branch: feat/node-health-render-device +title: The Intel render device is now self-re-measuring, not a fact someone checked once +--- + +## What changed + +### Why this probe has value at all + +**Nothing else in the platform would notice the device going missing.** Plex exposes no +ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the +manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node +*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly +rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU. +This check is the only thing in the system that would surface it, which is the actual +justification for the role carrying it. + +### The change + +`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module +was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working +Intel render device (measured by hand on all three on 2026-09-30) and the point of this change +is that the measurement now **re-runs forever** instead of decaying silently in a handoff note. + +The failure this exists to catch is silent by construction. Plex hardware-transcodes through +`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables +the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls +back to software transcoding on its own and the only symptom is a hot CPU. + +Four files: + +- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged** + pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`, + `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink + target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class + `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the + `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and + `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the + existing tool probe; both are optional and neither is depended on. +- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device` + (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack` + (INFO/SKIP). +- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the + check makes is data rather than task logic. +- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and + `roles/node_health/README.md` gained a section explaining the design and two new "known + limits". + +Three design rulings worth stating, because each is the kind of thing a later edit would get +wrong: + +1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**, + mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is + a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from + `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats + the device and joins whichever group owns it, so drift genuinely is not a fault. The + observed group name and number appear in the detail line, which makes drift *visible* + without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall. +2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists + only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would + otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately + is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same + convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on + the bus and there is still no render node" is a **FAIL**. +3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off + (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality* + only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver + inside its container. Grading either would put a permanent WARN on a chosen configuration, + or invite someone to `apt install vainfo` on a node to clear a health check, and this role + installs nothing. + +`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two +drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is +bound to nothing. + +## What I verified + +### The real prod run (read-only, all three nodes) + +`--list-hosts` first, every time. It named exactly three hosts and nothing else: + + play #1 (k3s_prod): Inspect node hardware health (read-only) + pattern: ['k3s_prod'] + hosts (3): k3s-prod-3 k3s-prod-2 k3s-prod-1 + +Then `--syntax-check` (rc 0), then the run: + + ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod + + k3s-prod-1 : ok=51 changed=3 unreachable=0 failed=0 skipped=1 + k3s-prod-2 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 + k3s-prod-3 : ok=51 changed=2 unreachable=0 failed=0 skipped=1 + PLAY rc=0 + +(The `changed=` counts are the *controller-side* artifact writes under +`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed — +the collector task is `changed_when: false` and the role touches nothing.) + +All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall: + + k3s-prod-1 VERDICT 0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO + k3s-prod-2 VERDICT 0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO + k3s-prod-3 VERDICT 0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO + + [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*) + [PASS] graphics.drm_driver — DRM driver bound to the display device (expected i915) + [INFO] graphics.media_stack — Media engines, GuC/HuC firmware and host VAAPI userspace (informational) + +(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not +mine; none of them moved.) + +The new report section, **byte-identical on all three nodes**: + + -------------------------------------------------------------------------------- + GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) + -------------------------------------------------------------------------------- + pci 0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02) + driver i915 + drm card0 driver i915 at 0000:00:02.0 engines bcs0 rcs0 vcs0 vecs0 + dev card0 character special mode 0660 root:video gid 44 + dev renderD128 character special mode 0660 root:render gid 992 + GuC/HuC NOT READ — /sys/module/i915/parameters/enable_guc is not + readable without root (i915 module parameters are mode 0400) + no i915/xe parameter on the kernel cmdline + modules amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides) + vainfo NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container + +That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128` +character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode +`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)` +CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`. + +**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three +prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was +absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in +`node_baseline_packages` and depending on an undeclared package would be wrong — but the +comments and README that asserted it was missing were factually wrong and are now corrected. +Its presence is why the `pci` line above carries a readable device name rather than bare IDs. + +### The gate + + DEV_KUBECONFIG=/dev/null ENGINE= ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' + GATE rc=3 + +**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly +three stages did not run, all three the known local ceiling from a worktree, and **no fourth +skip**: + + [skip] spec-immutability spec repo not present at /work/../home-network-k3s (set SPEC_REPO) + [skip] gitea-settings SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule + [skip] conformance no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN + + [ ok ] decisions-log: pass [ ok ] lint: pass [ ok ] l0: pass + [ ok ] render-golden: pass [ ok ] l1: pass [ ok ] approval-policy: pass + +Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null` +(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts +the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I +know the shim took effect rather than the git stages silently degrading. + +`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped +`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not +claiming to have matched a lint convention that does not exist. + +### The CI gate on the PR — a green tick that is exit 3, read from the log + +The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"** +on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with +"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s. + +**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is +`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job +log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this +Gitea, the web route serves it) and read the gate's own headline: + + [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. + [fail] Stages that did not run: spec-immutability + +**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea +check is the tolerance swallowing that 3, not a rc 0. + +**CI verified two things my local run could not, and both came back green:** + + [ ok ] stage gitea-settings: pass + [ ok ] enable_push = false enable_status_check = true + [ ok ] required_approvals = 1 block_admin_merge_override = true + [ ok ] block_on_rejected_reviews = true dismiss_stale_approvals = true + [ ok ] runners online: 1 (>= 1) + [ ok ] Gitea settings match the declaration. + [ ok ] stage approval-policy: pass + [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver. + +My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give +`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip +there would have failed the job. That is measured confirmation that the live Gitea matches +`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human +reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too +(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well. + +Corroborated independently: `GET /repos/.../branches/main` → +`protected=true user_can_push=false enable_status_check=true required_approvals=1`, +contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind +status=online`. And note the commit's **combined** state is `pending`, not success, because +`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is +not in the required list, so it does not block, but the PR is not "all green" either. + +The gate's `lint` stage passed with one `line too long` **warning** on +`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well +above my insertion point, and warnings do not fail the stage. + +### Offline work (before the lock was released) + +- `python3 -m py_compile` on the collector: clean. +- `collect_graphics()` executed for real against this workstation's hardware (an unrelated + machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole + structure with no exception, correctly reporting two cards, two render nodes, the bound + driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable + without root* rather than as a fabricated `0`. +- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is + **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports + `NOT READ — ` and `/proc/cmdline` is read as the world-readable second source. An + earlier draft would have silently reported a null. +- Two other bugs found and fixed by running it rather than reasoning about it: + `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and + `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be + excluded from the card scan. +- All three checks and the new template section were rendered through a standalone Jinja + harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real + workstation; GPU present with the render node removed; no display device at all; `xe` bound + in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a + character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and + the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`, + `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the + Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches + Ansible's `trim_blocks=True`. + **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence + whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated + dictionaries, not machines. They are not hardware tests and must not be written up as if + they were. + +## What I did NOT verify + +- **The FAIL path has never fired on a real node.** Every prod node has a working render + device, and this role is read-only, so a device cannot be taken away to prove the check + catches its absence. The failure branches were exercised against **synthetic fact sets + only** (see above). I am not claiming the failure path is tested against hardware. +- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the + module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only + ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of + `/sys/class/drm/card*` under it is something I have assumed matches i915's (same + `device/driver` symlink, same `engine/*` directories). That assumption is unverified. +- **The media-engine sysfs path is confirmed on these nodes but not proven portable.** + `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it + is measured here. I did **not** confirm against kernel source that no other kernel version or + driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO + line, not a false verdict, which is why this failure mode is tolerable. +- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is + honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes; + I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground + truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to + guess it. Reading it would need root, and it is INFO either way, so I left it alone. +- **Not a transcode test.** These checks prove the render *device* exists, is a character + device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a + VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a + node-level check that installs nothing. +- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local + run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped + one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not + describing this PR as green**, and the green Gitea tick must not be read as one: the workflow + line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3. +- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever + runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed. +- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my + worktree nor the CI workspace, so nothing confirms this change does not drift from the spec. + It is the expected, tolerated skip — but it is still a skip, not a pass. +- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is + not evidence about prod; the only prod facts I assert are the ones quoted verbatim above. + Branch protection and runner state I did query directly, and the CI `gitea-settings` stage + passed — so those two I assert on measurement, not on this file's prose. +- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the + `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer + exists. That is real, it is out of this change's scope, and it is unclaimed work. diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index 95f7dd6..88fcb97 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 ## Contract -- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`, - and — only when it can escalate — `smartctl`. It writes nothing to the node and - **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is +- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu` + (and `lspci` only if it happens to be installed), and — only when it can escalate — + `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is `node_baseline`'s job, never a health check's, never a manual `apt-get`). @@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters | | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | +| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) | +| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) | +| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | @@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t back to re-run the role that had already run. +## The render-device check + +Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing, +nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom +is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard +IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly +that silent degradation. + +The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A +fact measured once decays silently**, so it is re-measured on every health run instead of +living in a runbook. Nothing is installed, no module is loaded, no group is changed — the +role's read-only contract is unchanged. + +| fact | where it comes from | root? | +|---|---|---| +| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no | +| owning group **name** for the render node's gid | `grp.getgrgid()` | no | +| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no | +| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no | +| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no | +| module load state | `/proc/modules` | no | +| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** | + +### The PCI scan is separate from the DRM scan on purpose + +`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on +the bus with nothing attached has no `card0` at all — and that is precisely the failure being +hunted. So the display device is looked for on the PCI bus independently, from sysfs, which +is always there. That is what lets `graphics.render_device` distinguish + +- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention + `thermal.max` uses for "no sensor on this hardware"), from +- *a GPU is on the bus and there is still no render node* → **FAIL**. + +`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as +optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing +in this repo guarantees it. It did turn out to be present on all three prod nodes +(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending +on that would mean depending on an undeclared package. Absent, `description` is `null`, the +numeric IDs are printed instead, and no verdict changes. + +### The render node's GID is reported, never asserted + +The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on +2026-09-30. The check does **not** compare against `992`, and must not be changed to. +`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid +from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own +container entrypoint stats the device and joins whichever group owns it, so drift genuinely +is not a fault. Recording the observed group name and gid in the detail line makes drift +**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a +reinstall. + +What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver` +(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a +**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace +media stack and a libva/iHD build that transcodes correctly against one is not automatically +right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing. + +### Two things that are informational and must stay that way + +- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It + costs low-power H.264 encode *quality* only; decode and encode both still run on the media + engines. Grading it would put a permanent WARN on a chosen configuration. +- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and + media driver inside its container, so the host needs neither. Flagging it would invite + someone to `apt install vainfo` on a node to clear a health check — and this role installs + nothing. + +`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually +cannot read it. It reports as an honest `NOT READ — ` rather than a fabricated `0`, +with `/proc/cmdline` as the world-readable second source for an explicit boot override. + ## The EFI memattr check `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. @@ -273,3 +348,10 @@ stays `null` — no claim is made either way. but the wear/endurance context SKIPs rather than guessing. - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs `dmesg`, which needs root. +- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod + node has a working render device, and the role is read-only so one cannot be taken away to + prove it. The failure branches were exercised against synthetic fact sets only. The same + applies to the `xe`-bound WARN: no node here has `xe` bound. +- `graphics.*` proves the render **device** exists and which driver owns it. It does **not** + prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container, + with Plex's own libva, which is out of scope for a node-level check that installs nothing. diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index eae98c9..9b02dbc 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -35,6 +35,24 @@ node_health_try_privileged: true # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene. node_health_smartctl_args: "--json=c --all --nocheck=standby" +# --- Graphics / render device --------------------------------------------------------- +# The DRM driver this site EXPECTS to be bound to its display device. Measured on all three +# prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP +# EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to +# nothing, and that difference matters enough to surface: the two drivers do not share a +# userspace media stack (a libva/iHD build that works against i915 is not automatically the +# right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD +# or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic. +# +# WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning +# group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static +# one, so a rebuilt node may legitimately land on a different number — it was 992 on all +# three prod nodes on 2026-09-30, and an assertion against that literal would be a false +# alarm waiting to happen. Plex's own container entrypoint stats the device and joins +# whichever group owns it, so drift genuinely is not a fault. The observed group name and GID +# are therefore REPORTED in the check detail and never graded. +node_health_graphics_expected_driver: i915 + # --- Thresholds --------------------------------------------------------------------- # All verdicts come from here, so a site can retune without touching task logic. node_health_thresholds: diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index 8eb79e7..d6d6e1a 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -35,10 +35,13 @@ # ============================================================================= import glob +import grp import json import os +import pwd import re import shutil +import stat import subprocess import sys import time @@ -756,6 +759,212 @@ def collect_pressure(): return out +# --- Graphics / DRM render device ----------------------------------------------------- +# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that +# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that +# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software +# transcoding and the only symptom is a hot CPU. So the render device is measured on every +# health run rather than being a fact someone established once by hand. +# +# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are +# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so +# are the module parameters under /sys/module. Nothing here needs root, so nothing here +# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need +# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is +# what actually decides whether the firmware is loaded at all.) +# +# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists +# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no +# card0 at all, and that is precisely the failure this is meant to catch — so the display +# device is looked for on the PCI bus independently, from sysfs, which is always there. +# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as +# optional because it IS optional: pciutils is not declared in node_baseline_packages, so +# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes +# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on +# that would make the collector depend on an undeclared package. A missing lspci leaves +# `description` null, the numeric IDs are reported instead, and no verdict changes. +DISPLAY_PCI_CLASS_PREFIX = "0x03" # PCI base class 03 = display controller (VGA/3D/other) +# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative +# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is +# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact. +DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia") + + +def _uid_name(uid): + try: + return pwd.getpwuid(uid).pw_name + except (KeyError, OverflowError, TypeError, OSError): + return None + + +def _gid_name(gid): + """Group NAME for a numeric gid, or None when nothing resolves it. + + Reported as an OBSERVED value, never asserted against a literal. The `render` group is + a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's + static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own + container init stats the device and joins whatever group owns it, so a moved gid is + drift to notice, not a fault to fail on.""" + try: + return grp.getgrgid(gid).gr_name + except (KeyError, OverflowError, TypeError, OSError): + return None + + +def _module_param(mod, param): + """One /sys/module//parameters/, with WHY it is missing when it is. + + Three outcomes that must not be conflated: a real value; "the module exposes no such + parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no + enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915 + module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an + unprivileged pass legitimately cannot read them. Returning null with the reason keeps + that an honest UNKNOWN instead of a fabricated 0. read_text() also records the + permission denial in UNREADABLE, so the summary lists it with everything else root + would add.""" + path = "/sys/module/%s/parameters/%s" % (mod, param) + if not os.path.exists(path): + return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, " + "built in, or no such knob on this driver)" % (mod, param)} + value = read_text(path) + if value is None: + return {"value": None, "reason": "%s is not readable without root (i915 module " + "parameters are mode 0400)" % path} + return {"value": value, "reason": None} + + +def _dev_node(path): + """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is.""" + try: + st = os.lstat(path) + except OSError as exc: + UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc}) + return None + return { + "path": path, + "name": os.path.basename(path), + "char_device": stat.S_ISCHR(st.st_mode), + "mode": "%04o" % stat.S_IMODE(st.st_mode), + "uid": st.st_uid, + "gid": st.st_gid, + "owner": _uid_name(st.st_uid), + "group": _gid_name(st.st_gid), + } + + +def collect_graphics(): + dri_dir = "/dev/dri" + dri_present = os.path.isdir(dri_dir) + devices = [] + if dri_present: + for path in sorted(glob.glob(dri_dir + "/*")): + # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping + # directories rather than filtering to char devices is deliberate: anything + # in here that is NOT a character special file is an anomaly worth seeing, and + # `char_device: false` is how it becomes visible. + if os.path.isdir(path): + continue + entry = _dev_node(path) + if entry is not None: + devices.append(entry) + render_nodes = [d for d in devices if d["name"].startswith("renderD")] + + # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR + # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped. + cards = [] + for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")): + name = os.path.basename(cdir) + if "-" in name: + continue + driver = None + drvlink = cdir + "/device/driver" + if os.path.islink(drvlink): + driver = os.path.basename(os.path.realpath(drvlink)) + address = None + devlink = cdir + "/device" + if os.path.islink(devlink): + address = os.path.basename(os.path.realpath(devlink)) + # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC) + # engine and vecs* the video-enhancement engine: those two are what a transcode + # actually runs on, so their presence is the difference between "a GPU is bound" + # and "this GPU can do media work". + engines = sorted( + { + os.path.basename(p) + for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*")) + if os.path.isdir(p) + } + ) + cards.append( + { + "card": name, + "driver": driver, + "pci_address": address, + "vendor_id": read_text(cdir + "/device/vendor"), + "device_id": read_text(cdir + "/device/device"), + "engines": engines, + "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))], + } + ) + + # Display-class devices on the PCI bus, driver bound or not. + pci_display = [] + for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")): + class_id = read_text(pdir + "/class") or "" + if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX): + continue + address = os.path.basename(pdir) + driver = None + if os.path.islink(pdir + "/driver"): + driver = os.path.basename(os.path.realpath(pdir + "/driver")) + description = None + rc, out, _ = run(["lspci", "-nn", "-s", address]) + if rc == 0 and out.strip(): + description = out.strip().splitlines()[0] + pci_display.append( + { + "address": address, + "class_id": class_id or None, + "vendor_id": read_text(pdir + "/vendor"), + "device_id": read_text(pdir + "/device"), + "revision": read_text(pdir + "/revision"), + "driver": driver, + "description": description, + } + ) + + loaded = set() + for line in (read_text("/proc/modules") or "").splitlines(): + fields = line.split() + if fields: + loaded.add(fields[0]) + modules = {m: (m in loaded) for m in DRM_MODULES} + + # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC + # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the + # one place the state is visible without root AND without debugfs — but the sysfs file + # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel + # command line is the world-readable second source: an explicit `i915.enable_guc=` there + # is authoritative about what was ASKED for, even when the parameter file cannot be read. + uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")} + cmdline_params = [ + token + for token in (read_text("/proc/cmdline") or "").split() + if token.startswith(tuple("%s." % m for m in DRM_MODULES)) + ] + + return { + "dri_present": dri_present, + "devices": devices, + "render_nodes": render_nodes, + "drm_cards": cards, + "pci_display": pci_display, + "modules": modules, + "uc": uc, + "cmdline_params": cmdline_params, + } + + # --- Storage ------------------------------------------------------------------------ PSEUDO_FS = { "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore", @@ -1095,7 +1304,15 @@ def collect_os(): # we only DETECT, and the role skips the SMART checks with an accurate reason. def collect_tools(): tools = {} - for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"): + # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in + # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod + # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and + # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo + # is not needed on the host at all, because Plex ships its own libva and media driver + # inside its container. They are probed so the report can state their presence or absence + # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted + # to install them. This role installs nothing. + for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"): tools[tool] = which(tool) return tools @@ -1127,6 +1344,7 @@ def main(): "os": collect_os(), "dmi": collect_dmi(), "cpu": collect_cpu(), + "graphics": collect_graphics(), "memory": collect_memory(), "thermal": collect_thermal(), "pressure": collect_pressure(), diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 1e34098..20dcdb5 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -652,6 +652,204 @@ ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '') ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }} +# --- Graphics / render device ---------------------------------------------------------- +# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through +# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back +# to software transcoding on its own and the only symptom is a hot CPU and stuttering +# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that +# binds a different DRM driver all produce exactly that silent degradation. It was measured +# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once +# decays silently, so it is re-measured on every health run instead. +# +# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the +# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink, +# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs +# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role +# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400), +# and it reports as an honest UNKNOWN with the reason rather than as a 0. +# +# THE THREE VERDICTS ARE SPLIT ON PURPOSE: +# graphics.render_device the OUTCOME — can anything hardware-transcode on this node. +# graphics.drm_driver the CAUSE side — what is bound, and is it what this site expects. +# graphics.media_stack INFO only. GuC/HuC and host VAAPI userspace both belong here and +# NEITHER is ever a failure; see the task's own comment for why. +- name: Assess — locate the display device and its bound driver (once, shared by three checks) + ansible.builtin.set_fact: + node_health_render: + # Display-class PCI devices (base class 0x03), driver bound or not. This is the + # question "is there a GPU in this machine at all", and it is answered from the PCI + # bus rather than from /sys/class/drm because a card* entry only exists once a driver + # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look + # identical to a machine with no GPU. + display_devices: "{{ g.pci_display }}" + bound_drivers: >- + {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }} + # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD* + # is an anomaly, not a device, and must not be counted as one. + render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}" + # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the + # difference between "a GPU is bound" and "this GPU can do media work". + media_engines: >- + {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }} + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + +- name: Assess — DRM render node for hardware transcoding + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.render_device', + 'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)', + 'status': render_status, + 'detail': render_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is + # then nothing to grade, and the same convention thermal.max already uses for "no sensor + # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex — + # a GPU IS on the bus and there is still no render node. + render_status: >- + {{ 'SKIP' if (r.display_devices | length) == 0 + else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }} + dri_listing: >- + {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0 + else 'nothing' }} + render_detail: >- + {%- if (r.display_devices | length) == 0 -%} + this machine exposes no display-class PCI device (base class 0x03) at all, so there is + no GPU here for a render node to belong to and hardware transcoding is not available on + this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices, + not assumed. + {%- elif (r.render_chardevs | length) > 0 -%} + {{ r.render_chardevs | length }} render node(s): + {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }}, + owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }} + gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}. + /dev/dri holds: {{ dri_listing }}. + THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal: + `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt + node may legitimately land on a different number (all three prod nodes read gid 992 on + 2026-09-30), and Plex's container entrypoint stats the device and joins whichever group + owns it. Drift is therefore visible in this line without being a failure. + {%- else -%} + HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the + PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is + no usable character-special render node: /dev/dri + {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}. + {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', ')) + ~ ' — present, but NOT character special, which is an anomaly in its own right. ') + if (g.render_nodes | length) > 0 else '' }} + Nothing fails loudly in this state — Plex silently falls back to software transcoding + and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM + driver bound at all, then whether the onboard IGD is still enabled in BIOS. + {%- endif -%} + +- name: Assess — DRM driver bound to the display device + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.drm_driver', + 'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')', + 'status': driver_status, + 'detail': driver_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + expected: "{{ node_health_graphics_expected_driver }}" + # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device + # still works, but the userspace media stack is not the same one, so it is a change the + # operator must see before a transcode regression surfaces as a user complaint. FAIL is + # for nothing bound at all, which is the cause of a missing render node. + driver_status: >- + {{ 'SKIP' if (r.display_devices | length) == 0 + else ('FAIL' if (r.bound_drivers | length) == 0 + else ('PASS' if expected in r.bound_drivers else 'WARN')) }} + driver_detail: >- + {%- if (r.display_devices | length) == 0 -%} + no display-class PCI device on this machine, so no DRM driver can be bound to one. + Measured from /sys/bus/pci/devices. + {%- else -%} + {% for p in r.display_devices %}{{ p.address }} + {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }} + -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}. + {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver')) + | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0 + else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }} + Module load state (/proc/modules, corroboration only — a built-in driver never appears + there, the bound-driver symlink above is what decides): + {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string')) + | map('join', '=') | join(', ') }}. + {%- if (r.bound_drivers | length) == 0 %} + NOTHING IS BOUND to the display device. That is why there is no render node: the DRM + subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the + module exists for this kernel and whether it was blacklisted or failed to probe. + {%- elif expected in r.bound_drivers %} + This is the expected driver for this site (node_health_graphics_expected_driver). + {%- else %} + EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still + works, so this is not an outage — but the two DRM drivers do not share a userspace + media stack, and a libva/iHD build that transcodes correctly against one is not + automatically right for the other. Confirm Plex still hardware-transcodes before + treating this as benign. + {%- endif %} + {%- endif -%} + +# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT: +# +# GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It +# costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work +# through the media engines. Grading it would put a permanent WARN on a configuration the +# operator has chosen. +# No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media +# driver inside its container, so the host needs neither. Flagging it would invite someone +# to `apt install vainfo` on a node to clear a health check, and this role installs nothing. +# +# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs* +# (video-enhancement) are what a transcode actually runs on. +- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational) + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'graphics.media_stack', + 'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)', + 'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'), + 'detail': media_detail }] }}" + vars: + ansible_become: false + g: "{{ node_health_facts.graphics }}" + r: "{{ node_health_render }}" + # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site + # that set the expected driver to amdgpu must get a clean "no such parameters" line + # instead of an undefined-key traceback. + guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}" + guc_text: >- + {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none) + else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true)) + if guc is mapping + else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }} + media_detail: >- + {%- if (r.display_devices | length) == 0 -%} + no display-class PCI device on this machine, so there is no media engine, no GuC/HuC + firmware and nothing to transcode with. + {%- else -%} + media engines: + {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0 + else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }} + (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}). + GuC/HuC: {{ guc_text }}. + {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0 + else 'No i915/xe parameter on the kernel cmdline' }}. + GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs + low-power H.264 encode quality only, and both decode and encode still run on the + engines above. + Host VAAPI userspace: vainfo + {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo + else 'NOT installed on this host' }} + — also never graded, because Plex ships its own libva and media driver inside its + container, so the host needs none. This role installs nothing. + {%- endif -%} + # --- Thermals ----------------------------------------------------------------------- - name: Assess — temperatures ansible.builtin.set_fact: diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 41ebb69..7e9f5ab 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -136,6 +136,37 @@ {% endfor %} {% endfor %} +-------------------------------------------------------------------------------- + GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path) +-------------------------------------------------------------------------------- +{% set gfx = node_health_facts.graphics %} +{% if gfx.pci_display %} +{% for p in gfx.pci_display %} + pci {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }} + driver {{ p.driver | default('NONE BOUND', true) }} +{% endfor %} +{% else %} + pci no display-class device (PCI base class 0x03) on this machine — no GPU here +{% endif %} +{% for c in gfx.drm_cards %} + drm {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }} engines {{ c.engines | join(' ') | default('none exposed', true) }} +{% endfor %} +{% if gfx.dri_present %} +{% for d in gfx.devices %} + dev {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }} {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }} +{% endfor %} +{% else %} + dev /dev/dri DOES NOT EXIST — no DRM device node on this node at all +{% endif %} +{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may + legitimately move it and Plex's container init joins whichever group owns the device. See + defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #} +{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %} + GuC/HuC {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }} + {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }} + modules {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides) + vainfo {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }} + -------------------------------------------------------------------------------- PRESSURE STALL (/proc/pressure — % of wall time stalled) --------------------------------------------------------------------------------