feat(node-health): probe the Intel render device, read-only
gate / gate (pull_request) Successful in 1m40s
gate / conformance (pull_request) Has been skipped

The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.

The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.

All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.

Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:

  graphics.render_device  the outcome — a usable character-special renderD* node.
                          FAIL only when a display device IS on the PCI bus and
                          there is still no render node. SKIP when the machine has
                          no display device at all (a VM guest), which is why the
                          PCI scan is separate from the DRM scan: /sys/class/drm/
                          card* exists only once a driver has BOUND.
  graphics.drm_driver     what is bound, against the new
                          node_health_graphics_expected_driver default (i915), so
                          the comparison is data and not task logic. `xe` bound in
                          place of i915 is a WARN, not a FAIL: the device works but
                          the userspace media stack differs.
  graphics.media_stack    INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
                          is an accepted state that costs low-power H.264 encode
                          quality only, and no host vainfo is not a defect at all
                          because Plex ships its own libva inside its container.

The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-30 13:04:43 -05:00
1 parent 0fbf544d5e
commit a78f010368
6 files changed
+785 -4

No files matched your search

@@ -652,6 +652,204 @@
~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
# --- Graphics / render device ----------------------------------------------------------
# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
# to software transcoding on its own and the only symptom is a hot CPU and stuttering
# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
# binds a different DRM driver all produce exactly that silent degradation. It was measured
# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
# decays silently, so it is re-measured on every health run instead.
#
# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
# and it reports as an honest UNKNOWN with the reason rather than as a 0.
#
# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
# graphics.render_device the OUTCOME — can anything hardware-transcode on this node.
# graphics.drm_driver the CAUSE side — what is bound, and is it what this site expects.
# graphics.media_stack INFO only. GuC/HuC and host VAAPI userspace both belong here and
# NEITHER is ever a failure; see the task's own comment for why.
- name: Assess — locate the display device and its bound driver (once, shared by three checks)
ansible.builtin.set_fact:
node_health_render:
# Display-class PCI devices (base class 0x03), driver bound or not. This is the
# question "is there a GPU in this machine at all", and it is answered from the PCI
# bus rather than from /sys/class/drm because a card* entry only exists once a driver
# has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
# identical to a machine with no GPU.
display_devices: "{{ g.pci_display }}"
bound_drivers: >-
{{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
# Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
# is an anomaly, not a device, and must not be counted as one.
render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
# Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
# difference between "a GPU is bound" and "this GPU can do media work".
media_engines: >-
{{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
vars:
ansible_become: false
g: "{{ node_health_facts.graphics }}"
- name: Assess — DRM render node for hardware transcoding
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'graphics.render_device',
'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
'status': render_status,
'detail': render_detail }] }}"
vars:
ansible_become: false
g: "{{ node_health_facts.graphics }}"
r: "{{ node_health_render }}"
# SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
# then nothing to grade, and the same convention thermal.max already uses for "no sensor
# on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
# a GPU IS on the bus and there is still no render node.
render_status: >-
{{ 'SKIP' if (r.display_devices | length) == 0
else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
dri_listing: >-
{{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
else 'nothing' }}
render_detail: >-
{%- if (r.display_devices | length) == 0 -%}
this machine exposes no display-class PCI device (base class 0x03) at all, so there is
no GPU here for a render node to belong to and hardware transcoding is not available on
this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
not assumed.
{%- elif (r.render_chardevs | length) > 0 -%}
{{ r.render_chardevs | length }} render node(s):
{% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
/dev/dri holds: {{ dri_listing }}.
THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
`render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
node may legitimately land on a different number (all three prod nodes read gid 992 on
2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
owns it. Drift is therefore visible in this line without being a failure.
{%- else -%}
HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
no usable character-special render node: /dev/dri
{{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
{{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
~ ' — present, but NOT character special, which is an anomaly in its own right. ')
if (g.render_nodes | length) > 0 else '' }}
Nothing fails loudly in this state — Plex silently falls back to software transcoding
and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
driver bound at all, then whether the onboard IGD is still enabled in BIOS.
{%- endif -%}
- name: Assess — DRM driver bound to the display device
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'graphics.drm_driver',
'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
'status': driver_status,
'detail': driver_detail }] }}"
vars:
ansible_become: false
g: "{{ node_health_facts.graphics }}"
r: "{{ node_health_render }}"
expected: "{{ node_health_graphics_expected_driver }}"
# WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
# still works, but the userspace media stack is not the same one, so it is a change the
# operator must see before a transcode regression surfaces as a user complaint. FAIL is
# for nothing bound at all, which is the cause of a missing render node.
driver_status: >-
{{ 'SKIP' if (r.display_devices | length) == 0
else ('FAIL' if (r.bound_drivers | length) == 0
else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
driver_detail: >-
{%- if (r.display_devices | length) == 0 -%}
no display-class PCI device on this machine, so no DRM driver can be bound to one.
Measured from /sys/bus/pci/devices.
{%- else -%}
{% for p in r.display_devices %}{{ p.address }}
{{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
-> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
{{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
| map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
Module load state (/proc/modules, corroboration only — a built-in driver never appears
there, the bound-driver symlink above is what decides):
{{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
| map('join', '=') | join(', ') }}.
{%- if (r.bound_drivers | length) == 0 %}
NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
module exists for this kernel and whether it was blacklisted or failed to probe.
{%- elif expected in r.bound_drivers %}
This is the expected driver for this site (node_health_graphics_expected_driver).
{%- else %}
EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
works, so this is not an outage — but the two DRM drivers do not share a userspace
media stack, and a libva/iHD build that transcodes correctly against one is not
automatically right for the other. Confirm Plex still hardware-transcodes before
treating this as benign.
{%- endif %}
{%- endif -%}
# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
#
# GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
# costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
# through the media engines. Grading it would put a permanent WARN on a configuration the
# operator has chosen.
# No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
# driver inside its container, so the host needs neither. Flagging it would invite someone
# to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
#
# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
# (video-enhancement) are what a transcode actually runs on.
- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'graphics.media_stack',
'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
'detail': media_detail }] }}"
vars:
ansible_become: false
g: "{{ node_health_facts.graphics }}"
r: "{{ node_health_render }}"
# `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
# that set the expected driver to amdgpu must get a clean "no such parameters" line
# instead of an undefined-key traceback.
guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
guc_text: >-
{{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
if guc is mapping
else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
media_detail: >-
{%- if (r.display_devices | length) == 0 -%}
no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
firmware and nothing to transcode with.
{%- else -%}
media engines:
{{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
(all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
GuC/HuC: {{ guc_text }}.
{{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
else 'No i915/xe parameter on the kernel cmdline' }}.
GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
low-power H.264 encode quality only, and both decode and encode still run on the
engines above.
Host VAAPI userspace: vainfo
{{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
else 'NOT installed on this host' }}
— also never graded, because Plex ships its own libva and media driver inside its
container, so the host needs none. This role installs nothing.
{%- endif -%}
# --- Thermals -----------------------------------------------------------------------
- name: Assess — temperatures
ansible.builtin.set_fact: