feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
0fbf544d5e
commit
a78f010368
6 files changed
+785
-4
No files matched your search
@@ -652,6 +652,204 @@
|
||||
~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
|
||||
~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
|
||||
|
||||
# --- Graphics / render device ----------------------------------------------------------
|
||||
# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
|
||||
# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
|
||||
# to software transcoding on its own and the only symptom is a hot CPU and stuttering
|
||||
# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
|
||||
# binds a different DRM driver all produce exactly that silent degradation. It was measured
|
||||
# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
|
||||
# decays silently, so it is re-measured on every health run instead.
|
||||
#
|
||||
# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
|
||||
# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
|
||||
# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
|
||||
# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
|
||||
# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
|
||||
# and it reports as an honest UNKNOWN with the reason rather than as a 0.
|
||||
#
|
||||
# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
|
||||
# graphics.render_device the OUTCOME — can anything hardware-transcode on this node.
|
||||
# graphics.drm_driver the CAUSE side — what is bound, and is it what this site expects.
|
||||
# graphics.media_stack INFO only. GuC/HuC and host VAAPI userspace both belong here and
|
||||
# NEITHER is ever a failure; see the task's own comment for why.
|
||||
- name: Assess — locate the display device and its bound driver (once, shared by three checks)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_render:
|
||||
# Display-class PCI devices (base class 0x03), driver bound or not. This is the
|
||||
# question "is there a GPU in this machine at all", and it is answered from the PCI
|
||||
# bus rather than from /sys/class/drm because a card* entry only exists once a driver
|
||||
# has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
|
||||
# identical to a machine with no GPU.
|
||||
display_devices: "{{ g.pci_display }}"
|
||||
bound_drivers: >-
|
||||
{{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
|
||||
# Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
|
||||
# is an anomaly, not a device, and must not be counted as one.
|
||||
render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
|
||||
# Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
|
||||
# difference between "a GPU is bound" and "this GPU can do media work".
|
||||
media_engines: >-
|
||||
{{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
|
||||
vars:
|
||||
ansible_become: false
|
||||
g: "{{ node_health_facts.graphics }}"
|
||||
|
||||
- name: Assess — DRM render node for hardware transcoding
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'graphics.render_device',
|
||||
'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
|
||||
'status': render_status,
|
||||
'detail': render_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
g: "{{ node_health_facts.graphics }}"
|
||||
r: "{{ node_health_render }}"
|
||||
# SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
|
||||
# then nothing to grade, and the same convention thermal.max already uses for "no sensor
|
||||
# on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
|
||||
# a GPU IS on the bus and there is still no render node.
|
||||
render_status: >-
|
||||
{{ 'SKIP' if (r.display_devices | length) == 0
|
||||
else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
|
||||
dri_listing: >-
|
||||
{{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
|
||||
else 'nothing' }}
|
||||
render_detail: >-
|
||||
{%- if (r.display_devices | length) == 0 -%}
|
||||
this machine exposes no display-class PCI device (base class 0x03) at all, so there is
|
||||
no GPU here for a render node to belong to and hardware transcoding is not available on
|
||||
this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
|
||||
not assumed.
|
||||
{%- elif (r.render_chardevs | length) > 0 -%}
|
||||
{{ r.render_chardevs | length }} render node(s):
|
||||
{% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
|
||||
owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
|
||||
gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
|
||||
/dev/dri holds: {{ dri_listing }}.
|
||||
THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
|
||||
`render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
|
||||
node may legitimately land on a different number (all three prod nodes read gid 992 on
|
||||
2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
|
||||
owns it. Drift is therefore visible in this line without being a failure.
|
||||
{%- else -%}
|
||||
HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
|
||||
PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
|
||||
no usable character-special render node: /dev/dri
|
||||
{{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
|
||||
{{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
|
||||
~ ' — present, but NOT character special, which is an anomaly in its own right. ')
|
||||
if (g.render_nodes | length) > 0 else '' }}
|
||||
Nothing fails loudly in this state — Plex silently falls back to software transcoding
|
||||
and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
|
||||
driver bound at all, then whether the onboard IGD is still enabled in BIOS.
|
||||
{%- endif -%}
|
||||
|
||||
- name: Assess — DRM driver bound to the display device
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'graphics.drm_driver',
|
||||
'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
|
||||
'status': driver_status,
|
||||
'detail': driver_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
g: "{{ node_health_facts.graphics }}"
|
||||
r: "{{ node_health_render }}"
|
||||
expected: "{{ node_health_graphics_expected_driver }}"
|
||||
# WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
|
||||
# still works, but the userspace media stack is not the same one, so it is a change the
|
||||
# operator must see before a transcode regression surfaces as a user complaint. FAIL is
|
||||
# for nothing bound at all, which is the cause of a missing render node.
|
||||
driver_status: >-
|
||||
{{ 'SKIP' if (r.display_devices | length) == 0
|
||||
else ('FAIL' if (r.bound_drivers | length) == 0
|
||||
else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
|
||||
driver_detail: >-
|
||||
{%- if (r.display_devices | length) == 0 -%}
|
||||
no display-class PCI device on this machine, so no DRM driver can be bound to one.
|
||||
Measured from /sys/bus/pci/devices.
|
||||
{%- else -%}
|
||||
{% for p in r.display_devices %}{{ p.address }}
|
||||
{{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
|
||||
-> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
|
||||
{{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
|
||||
| map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
|
||||
else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
|
||||
Module load state (/proc/modules, corroboration only — a built-in driver never appears
|
||||
there, the bound-driver symlink above is what decides):
|
||||
{{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
|
||||
| map('join', '=') | join(', ') }}.
|
||||
{%- if (r.bound_drivers | length) == 0 %}
|
||||
NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
|
||||
subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
|
||||
module exists for this kernel and whether it was blacklisted or failed to probe.
|
||||
{%- elif expected in r.bound_drivers %}
|
||||
This is the expected driver for this site (node_health_graphics_expected_driver).
|
||||
{%- else %}
|
||||
EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
|
||||
works, so this is not an outage — but the two DRM drivers do not share a userspace
|
||||
media stack, and a libva/iHD build that transcodes correctly against one is not
|
||||
automatically right for the other. Confirm Plex still hardware-transcodes before
|
||||
treating this as benign.
|
||||
{%- endif %}
|
||||
{%- endif -%}
|
||||
|
||||
# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
|
||||
#
|
||||
# GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
|
||||
# costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
|
||||
# through the media engines. Grading it would put a permanent WARN on a configuration the
|
||||
# operator has chosen.
|
||||
# No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
|
||||
# driver inside its container, so the host needs neither. Flagging it would invite someone
|
||||
# to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
|
||||
#
|
||||
# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
|
||||
# (video-enhancement) are what a transcode actually runs on.
|
||||
- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'graphics.media_stack',
|
||||
'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
|
||||
'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
|
||||
'detail': media_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
g: "{{ node_health_facts.graphics }}"
|
||||
r: "{{ node_health_render }}"
|
||||
# `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
|
||||
# that set the expected driver to amdgpu must get a clean "no such parameters" line
|
||||
# instead of an undefined-key traceback.
|
||||
guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
|
||||
guc_text: >-
|
||||
{{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
|
||||
else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
|
||||
if guc is mapping
|
||||
else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
|
||||
media_detail: >-
|
||||
{%- if (r.display_devices | length) == 0 -%}
|
||||
no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
|
||||
firmware and nothing to transcode with.
|
||||
{%- else -%}
|
||||
media engines:
|
||||
{{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
|
||||
else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
|
||||
(all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
|
||||
GuC/HuC: {{ guc_text }}.
|
||||
{{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
|
||||
else 'No i915/xe parameter on the kernel cmdline' }}.
|
||||
GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
|
||||
low-power H.264 encode quality only, and both decode and encode still run on the
|
||||
engines above.
|
||||
Host VAAPI userspace: vainfo
|
||||
{{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
|
||||
else 'NOT installed on this host' }}
|
||||
— also never graded, because Plex ships its own libva and media driver inside its
|
||||
container, so the host needs none. This role installs nothing.
|
||||
{%- endif -%}
|
||||
|
||||
# --- Thermals -----------------------------------------------------------------------
|
||||
- name: Assess — temperatures
|
||||
ansible.builtin.set_fact:
|
||||
|
||||
Reference in new issue
Block a user