jakman85andClaude Opus 5
a78f010368
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
219 lines
18 KiB
Django/Jinja
219 lines
18 KiB
Django/Jinja
{# ===========================================================================
|
||
node_health human summary. Rendered ONCE (tasks/report.yml) and used for both
|
||
the on-screen report and the .txt artifact, so the two can never drift.
|
||
|
||
Detail strings are normalised with regex_replace('\s+',' ') because the check
|
||
details are authored as YAML folded scalars in assess.yml and can carry stray
|
||
newlines/indentation from that folding.
|
||
=========================================================================== #}
|
||
{% macro fmt(text, width=74, pad=10) -%}
|
||
{{ text | regex_replace('\s+', ' ') | trim | wordwrap(width) | indent(pad, first=True) }}
|
||
{%- endmacro %}
|
||
{% macro gib(n) -%}
|
||
{{ (n | int / 1073741824) | round(1) }} GiB
|
||
{%- endmacro %}
|
||
{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
|
||
report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
|
||
its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
|
||
"object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
|
||
as `n/a` (the same token this template already uses for unreported readings) rather than
|
||
defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
|
||
collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
|
||
'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
|
||
{% macro thr(a) -%}
|
||
{{ a.thresh | default('n/a') }}
|
||
{%- endmacro %}
|
||
================================================================================
|
||
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
|
||
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
|
||
================================================================================
|
||
|
||
VERDICT {{ node_health_counts.FAIL }} FAIL · {{ node_health_counts.WARN }} WARN · {{ node_health_counts.PASS }} PASS · {{ node_health_counts.SKIP }} SKIP · {{ node_health_counts.INFO }} INFO
|
||
{{ 'privileged checks RAN' if node_health_root_available | bool else 'privileged checks SKIPPED — root not available' }}
|
||
|
||
--------------------------------------------------------------------------------
|
||
CHECKS
|
||
--------------------------------------------------------------------------------
|
||
{% for c in node_health_checks %}
|
||
[{{ '%-4s' | format(c.status) }}] {{ c.id }} — {{ c.title }}
|
||
{{ fmt(c.detail) }}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
MACHINE — {{ node_health_facts.dmi.sys_vendor | default('?') }} {{ node_health_facts.dmi.product_name | default('?') }}
|
||
--------------------------------------------------------------------------------
|
||
OS {{ node_health_facts.os.pretty_name | default('?') }}
|
||
Kernel {{ node_health_facts.os.kernel | default('?') }}
|
||
Uptime {{ (node_health_facts.os.uptime_seconds | float / 86400) | round(2) }} days
|
||
BIOS {{ node_health_facts.dmi.bios_vendor | default('?') }} {{ node_health_facts.dmi.bios_version | default('?') }} ({{ node_health_facts.dmi.bios_date | default('?') }}){% if node_health_facts.dmi.ec_firmware_release %}, EC {{ node_health_facts.dmi.ec_firmware_release }}{% endif %}
|
||
|
||
Board {{ node_health_facts.dmi.board_vendor | default('?') }} {{ node_health_facts.dmi.board_name | default('?') }}
|
||
CPU {{ node_health_facts.cpu.model | default('?') }}
|
||
{{ node_health_facts.cpu.cpus | default('?') }} logical / {{ node_health_facts.cpu.sockets | default('?') }} socket(s) x {{ node_health_facts.cpu.cores_per_socket | default('?') }} core(s) x {{ node_health_facts.cpu.threads_per_core | default('?') }} thread(s)
|
||
virtualisation: {{ node_health_facts.cpu.virtualization | default('none reported') }}, /dev/kvm={{ node_health_facts.cpu.kvm_device }}
|
||
{% if node_health_facts.cpu.scaling %}
|
||
scaling: {{ node_health_facts.cpu.scaling.driver }}/{{ node_health_facts.cpu.scaling.governor }}, {{ (node_health_facts.cpu.scaling.min_khz | int / 1000) | round | int }}–{{ (node_health_facts.cpu.scaling.max_khz | int / 1000) | round | int }} MHz (now {{ (node_health_facts.cpu.scaling.cur_khz | int / 1000) | round | int }} MHz)
|
||
{% endif %}
|
||
Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
|
||
ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
|
||
{% set mz = node_health_facts.memory.zones %}
|
||
{% set md = node_health_facts.memory.devices %}
|
||
Present RAM {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
|
||
DIMMs {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
|
||
{% for dimm in md.devices | default([]) if dimm.size_kb %}
|
||
{{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
|
||
{% endfor %}
|
||
{% set klog = node_health_facts.klog %}
|
||
EFI memattr {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
|
||
Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
|
||
|
||
--------------------------------------------------------------------------------
|
||
STORAGE
|
||
--------------------------------------------------------------------------------
|
||
{% for d in node_health_facts.storage.disks %}
|
||
{{ d.path }} {{ gib(d.size_bytes) }} {{ 'rotational' if d.rotational else 'non-rotating' }} {{ d.transport | default('?', true) }}{% if d.fstype %} whole-device {{ d.fstype }}{% endif %}{% if d.mountpoints %} mounted at {{ d.mountpoints | join(',') }}{% endif %}
|
||
|
||
model {{ d.model | default('(none reported)', true) }}{% if d.firmware_rev %} fw {{ d.firmware_rev }}{% endif %}{% if d.serial %} s/n {{ d.serial }}{% endif %}
|
||
|
||
{% if d.scsi_counters %}
|
||
kernel I/O completed={{ d.scsi_counters.iodone_cnt }} errors={{ d.scsi_counters.ioerr_cnt }} timeouts={{ d.scsi_counters.iotmo_cnt }}
|
||
{% endif %}
|
||
{% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %}
|
||
{% if sm and sm.available %}
|
||
SMART overall-health={{ 'PASSED' if sm.passed else 'FAILED' }} power_on_hours={{ sm.power_on_hours | default('?') }} power_cycles={{ sm.power_cycles | default('?') }}
|
||
lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }} temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
|
||
pending(197)={{ sm.pending_sectors | default('?') }} offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }} reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }} crc(199)={{ sm.crc_errors | default('?') }}
|
||
{% if sm.failing_now or sm.failed_past %}
|
||
THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
|
||
|
||
{% elif sm.thresholds_present %}
|
||
no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
|
||
|
||
{% endif %}
|
||
vendor context (NOT used for the verdict — raw units are vendor-defined):
|
||
reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }} grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }} erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }} program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }} unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
|
||
{% if sm.realloc_attr %}
|
||
attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
|
||
{% endif %}
|
||
{% for a in sm.attributes | default([]) %}
|
||
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %}
|
||
|
||
{% endfor %}
|
||
{% else %}
|
||
SMART NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}
|
||
{% endif %}
|
||
{% for p in d.partitions %}
|
||
{{ '%-12s' | format(p.kname) }} {{ '%9s' | format(gib(p.size_bytes)) }} {{ '%-6s' | format(p.fstype | default('-', true)) }} {{ (p.mountpoints[:2] | join(', ') ~ (' (+%d more)' | format(p.mountpoints | length - 2) if (p.mountpoints | length) > 2 else '')) if p.mountpoints else '(unmounted)' }}
|
||
{% endfor %}
|
||
|
||
{% endfor %}
|
||
{% if node_health_facts.storage.attached_volumes | default([]) %}
|
||
ATTACHED VOLUMES (iSCSI/FC — Longhorn replicas etc., not hardware in this box)
|
||
{% for a in node_health_facts.storage.attached_volumes %}
|
||
{{ '%-10s' | format(a.path) }} {{ '%9s' | format(gib(a.size_bytes)) }} {{ a.transport }} {{ a.model | default('-', true) }}
|
||
{% endfor %}
|
||
|
||
{% endif %}
|
||
MOUNTED FILESYSTEMS
|
||
{% for m in node_health_facts.storage.mounts | sort(attribute='mountpoint') %}
|
||
{{ '%-14s' | format(m.mountpoint) }} {{ '%-6s' | format(m.fstype) }} {{ '%9s' | format(gib(m.size_bytes)) }} total, {{ '%9s' | format(gib(m.avail_bytes)) }} free ({{ m.used_pct }}% used, inodes {{ (m.inodes_used_pct | string ~ '%') if m.inodes_used_pct is not none else 'n/a' }})
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
THERMAL
|
||
--------------------------------------------------------------------------------
|
||
{% for z in node_health_facts.thermal.zones %}
|
||
zone {{ '%-18s' | format(z.type | default(z.zone)) }} {{ (z.temp_c | string ~ 'C') if z.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if z.critical_c %} (critical {{ z.critical_c }}C){% endif %}
|
||
|
||
{% endfor %}
|
||
{% for chip in node_health_facts.thermal.hwmon %}
|
||
{% for t in chip.temps %}
|
||
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(t.label) }} {{ (t.temp_c | string ~ 'C') if t.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if t.crit_c %} (crit {{ t.crit_c }}C){% endif %}
|
||
|
||
{% endfor %}
|
||
{% for f in chip.fans %}
|
||
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(f.label) }} {{ f.rpm }} rpm
|
||
{% endfor %}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
|
||
--------------------------------------------------------------------------------
|
||
{% set gfx = node_health_facts.graphics %}
|
||
{% if gfx.pci_display %}
|
||
{% for p in gfx.pci_display %}
|
||
pci {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }}
|
||
driver {{ p.driver | default('NONE BOUND', true) }}
|
||
{% endfor %}
|
||
{% else %}
|
||
pci no display-class device (PCI base class 0x03) on this machine — no GPU here
|
||
{% endif %}
|
||
{% for c in gfx.drm_cards %}
|
||
drm {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }} engines {{ c.engines | join(' ') | default('none exposed', true) }}
|
||
{% endfor %}
|
||
{% if gfx.dri_present %}
|
||
{% for d in gfx.devices %}
|
||
dev {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }} {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }}
|
||
{% endfor %}
|
||
{% else %}
|
||
dev /dev/dri DOES NOT EXIST — no DRM device node on this node at all
|
||
{% endif %}
|
||
{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may
|
||
legitimately move it and Plex's container init joins whichever group owns the device. See
|
||
defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #}
|
||
{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %}
|
||
GuC/HuC {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
|
||
{{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }}
|
||
modules {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides)
|
||
vainfo {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }}
|
||
|
||
--------------------------------------------------------------------------------
|
||
PRESSURE STALL (/proc/pressure — % of wall time stalled)
|
||
--------------------------------------------------------------------------------
|
||
{% for res in ['cpu', 'memory', 'io'] %}
|
||
{% set p = node_health_facts.pressure[res] %}
|
||
{% if p %}
|
||
{{ '%-7s' | format(res) }} some avg10={{ p.some.avg10 }} avg60={{ p.some.avg60 }} avg300={{ p.some.avg300 }}{% if p.full is defined %} | full avg60={{ p.full.avg60 }}{% endif %}
|
||
|
||
{% else %}
|
||
{{ '%-7s' | format(res) }} unavailable
|
||
{% endif %}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
NETWORK
|
||
--------------------------------------------------------------------------------
|
||
{% for n in node_health_facts.network.interfaces %}
|
||
{{ '%-10s' | format(n.name) }} {{ '%-6s' | format(n.operstate) }} {{ '%-8s' | format((n.speed_mbps | string ~ 'Mb/s') if n.speed_mbps else 'no-link') }} {{ '%-9s' | format(n.duplex | default('-')) }} mtu {{ n.mtu }} driver {{ n.driver | default('-') }}
|
||
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
|
||
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
|
||
{% endfor %}
|
||
{% if node_health_facts.network.nameservers is none %}
|
||
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
|
||
{% else %}
|
||
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
|
||
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
|
||
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
|
||
{% endif %}
|
||
{% endif %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
NOT READ (and why) — these are UNKNOWNS, not passes
|
||
--------------------------------------------------------------------------------
|
||
{% for c in node_health_checks if c.status == 'SKIP' %}
|
||
- {{ c.id }}: {{ c.detail | regex_replace('\s+', ' ') | trim | wordwrap(70) | indent(3) }}
|
||
{% else %}
|
||
(nothing skipped)
|
||
{% endfor %}
|
||
{% if node_health_facts.unreadable %}
|
||
|
||
Unreadable paths reported by the collector:
|
||
{% for u in node_health_facts.unreadable %}
|
||
- {{ u.path }} — {{ u.reason }}
|
||
{% endfor %}
|
||
{% endif %}
|
||
|
||
tools present: {% for t, p in node_health_facts.tools.items() %}{{ t }}={{ p | default('MISSING', true) }}{{ ', ' if not loop.last }}{% endfor %}
|
||
|
||
================================================================================
|