jakman85andClaude Opus 5
a78f010368
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1280 lines
78 KiB
YAML
1280 lines
78 KiB
YAML
---
|
|
# Turn raw facts into explicit, named checks. Every check is one of:
|
|
#
|
|
# PASS measured, and within threshold
|
|
# WARN measured, outside the warn threshold — worth a human look
|
|
# FAIL measured, outside the fail threshold — do not put production state here
|
|
# SKIP NOT measured, with the reason attached (missing root, missing tool, sensor not
|
|
# present on this hardware). A SKIP is an honest "unknown" and is NEVER rendered
|
|
# as a pass — this repo's discipline is absolute on that point.
|
|
# INFO measured, no pass/fail semantics (inventory: firmware, uptime, CPU scaling)
|
|
#
|
|
# All thresholds come from node_health_thresholds (defaults/main.yml) so retuning never
|
|
# means editing logic here.
|
|
|
|
- name: Derive intermediate values used by several checks
|
|
ansible.builtin.set_fact:
|
|
# Every temperature the box exposes, from both thermal zones and hwmon chips.
|
|
# `select('number')` drops any sensor that returned null (unreadable/absent).
|
|
node_health_temps_c: >-
|
|
{{ ((node_health_facts.thermal.zones | map(attribute='temp_c') | list)
|
|
+ (node_health_facts.thermal.hwmon | map(attribute='temps') | flatten
|
|
| map(attribute='temp_c') | list)) | select('number') | list }}
|
|
# Filesystems, worst-first by percentage used / inodes used.
|
|
node_health_mounts_by_use: >-
|
|
{{ node_health_facts.storage.mounts | selectattr('used_pct', 'number')
|
|
| sort(attribute='used_pct') | reverse | list }}
|
|
node_health_mounts_by_inode: >-
|
|
{{ node_health_facts.storage.mounts | selectattr('inodes_used_pct', 'number')
|
|
| sort(attribute='inodes_used_pct') | reverse | list }}
|
|
# Physical NICs that are actually up. Virtual plumbing was already excluded by the
|
|
# collector; `virtual` here catches bonds/VLANs the regex let through.
|
|
node_health_nics_up: >-
|
|
{{ node_health_facts.network.interfaces
|
|
| selectattr('operstate', 'equalto', 'up') | list }}
|
|
# Devices that could actually BECOME a dedicated Longhorn or etcd volume: no
|
|
# partitions, no whole-device filesystem, nothing mounted from them. All three
|
|
# conditions matter — node_baseline formats the data disks with no partition table at
|
|
# all (mkfs straight onto /dev/vdb, mounted by LABEL), so "no partitions" alone would
|
|
# nominate a node's live Longhorn disk for reformatting. iSCSI/FC volumes were already
|
|
# split out by the collector for the same reason. On a single-disk node this list is
|
|
# empty — precisely the finding that blocks PROD-PROVISIONING.md §2.
|
|
node_health_free_devices: >-
|
|
{{ node_health_facts.storage.disks
|
|
| selectattr('partitions', 'equalto', [])
|
|
| selectattr('fstype', 'none')
|
|
| selectattr('mountpoints', 'equalto', [])
|
|
| list }}
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Assess — privilege escalation (gates every root-only check below)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'access.privilege',
|
|
'title': 'Privilege escalation (become) for root-only checks',
|
|
'status': ('INFO' if node_health_root_available | bool else 'SKIP'),
|
|
'detail': node_health_root_skip_reason }] }}"
|
|
|
|
# --- Storage ------------------------------------------------------------------------
|
|
# SMART VERDICTS — READ defaults/main.yml's SMART threshold block BEFORE EDITING THIS.
|
|
# Short version: this check judges a drive on the signals smartctl(8) itself defines as
|
|
# vendor-independent (the drive's own overall-health verdict; NORMALISED value vs the
|
|
# MANUFACTURER'S OWN threshold, via smartctl's `when_failed`; and the ATA-standard error
|
|
# counters 197/198/187/199 whose raw meaning is a plain event count everywhere, plus NVMe's
|
|
# structured health log). Vendor raw counts are reported with their numbers and can raise a
|
|
# WARN, but they can never produce a FAIL — because their units are vendor-defined and
|
|
# reading one as if it were a portable sector count is exactly the defect this replaces.
|
|
- name: Assess — SMART per physical disk
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'storage.smart.' ~ item.kname,
|
|
'title': 'SMART health — ' ~ item.path ~ ' (' ~ (item.model | default('unknown model')) ~ ')',
|
|
'status': smart_status,
|
|
'detail': smart_detail }] }}"
|
|
loop: "{{ node_health_facts.storage.disks }}"
|
|
loop_control:
|
|
label: "{{ item.path }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
disk_smart: "{{ (node_health_smart | selectattr('path', 'equalto', item.path) | list | first) | default({}) }}"
|
|
|
|
# --- Sub-verdicts. Each is SKIP unless the number behind it was actually read; a SKIP is
|
|
# an honest "unknown" and never contributes a PASS. ------------------------------------
|
|
|
|
# 1. The drive's own overall-health self-assessment. Vendor-independent by construction.
|
|
s_health: >-
|
|
{{ 'SKIP' if (disk_smart.passed is not defined or disk_smart.passed is none)
|
|
else ('PASS' if disk_smart.passed else 'FAIL') }}
|
|
# 2. NVMe critical warning bitfield (spare below threshold, media in read-only, etc.).
|
|
s_nvme_crit: >-
|
|
{{ 'SKIP' if (disk_smart.nvme_critical_warning is not defined or disk_smart.nvme_critical_warning is not number)
|
|
else ('PASS' if disk_smart.nvme_critical_warning | int == 0 else 'FAIL') }}
|
|
# 3. THE portable per-attribute failure signal: normalised value at or below the
|
|
# threshold the MANUFACTURER set for that attribute on that model. Pre-fail failing
|
|
# now = FAIL. Old_age failing now, or anything that failed in the past, = WARN.
|
|
s_thresh: >-
|
|
{{ 'SKIP' if not (disk_smart.thresholds_present | default(false))
|
|
else ('FAIL' if (disk_smart.failing_now_prefail | default([]) | length) > 0
|
|
else ('WARN' if ((disk_smart.failing_now | default([]) | length) > 0
|
|
or (disk_smart.failed_past | default([]) | length) > 0)
|
|
else 'PASS')) }}
|
|
# 4. ATA-standard hard-error counters. 197 pending; 198/187 + NVMe media_errors are
|
|
# uncorrectable reads — data the drive could NOT return. Same meaning on every drive.
|
|
s_pending: >-
|
|
{{ 'SKIP' if (disk_smart.pending_sectors is not defined or disk_smart.pending_sectors is not number)
|
|
else ('FAIL' if disk_smart.pending_sectors | int >= th.smart_pending_fail
|
|
else ('WARN' if disk_smart.pending_sectors | int >= th.smart_pending_warn else 'PASS')) }}
|
|
uncorr_values: >-
|
|
{{ [disk_smart.offline_uncorrectable | default(none),
|
|
disk_smart.reported_uncorrect | default(none),
|
|
disk_smart.nvme_media_errors | default(none)] | select('number') | list }}
|
|
s_uncorr: >-
|
|
{{ 'SKIP' if (uncorr_values | length) == 0
|
|
else ('FAIL' if (uncorr_values | max) >= th.smart_uncorrectable_fail
|
|
else ('WARN' if (uncorr_values | max) >= th.smart_uncorrectable_warn else 'PASS')) }}
|
|
# 5. 199 CRC — SATA link integrity (cable/backplane), not media. WARN only: the counter
|
|
# never resets, so one historic bad seating must not condemn the node forever.
|
|
s_crc: >-
|
|
{{ 'SKIP' if (disk_smart.crc_errors is not defined or disk_smart.crc_errors is not number)
|
|
else ('WARN' if disk_smart.crc_errors | int >= th.smart_crc_warn else 'PASS') }}
|
|
# 6. Wear, from the NORMALISED lifetime indicator — a percentage by definition.
|
|
life_rem: "{{ disk_smart.lifetime_remaining_pct | default(none) }}"
|
|
life_used: "{{ (100 - (life_rem | int)) if (life_rem is number) else none }}"
|
|
s_life: >-
|
|
{{ 'SKIP' if (life_rem is not number)
|
|
else ('FAIL' if life_rem | int <= th.smart_lifetime_remaining_fail_pct
|
|
else ('WARN' if (life_rem | int <= th.smart_lifetime_remaining_warn_pct
|
|
or life_used | int >= th.smart_life_used_pct_warn)
|
|
else 'PASS')) }}
|
|
# 7. Reallocation ACTIVITY. Portable evidence first (196 event count, and the grown
|
|
# bad-block counter smartctl's drivedb named for this model). Only if neither exists
|
|
# do we fall back to attribute 5's raw — and then ONLY to say "nonzero, so something
|
|
# was remapped", which is portable, never to interpret its magnitude, which is not.
|
|
remap_evidence: >-
|
|
{{ [disk_smart.realloc_events | default(none),
|
|
disk_smart.grown_bad_blocks | default(none)] | select('number') | list }}
|
|
realloc_raw: "{{ (disk_smart.realloc_attr | default({}, true)).get('raw', {}).get('value', none) }}"
|
|
s_remap: >-
|
|
{{ ('WARN' if (remap_evidence | max) >= th.smart_realloc_events_warn else 'PASS')
|
|
if (remap_evidence | length) > 0
|
|
else (('WARN' if realloc_raw | int > 0 else 'PASS') if realloc_raw is number else 'SKIP') }}
|
|
# 8. Endurance stress — erase/program failures are NAND wearing out under the FTL.
|
|
s_erase: >-
|
|
{{ 'SKIP' if (disk_smart.erase_fail_count is not defined or disk_smart.erase_fail_count is not number)
|
|
else ('WARN' if disk_smart.erase_fail_count | int >= th.smart_erase_fail_warn else 'PASS') }}
|
|
s_program: >-
|
|
{{ 'SKIP' if (disk_smart.program_fail_count is not defined or disk_smart.program_fail_count is not number)
|
|
else ('WARN' if disk_smart.program_fail_count | int >= th.smart_program_fail_warn else 'PASS') }}
|
|
# 9. Unexpected power loss. On a consumer SSD with no power-loss protection each one is
|
|
# an opportunity for a torn or lost in-flight write — which is precisely the guarantee
|
|
# etcd's fsync contract depends on. Directly relevant to PROD-PROVISIONING.md Section 2.
|
|
s_ploss: >-
|
|
{{ 'SKIP' if (disk_smart.unexpected_power_loss is not defined or disk_smart.unexpected_power_loss is not number)
|
|
else ('WARN' if disk_smart.unexpected_power_loss | int >= th.smart_unexpected_power_loss_warn else 'PASS') }}
|
|
s_temp: >-
|
|
{{ 'SKIP' if (disk_smart.temperature_c is not defined or disk_smart.temperature_c is not number)
|
|
else ('FAIL' if disk_smart.temperature_c | int >= th.smart_temp_fail_c
|
|
else ('WARN' if disk_smart.temperature_c | int >= th.smart_temp_warn_c else 'PASS')) }}
|
|
# Worst sub-verdict wins. `select('in', ...)` preserves the order of the left-hand
|
|
# list, so the LAST survivor is the most severe present.
|
|
smart_status: >-
|
|
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_nvme_crit, s_thresh, s_pending,
|
|
s_uncorr, s_crc, s_life, s_remap, s_erase, s_program, s_ploss, s_temp]) | list | last)
|
|
| default('SKIP', true) }}
|
|
|
|
# --- Detail. Every WARN/FAIL must arrive WITH the number that caused it; a verdict the
|
|
# operator cannot check is not much better than a wrong one. -------------------------
|
|
smart_findings: >-
|
|
{%- set n = namespace(l=[]) -%}
|
|
{%- if s_health == 'FAIL' -%}
|
|
{%- set _ = n.l.append('the drive reports its OWN overall-health self-assessment as FAILED') -%}
|
|
{%- endif -%}
|
|
{%- if s_nvme_crit == 'FAIL' -%}
|
|
{%- set _ = n.l.append('NVMe critical_warning=' ~ disk_smart.nvme_critical_warning ~ ' (non-zero)') -%}
|
|
{%- endif -%}
|
|
{%- for a in (disk_smart.failing_now_prefail | default([])) -%}
|
|
{%- set _ = n.l.append('Pre-fail attribute ' ~ a.id ~ ' ' ~ a.name ~ ' is AT OR BELOW its manufacturer threshold NOW (value='
|
|
~ a.value ~ ', thresh=' ~ a.thresh ~ ')') -%}
|
|
{%- endfor -%}
|
|
{%- for a in (disk_smart.failed_past | default([])) -%}
|
|
{%- set _ = n.l.append('attribute ' ~ a.id ~ ' ' ~ a.name ~ ' crossed its manufacturer threshold IN THE PAST (worst='
|
|
~ a.worst ~ ', thresh=' ~ a.thresh ~ ')') -%}
|
|
{%- endfor -%}
|
|
{%- if s_pending in ['WARN', 'FAIL'] -%}
|
|
{%- set _ = n.l.append('current pending sectors (197)=' ~ disk_smart.pending_sectors) -%}
|
|
{%- endif -%}
|
|
{%- if s_uncorr in ['WARN', 'FAIL'] -%}
|
|
{%- set _ = n.l.append('UNCORRECTABLE read errors — offline(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
|
|
~ ', reported(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
|
|
~ ', nvme media_errors=' ~ (disk_smart.nvme_media_errors if disk_smart.nvme_media_errors is number else 'n/a')) -%}
|
|
{%- endif -%}
|
|
{%- if s_crc == 'WARN' -%}
|
|
{%- set _ = n.l.append('SATA link CRC errors (199)=' ~ disk_smart.crc_errors ~ ' — cable/backplane, not media') -%}
|
|
{%- endif -%}
|
|
{%- if s_life in ['WARN', 'FAIL'] -%}
|
|
{%- set _ = n.l.append(life_used ~ '% of rated life CONSUMED (' ~ life_rem
|
|
~ '% remaining, from the normalised wear indicator)') -%}
|
|
{%- endif -%}
|
|
{%- if s_remap == 'WARN' and (remap_evidence | length) > 0 -%}
|
|
{%- set _ = n.l.append((remap_evidence | max) ~ ' grown bad block(s)/reallocation event(s)'
|
|
~ (' [196 events=' ~ disk_smart.realloc_events ~ ']' if disk_smart.realloc_events is number else '')
|
|
~ (' [grown failing blocks=' ~ disk_smart.grown_bad_blocks ~ ']' if disk_smart.grown_bad_blocks is number else '')) -%}
|
|
{%- elif s_remap == 'WARN' -%}
|
|
{%- set _ = n.l.append('the drive reports a non-zero reallocation attribute (5 raw=' ~ realloc_raw
|
|
~ '); its UNIT is vendor-defined, so the count is not interpreted here') -%}
|
|
{%- endif -%}
|
|
{%- if s_erase == 'WARN' -%}
|
|
{%- set _ = n.l.append('NAND erase failures=' ~ disk_smart.erase_fail_count) -%}
|
|
{%- endif -%}
|
|
{%- if s_program == 'WARN' -%}
|
|
{%- set _ = n.l.append('NAND program failures=' ~ disk_smart.program_fail_count) -%}
|
|
{%- endif -%}
|
|
{%- if s_ploss == 'WARN' -%}
|
|
{%- set _ = n.l.append('unexpected power losses=' ~ disk_smart.unexpected_power_loss
|
|
~ ' — a consumer SSD has no power-loss protection, so each one risks a torn or lost in-flight write') -%}
|
|
{%- endif -%}
|
|
{%- if s_temp in ['WARN', 'FAIL'] -%}
|
|
{%- set _ = n.l.append('temperature=' ~ disk_smart.temperature_c ~ 'C') -%}
|
|
{%- endif -%}
|
|
{{ n.l | join('; ') }}
|
|
# The raw value that started this. Shown ALWAYS when it is non-zero and contradicts the
|
|
# normalised reading, explicitly labelled as vendor-encoded and explicitly excluded from
|
|
# the verdict — so the next operator to see 61440 sees why it is not 61440 bad sectors.
|
|
realloc_raw_note: >-
|
|
{%- if realloc_raw is number and realloc_raw | int > 0 -%}
|
|
{%- set a = disk_smart.realloc_attr -%}
|
|
attribute 5 ({{ a.name }}) raw={{ realloc_raw }} normalised value={{ a.value }} vs thresh={{ a.thresh }}
|
|
{%- if (remap_evidence | length) > 0 and (remap_evidence | max) > 0
|
|
and realloc_raw | int != (remap_evidence | max) | int
|
|
and realloc_raw | int % (remap_evidence | max) | int == 0 -%}
|
|
{{ ' (= ' ~ (remap_evidence | max) ~ ' x ' ~ (realloc_raw | int // (remap_evidence | max) | int) ~ ')' }}
|
|
{%- endif -%}
|
|
. SMART raw values are VENDOR-DEFINED (smartctl(8): "entirely vendor-specific") and this
|
|
one is NOT a portable bad-sector count — it is reported as context and is NOT used for
|
|
the verdict. The normalised value/threshold pair beside it is the manufacturer's own
|
|
trip point and IS.
|
|
{%- endif -%}
|
|
smart_detail: >-
|
|
{{ ('not read — ' ~ node_health_root_skip_reason)
|
|
if not (node_health_root_available | bool)
|
|
else ('not read — smartmontools is not installed on this node; it ships with
|
|
roles/node_baseline (node_baseline_packages), which has not been run here.
|
|
This role never installs packages.')
|
|
if not (node_health_facts.tools.smartctl is truthy)
|
|
else ('smartctl ran but this device exposes no SMART data ('
|
|
~ (disk_smart.messages | default([]) | join('; ')) ~ ') — normal for virtio/VM disks')
|
|
if not (disk_smart.available | default(false))
|
|
else ('overall-health=' ~ ('PASSED' if disk_smart.passed else 'FAILED')
|
|
~ ', power_on_hours=' ~ (disk_smart.power_on_hours | default('?'))
|
|
~ ', power_cycles=' ~ (disk_smart.power_cycles | default('?'))
|
|
~ ', life_remaining=' ~ ((life_rem | string ~ '%') if life_rem is number else 'not reported')
|
|
~ ', pending(197)=' ~ (disk_smart.pending_sectors if disk_smart.pending_sectors is number else 'n/a')
|
|
~ ', offline_uncorrectable(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
|
|
~ ', reported_uncorrect(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
|
|
~ ', crc(199)=' ~ (disk_smart.crc_errors if disk_smart.crc_errors is number else 'n/a')
|
|
~ (('. NO attribute is at or below its manufacturer threshold; closest is '
|
|
~ disk_smart.prefail_closest.id ~ ' ' ~ disk_smart.prefail_closest.name
|
|
~ ' value=' ~ disk_smart.prefail_closest.value
|
|
~ ' vs thresh=' ~ disk_smart.prefail_closest.thresh)
|
|
if (s_thresh == 'PASS' and (disk_smart.prefail_closest | default(none)) is mapping) else '')
|
|
~ (('. FINDINGS: ' ~ smart_findings) if smart_findings | trim else '')
|
|
~ (('. CONTEXT: ' ~ realloc_raw_note) if realloc_raw_note | trim else '')) }}
|
|
|
|
- name: Assess — kernel-visible I/O errors per disk (no root needed)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'storage.io_errors',
|
|
'title': 'Kernel I/O error counters on physical disks (SCSI/ATA layer)',
|
|
'status': io_status,
|
|
'detail': io_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
# ioerr_cnt/iotmo_cnt are readable WITHOUT root. They are not a substitute for SMART
|
|
# (no wear, no power-on hours, no reallocated sectors) but they are a real signal: a
|
|
# non-zero count means the kernel actually saw failed or timed-out commands on that
|
|
# device since boot.
|
|
counted: "{{ node_health_facts.storage.disks | selectattr('scsi_counters', 'mapping') | list }}"
|
|
err_total: >-
|
|
{{ (counted | map(attribute='scsi_counters.ioerr_cnt') | select('number') | list | sum)
|
|
+ (counted | map(attribute='scsi_counters.iotmo_cnt') | select('number') | list | sum) }}
|
|
io_status: >-
|
|
{{ 'SKIP' if (counted | length) == 0
|
|
else ('PASS' if err_total | int == 0 else 'WARN') }}
|
|
io_sum_done: "{{ counted | map(attribute='scsi_counters.iodone_cnt') | select('number') | list | sum }}"
|
|
io_sum_err: "{{ counted | map(attribute='scsi_counters.ioerr_cnt') | select('number') | list | sum }}"
|
|
io_sum_tmo: "{{ counted | map(attribute='scsi_counters.iotmo_cnt') | select('number') | list | sum }}"
|
|
# Aggregate here; the STORAGE section of the summary prints the per-disk counters.
|
|
io_detail: >-
|
|
{{ 'no disk exposes SCSI/ATA error counters (virtio or NVMe-only node)'
|
|
if (counted | length) == 0
|
|
else ((counted | map(attribute='kname') | join(', ')) ~ ': completed='
|
|
~ io_sum_done | string ~ ', errors=' ~ io_sum_err | string
|
|
~ ', timeouts=' ~ io_sum_tmo | string
|
|
~ ('. Non-zero errors are commands the kernel saw fail on that device
|
|
since boot. They are NOT proof of media failure (a rejected or
|
|
unsupported command during discovery counts too) and they cannot be
|
|
attributed without dmesg, which needs root.'
|
|
if err_total | int > 0 else '')) }}
|
|
|
|
- name: Assess — filesystem free space
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'storage.capacity',
|
|
'title': 'Filesystem free space (worst mount)',
|
|
'status': fs_status,
|
|
'detail': fs_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
worst: "{{ (node_health_mounts_by_use | first) | default({}) }}"
|
|
fs_status: >-
|
|
{{ 'SKIP' if (node_health_mounts_by_use | length) == 0
|
|
else ('FAIL' if worst.used_pct >= th.fs_used_pct_fail
|
|
else ('WARN' if worst.used_pct >= th.fs_used_pct_warn else 'PASS')) }}
|
|
fs_detail: >-
|
|
{{ 'no block-backed filesystem found' if (node_health_mounts_by_use | length) == 0
|
|
else (worst.mountpoint ~ ' ' ~ worst.used_pct | string ~ '% used ('
|
|
~ (worst.avail_bytes | int / 1073741824) | round(1) | string ~ ' GiB free of '
|
|
~ (worst.size_bytes | int / 1073741824) | round(1) | string ~ ' GiB)') }}
|
|
|
|
- name: Assess — filesystem inode headroom
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'storage.inodes',
|
|
'title': 'Filesystem inode headroom (worst mount)',
|
|
'status': in_status,
|
|
'detail': in_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
worst: "{{ (node_health_mounts_by_inode | first) | default({}) }}"
|
|
in_status: >-
|
|
{{ 'SKIP' if (node_health_mounts_by_inode | length) == 0
|
|
else ('WARN' if worst.inodes_used_pct >= th.inode_used_pct_warn else 'PASS') }}
|
|
in_detail: >-
|
|
{{ 'no filesystem reported inode usage' if (node_health_mounts_by_inode | length) == 0
|
|
else (worst.mountpoint ~ ' ' ~ worst.inodes_used_pct | string ~ '% of inodes used') }}
|
|
|
|
- name: Assess — data-disk layout (feeds PROD-PROVISIONING.md §2)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'storage.layout',
|
|
'title': 'Physical disks and spare block devices for Longhorn/etcd',
|
|
'status': 'INFO',
|
|
'detail': layout_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
disks: "{{ node_health_facts.storage.disks }}"
|
|
attached: "{{ node_health_facts.storage.attached_volumes | default([]) }}"
|
|
layout_detail: >-
|
|
{{ (disks | length) | string ~ ' physical disk(s): '
|
|
~ (disks | map(attribute='kname') | join(', ')) ~ '. '
|
|
~ ((attached | length) | string ~ ' attached iSCSI/FC volume(s) excluded ('
|
|
~ (attached | map(attribute='kname') | join(', ')) ~ '). '
|
|
if (attached | length) > 0 else '')
|
|
~ ((node_health_free_devices | length) | string ~ ' device(s) with no partitions ('
|
|
~ (node_health_free_devices | map(attribute='kname') | join(', ')) ~ ')'
|
|
if (node_health_free_devices | length) > 0
|
|
else 'NO unpartitioned spare device — node_baseline_longhorn_device /
|
|
node_baseline_etcd_device have nothing to claim without repartitioning
|
|
or adding a drive') }}
|
|
|
|
# --- Memory -------------------------------------------------------------------------
|
|
- name: Assess — memory headroom
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'memory.available',
|
|
'title': 'Available memory',
|
|
'status': mem_status,
|
|
'detail': mem_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
m: "{{ node_health_facts.memory }}"
|
|
avail_pct: >-
|
|
{{ ((m.mem_available_kb | int) * 100.0 / (m.mem_total_kb | int)) | round(1)
|
|
if (m.mem_total_kb is number and m.mem_available_kb is number and m.mem_total_kb | int > 0)
|
|
else none }}
|
|
mem_status: >-
|
|
{{ 'SKIP' if avail_pct is not number
|
|
else ('FAIL' if avail_pct | float <= th.mem_available_pct_fail
|
|
else ('WARN' if avail_pct | float <= th.mem_available_pct_warn else 'PASS')) }}
|
|
mem_detail: >-
|
|
{{ '/proc/meminfo unreadable' if avail_pct is not number
|
|
else ((m.mem_total_kb | int / 1048576) | round(1) | string ~ ' GiB total, '
|
|
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
|
|
~ avail_pct | string ~ '%)') }}
|
|
|
|
# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
|
|
# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
|
|
# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
|
|
# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
|
|
# from the presence of a kernel-log line — and they duly contradicted each other on the same
|
|
# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
|
|
# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
|
|
# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
|
|
# measurement was right; the inference from the log line was wrong — the memattr block below
|
|
# has the refined mechanism.
|
|
#
|
|
# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
|
|
# place and both checks read this fact: neither derives a shortfall of its own, and the
|
|
# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
|
|
# disagree about whether memory was lost — there is only one verdict on that question, and
|
|
# it is this one. Thresholds and their measured baselines: defaults/main.yml.
|
|
- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
|
|
ansible.builtin.set_fact:
|
|
node_health_mem_shortfall: "{{ {
|
|
'zones_pct': z_pct, 'zones_status': z_status,
|
|
'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
|
|
'status': worst_status } }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
z: "{{ node_health_facts.memory.zones }}"
|
|
dev: "{{ node_health_facts.memory.devices }}"
|
|
m: "{{ node_health_facts.memory }}"
|
|
|
|
# 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
|
|
# actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
|
|
# tools, identical on metal and in a VM — so this half is available almost always.
|
|
z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
|
|
z_status: >-
|
|
{{ 'SKIP' if z_pct is not number
|
|
else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
|
|
else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
|
|
|
|
# 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
|
|
# VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
|
|
# never as a zero DIMM count. Independent of the kernel's memory map entirely.
|
|
inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
|
|
d_pct: >-
|
|
{{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
|
|
if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
|
|
else none }}
|
|
d_status: >-
|
|
{{ 'SKIP' if d_pct is not number
|
|
else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
|
|
else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
|
|
|
|
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
|
|
# SKIPs must never add up to a PASS.
|
|
worst_status: >-
|
|
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
|
|
| default('SKIP', true) }}
|
|
|
|
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
|
|
# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
|
|
# blunter question — "is this machine using the RAM it has?" — and answers it from two
|
|
# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
|
|
# losing memory for some other reason still shows up.
|
|
- name: Assess — usable memory vs the memory this machine physically has
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'memory.physical_total',
|
|
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
|
'status': phys_status,
|
|
'detail': phys_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
ms: "{{ node_health_mem_shortfall }}"
|
|
z: "{{ node_health_facts.memory.zones }}"
|
|
dev: "{{ node_health_facts.memory.devices }}"
|
|
m: "{{ node_health_facts.memory }}"
|
|
# Straight from the shared measurement — this check re-derives nothing.
|
|
z_pct: "{{ ms.zones_pct }}"
|
|
inst_kb: "{{ ms.installed_kb }}"
|
|
d_pct: "{{ ms.dimm_pct }}"
|
|
phys_status: "{{ ms.status }}"
|
|
# What the kernel log says, IF it was read — quoted here only as attribution for the
|
|
# measured shortfall. The verdict above is the shared measurement and never depends on
|
|
# the log; firmware.efi_memattr reads that same verdict, so the two always agree.
|
|
phys_cause: >-
|
|
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
|
|
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
|
|
if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
|
|
else ('. The kernel log was NOT read here, so this shortfall is unattributed —
|
|
see check firmware.efi_memattr')
|
|
if not (node_health_facts.klog.available | bool)
|
|
else '' }}
|
|
phys_detail: >-
|
|
{{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
|
|
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
|
|
~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
|
|
was never handed to the allocator (/proc/zoneinfo)')
|
|
if z_pct is number
|
|
else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
|
|
~ '. '
|
|
~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
|
|
~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
|
|
~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
|
|
~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
|
|
~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
|
|
if d_pct is number
|
|
else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
|
|
~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
|
|
reservations and (on this platform) 64 MiB of graphics stolen memory.
|
|
A shortfall of this size is memory the machine HAS and cannot use')
|
|
if phys_status in ['WARN', 'FAIL'] else '')
|
|
~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
|
|
|
|
# WHY THIS CHECK IS NOT JUST `SwapTotal == 0`. It used to be, and it read the answer
|
|
# wrong in both directions. It called active swap "this node has not been baselined yet",
|
|
# which was a MIS-DIAGNOSIS — measured 2026-09-07, all three prod nodes had been
|
|
# baselined, had their fstab swap line commented out by node_baseline, and were STILL
|
|
# swapping 977 MiB each after a reboot. Whoever read that WARN would have re-run the very
|
|
# role that had already run, and watched swap come back at the next boot again.
|
|
#
|
|
# The real mechanism is systemd-gpt-auto-generator: it claims any GPT partition typed
|
|
# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, builds a
|
|
# .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It
|
|
# never reads /etc/fstab, so `swapoff -a` + a commented fstab line only holds until the
|
|
# next boot. roles/node_baseline now masks swap.target (the single unit those generated
|
|
# units are `WantedBy`) plus the generated units themselves.
|
|
#
|
|
# The other direction matters just as much: SwapTotal == 0 is NOT proof that swap is
|
|
# disabled — a node freshly `swapoff`-ed reads zero and swaps again on reboot. So this
|
|
# check now asks both questions, and WARNs on a node that is clean right now but has an
|
|
# unmasked generated swap unit waiting for the next boot.
|
|
- name: Assess — swap (kubelet requires it off, and off ACROSS REBOOTS)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'memory.swap',
|
|
'title': 'Swap disabled, and disabled across reboots (kubelet prerequisite)',
|
|
'status': swap_status,
|
|
'detail': swap_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
m: "{{ node_health_facts.memory }}"
|
|
p: "{{ node_health_facts.memory.swap_persistence | default({}, true) }}"
|
|
# Durability was actually measurable on this node (the collector could read
|
|
# /etc/systemd/system and the generator dirs). When false, say so rather than
|
|
# implying a clean bill from a failed read.
|
|
swap_durable_known: "{{ (p.readable | default(false)) | bool }}"
|
|
# Units a reboot would bring back: produced by a generator, not masked, and with
|
|
# swap.target itself unmasked so something can still pull them in.
|
|
swap_will_return: >-
|
|
{{ (swap_durable_known | bool)
|
|
and not (p.swap_target_masked | default(false) | bool)
|
|
and (p.unmasked_generated_units | default([], true) | length) > 0 }}
|
|
swap_units_returning: "{{ p.unmasked_generated_units | default([], true) | join(', ') }}"
|
|
swap_active_gib: "{{ (m.swap_total_kb | int / 1048576) | round(2) | string }}"
|
|
swap_status: >-
|
|
{{ 'SKIP' if m.swap_total_kb is not number
|
|
else ('WARN' if (m.swap_total_kb | int) > 0
|
|
else ('WARN' if (swap_will_return | bool) else 'PASS')) }}
|
|
swap_cause: >-
|
|
systemd-gpt-auto-generator re-creates a .swap unit for the GPT partition typed
|
|
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") at EVERY boot, straight from
|
|
the partition table and without reading /etc/fstab. This is NOT "the node has not
|
|
been baselined" — `swapoff -a` and a commented fstab line simply do not survive a
|
|
reboot. The fix is roles/node_baseline masking swap.target and the generated units:
|
|
re-run `node-baseline.yml --tags system` against this node.
|
|
swap_detail: >-
|
|
{{ 'swap state unreadable' if m.swap_total_kb is not number
|
|
else ((swap_active_gib ~ ' GiB swap active'
|
|
~ (' via ' ~ swap_units_returning if swap_units_returning else '')
|
|
~ ' — ' ~ swap_cause)
|
|
if (m.swap_total_kb | int) > 0
|
|
else ('no swap active, but it would come back at the next boot via '
|
|
~ swap_units_returning ~ ' — ' ~ swap_cause
|
|
if (swap_will_return | bool)
|
|
else ('no swap active'
|
|
~ (' and none can be activated: swap.target is masked'
|
|
if (p.swap_target_masked | default(false) | bool)
|
|
else (' and no swap unit exists on this node'
|
|
if (p.generated_units | default([], true) | length) == 0
|
|
else ' and every generated swap unit is masked'))
|
|
if (swap_durable_known | bool)
|
|
else 'no swap active (durability NOT verified — the collector
|
|
could not read /etc/systemd/system or /run/systemd)'))) }}
|
|
|
|
- name: Assess — ECC / EDAC correctable errors
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'memory.ecc',
|
|
'title': 'ECC memory errors (EDAC correctable/uncorrectable counters)',
|
|
'status': ecc_status,
|
|
'detail': ecc_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
e: "{{ node_health_facts.memory.edac }}"
|
|
ecc_status: >-
|
|
{{ 'SKIP' if not (e.controllers_present | bool)
|
|
else ('FAIL' if (e.ue_total | int) > 0
|
|
else ('WARN' if (e.ce_total | int) > 0 else 'PASS')) }}
|
|
ecc_detail: >-
|
|
{{ ('no EDAC memory controller is bound (' ~ ('/sys/devices/system/edac/mc exists but is empty'
|
|
if e.sysfs_present | bool else 'no EDAC sysfs at all')
|
|
~ ') — this platform does not report ECC status. Non-ECC consumer RAM or a VM;
|
|
"no errors" CANNOT be confirmed on this hardware.')
|
|
if not (e.controllers_present | bool)
|
|
else ((e.controllers | length) | string ~ ' controller(s), correctable='
|
|
~ e.ce_total | string ~ ', uncorrectable=' ~ e.ue_total | string) }}
|
|
|
|
# --- CPU ----------------------------------------------------------------------------
|
|
- name: Assess — CPU thermal throttling
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'cpu.throttle',
|
|
'title': 'CPU thermal throttling since boot',
|
|
'status': thr_status,
|
|
'detail': thr_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
t: "{{ node_health_facts.cpu.throttle }}"
|
|
thr_status: >-
|
|
{{ 'SKIP' if t is none
|
|
else ('PASS' if ((t.core_throttle_count_total | int) + (t.package_throttle_count | int)) == 0
|
|
else 'WARN') }}
|
|
thr_detail: >-
|
|
{{ 'no thermal_throttle counters (VM guest or unsupported CPU)' if t is none
|
|
else ('core throttle events=' ~ t.core_throttle_count_total | string
|
|
~ ' (' ~ t.core_throttle_total_time_ms | string ~ ' ms), package events='
|
|
~ t.package_throttle_count | string) }}
|
|
|
|
- name: Assess — CPU inventory and frequency scaling
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'cpu.inventory',
|
|
'title': 'CPU model, topology and frequency scaling',
|
|
'status': 'INFO',
|
|
'detail': cpu_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
c: "{{ node_health_facts.cpu }}"
|
|
cpu_detail: >-
|
|
{{ (c.model | default('unknown')) ~ ' — ' ~ (c.cpus | default('?')) ~ ' logical CPU(s), '
|
|
~ (c.sockets | default('?')) ~ ' socket(s) x ' ~ (c.cores_per_socket | default('?'))
|
|
~ ' core(s) x ' ~ (c.threads_per_core | default('?')) ~ ' thread(s); load '
|
|
~ (c.loadavg | default([]) | join(' ')) ~ '; '
|
|
~ ('governor=' ~ c.scaling.governor ~ ' driver=' ~ c.scaling.driver
|
|
~ ' cur=' ~ ((c.scaling.cur_khz | int) / 1000) | round | int | string ~ 'MHz'
|
|
~ ' max=' ~ ((c.scaling.max_khz | int) / 1000) | round | int | string ~ 'MHz'
|
|
if c.scaling is mapping else 'no cpufreq scaling exposed') }}
|
|
|
|
- name: Assess — hardware virtualisation (KubeVirt prerequisite, R-PLAT-03)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'cpu.virtualisation',
|
|
'title': 'Hardware virtualisation available (VT-x/AMD-V + /dev/kvm)',
|
|
'status': virt_status,
|
|
'detail': virt_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
c: "{{ node_health_facts.cpu }}"
|
|
virt_status: >-
|
|
{{ 'PASS' if (c.hardware_virt_flag | bool and c.kvm_device | bool) else 'WARN' }}
|
|
virt_detail: >-
|
|
{{ 'vmx/svm flag=' ~ c.hardware_virt_flag | string ~ ', /dev/kvm='
|
|
~ c.kvm_device | string
|
|
~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
|
|
~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
|
|
|
|
# --- Graphics / render device ----------------------------------------------------------
|
|
# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
|
|
# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
|
|
# to software transcoding on its own and the only symptom is a hot CPU and stuttering
|
|
# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
|
|
# binds a different DRM driver all produce exactly that silent degradation. It was measured
|
|
# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
|
|
# decays silently, so it is re-measured on every health run instead.
|
|
#
|
|
# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
|
|
# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
|
|
# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
|
|
# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
|
|
# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
|
|
# and it reports as an honest UNKNOWN with the reason rather than as a 0.
|
|
#
|
|
# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
|
|
# graphics.render_device the OUTCOME — can anything hardware-transcode on this node.
|
|
# graphics.drm_driver the CAUSE side — what is bound, and is it what this site expects.
|
|
# graphics.media_stack INFO only. GuC/HuC and host VAAPI userspace both belong here and
|
|
# NEITHER is ever a failure; see the task's own comment for why.
|
|
- name: Assess — locate the display device and its bound driver (once, shared by three checks)
|
|
ansible.builtin.set_fact:
|
|
node_health_render:
|
|
# Display-class PCI devices (base class 0x03), driver bound or not. This is the
|
|
# question "is there a GPU in this machine at all", and it is answered from the PCI
|
|
# bus rather than from /sys/class/drm because a card* entry only exists once a driver
|
|
# has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
|
|
# identical to a machine with no GPU.
|
|
display_devices: "{{ g.pci_display }}"
|
|
bound_drivers: >-
|
|
{{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
|
|
# Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
|
|
# is an anomaly, not a device, and must not be counted as one.
|
|
render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
|
|
# Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
|
|
# difference between "a GPU is bound" and "this GPU can do media work".
|
|
media_engines: >-
|
|
{{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
|
|
vars:
|
|
ansible_become: false
|
|
g: "{{ node_health_facts.graphics }}"
|
|
|
|
- name: Assess — DRM render node for hardware transcoding
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'graphics.render_device',
|
|
'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
|
|
'status': render_status,
|
|
'detail': render_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
g: "{{ node_health_facts.graphics }}"
|
|
r: "{{ node_health_render }}"
|
|
# SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
|
|
# then nothing to grade, and the same convention thermal.max already uses for "no sensor
|
|
# on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
|
|
# a GPU IS on the bus and there is still no render node.
|
|
render_status: >-
|
|
{{ 'SKIP' if (r.display_devices | length) == 0
|
|
else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
|
|
dri_listing: >-
|
|
{{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
|
|
else 'nothing' }}
|
|
render_detail: >-
|
|
{%- if (r.display_devices | length) == 0 -%}
|
|
this machine exposes no display-class PCI device (base class 0x03) at all, so there is
|
|
no GPU here for a render node to belong to and hardware transcoding is not available on
|
|
this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
|
|
not assumed.
|
|
{%- elif (r.render_chardevs | length) > 0 -%}
|
|
{{ r.render_chardevs | length }} render node(s):
|
|
{% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
|
|
owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
|
|
gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
|
|
/dev/dri holds: {{ dri_listing }}.
|
|
THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
|
|
`render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
|
|
node may legitimately land on a different number (all three prod nodes read gid 992 on
|
|
2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
|
|
owns it. Drift is therefore visible in this line without being a failure.
|
|
{%- else -%}
|
|
HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
|
|
PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
|
|
no usable character-special render node: /dev/dri
|
|
{{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
|
|
{{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
|
|
~ ' — present, but NOT character special, which is an anomaly in its own right. ')
|
|
if (g.render_nodes | length) > 0 else '' }}
|
|
Nothing fails loudly in this state — Plex silently falls back to software transcoding
|
|
and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
|
|
driver bound at all, then whether the onboard IGD is still enabled in BIOS.
|
|
{%- endif -%}
|
|
|
|
- name: Assess — DRM driver bound to the display device
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'graphics.drm_driver',
|
|
'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
|
|
'status': driver_status,
|
|
'detail': driver_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
g: "{{ node_health_facts.graphics }}"
|
|
r: "{{ node_health_render }}"
|
|
expected: "{{ node_health_graphics_expected_driver }}"
|
|
# WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
|
|
# still works, but the userspace media stack is not the same one, so it is a change the
|
|
# operator must see before a transcode regression surfaces as a user complaint. FAIL is
|
|
# for nothing bound at all, which is the cause of a missing render node.
|
|
driver_status: >-
|
|
{{ 'SKIP' if (r.display_devices | length) == 0
|
|
else ('FAIL' if (r.bound_drivers | length) == 0
|
|
else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
|
|
driver_detail: >-
|
|
{%- if (r.display_devices | length) == 0 -%}
|
|
no display-class PCI device on this machine, so no DRM driver can be bound to one.
|
|
Measured from /sys/bus/pci/devices.
|
|
{%- else -%}
|
|
{% for p in r.display_devices %}{{ p.address }}
|
|
{{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
|
|
-> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
|
|
{{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
|
|
| map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
|
|
else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
|
|
Module load state (/proc/modules, corroboration only — a built-in driver never appears
|
|
there, the bound-driver symlink above is what decides):
|
|
{{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
|
|
| map('join', '=') | join(', ') }}.
|
|
{%- if (r.bound_drivers | length) == 0 %}
|
|
NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
|
|
subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
|
|
module exists for this kernel and whether it was blacklisted or failed to probe.
|
|
{%- elif expected in r.bound_drivers %}
|
|
This is the expected driver for this site (node_health_graphics_expected_driver).
|
|
{%- else %}
|
|
EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
|
|
works, so this is not an outage — but the two DRM drivers do not share a userspace
|
|
media stack, and a libva/iHD build that transcodes correctly against one is not
|
|
automatically right for the other. Confirm Plex still hardware-transcodes before
|
|
treating this as benign.
|
|
{%- endif %}
|
|
{%- endif -%}
|
|
|
|
# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
|
|
#
|
|
# GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
|
|
# costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
|
|
# through the media engines. Grading it would put a permanent WARN on a configuration the
|
|
# operator has chosen.
|
|
# No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
|
|
# driver inside its container, so the host needs neither. Flagging it would invite someone
|
|
# to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
|
|
#
|
|
# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
|
|
# (video-enhancement) are what a transcode actually runs on.
|
|
- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'graphics.media_stack',
|
|
'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
|
|
'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
|
|
'detail': media_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
g: "{{ node_health_facts.graphics }}"
|
|
r: "{{ node_health_render }}"
|
|
# `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
|
|
# that set the expected driver to amdgpu must get a clean "no such parameters" line
|
|
# instead of an undefined-key traceback.
|
|
guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
|
|
guc_text: >-
|
|
{{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
|
|
else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
|
|
if guc is mapping
|
|
else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
|
|
media_detail: >-
|
|
{%- if (r.display_devices | length) == 0 -%}
|
|
no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
|
|
firmware and nothing to transcode with.
|
|
{%- else -%}
|
|
media engines:
|
|
{{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
|
|
else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
|
|
(all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
|
|
GuC/HuC: {{ guc_text }}.
|
|
{{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
|
|
else 'No i915/xe parameter on the kernel cmdline' }}.
|
|
GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
|
|
low-power H.264 encode quality only, and both decode and encode still run on the
|
|
engines above.
|
|
Host VAAPI userspace: vainfo
|
|
{{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
|
|
else 'NOT installed on this host' }}
|
|
— also never graded, because Plex ships its own libva and media driver inside its
|
|
container, so the host needs none. This role installs nothing.
|
|
{%- endif -%}
|
|
|
|
# --- Thermals -----------------------------------------------------------------------
|
|
- name: Assess — temperatures
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'thermal.max',
|
|
'title': 'Hottest reported sensor',
|
|
'status': temp_status,
|
|
'detail': temp_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
hottest: "{{ (node_health_temps_c | max) if (node_health_temps_c | length) > 0 else none }}"
|
|
temp_status: >-
|
|
{{ 'SKIP' if hottest is not number
|
|
else ('FAIL' if hottest | float >= th.temp_fail_c
|
|
else ('WARN' if hottest | float >= th.temp_warn_c else 'PASS')) }}
|
|
temp_detail: >-
|
|
{{ 'no thermal sensor exposed (typical in a VM)' if hottest is not number
|
|
else (hottest | string ~ 'C hottest of ' ~ (node_health_temps_c | length) | string
|
|
~ ' sensors; zones: '
|
|
~ (node_health_facts.thermal.zones
|
|
| map(attribute='type') | join(', ')) ) }}
|
|
|
|
# --- Pressure-stall (PSI) -----------------------------------------------------------
|
|
- name: Assess — pressure stall (CPU, memory, I/O)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'pressure.' ~ item,
|
|
'title': 'PSI ' ~ item ~ ' — some avg60 (% of wall time stalled)',
|
|
'status': psi_status,
|
|
'detail': psi_detail }] }}"
|
|
loop: [cpu, memory, io]
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
psi: "{{ node_health_facts.pressure[item] | default(none) }}"
|
|
avg60: "{{ psi.some.avg60 | default(none) if psi is mapping else none }}"
|
|
psi_status: >-
|
|
{{ 'SKIP' if avg60 is not number
|
|
else ('FAIL' if avg60 | float >= th.psi_some_avg60_fail
|
|
else ('WARN' if avg60 | float >= th.psi_some_avg60_warn else 'PASS')) }}
|
|
psi_detail: >-
|
|
{{ '/proc/pressure/' ~ item ~ ' unavailable (kernel without CONFIG_PSI, or psi=0)'
|
|
if avg60 is not number
|
|
else ('some avg10=' ~ psi.some.avg10 | string ~ ' avg60=' ~ psi.some.avg60 | string
|
|
~ ' avg300=' ~ psi.some.avg300 | string
|
|
~ (' | full avg60=' ~ psi.full.avg60 | string if psi.full is defined else '')) }}
|
|
|
|
# --- Network ------------------------------------------------------------------------
|
|
- name: Assess — link state and speed
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'network.link',
|
|
'title': 'Physical NIC link state and negotiated speed',
|
|
'status': link_status,
|
|
'detail': link_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
th: "{{ node_health_thresholds }}"
|
|
up: "{{ node_health_nics_up }}"
|
|
slow: >-
|
|
{{ up | selectattr('speed_mbps', 'number')
|
|
| selectattr('speed_mbps', 'lt', th.nic_min_speed_mbps | int) | list }}
|
|
link_status: >-
|
|
{{ 'FAIL' if (up | length) == 0
|
|
else ('WARN' if (slow | length) > 0 else 'PASS') }}
|
|
link_detail: >-
|
|
{{ 'no physical interface is up' if (up | length) == 0
|
|
else (up | map(attribute='name') | zip(up | map(attribute='speed_mbps'))
|
|
| map('join', '=') | join('Mb/s, ') ~ 'Mb/s'
|
|
~ ' (driver ' ~ (up | map(attribute='driver') | join(', ')) ~ ')') }}
|
|
|
|
- name: Assess — link error counters
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'network.errors',
|
|
'title': 'NIC error counters since boot (rx/tx errors, CRC, carrier)',
|
|
'status': ('PASS' if err_total | int == 0 else 'WARN'),
|
|
'detail': err_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
up: "{{ node_health_nics_up }}"
|
|
err_keys: [rx_errors, tx_errors, rx_crc_errors, rx_frame_errors, rx_over_errors, tx_carrier_errors]
|
|
err_total: >-
|
|
{{ up | map(attribute='statistics') | map('dict2items') | flatten
|
|
| selectattr('key', 'in', err_keys) | map(attribute='value')
|
|
| select('number') | list | sum }}
|
|
drop_total: >-
|
|
{{ up | map(attribute='statistics') | map('dict2items') | flatten
|
|
| selectattr('key', 'in', ['rx_dropped', 'tx_dropped']) | map(attribute='value')
|
|
| select('number') | list | sum }}
|
|
err_detail: >-
|
|
{{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across '
|
|
~ (up | map(attribute='name') | join(', ')) }}
|
|
|
|
# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only
|
|
# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind
|
|
# it, because counting the stub would be a permanent FALSE single-upstream WARN that no
|
|
# correct configuration could ever clear. When that lookup fails it reports null WITH a
|
|
# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from
|
|
# an unknown is a fabricated verdict, and a PASS would be worse.
|
|
- name: Assess — DNS resolver redundancy
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'network.resolvers',
|
|
'title': 'Configured DNS upstreams (single upstream is a known SPOF here)',
|
|
'status': ('SKIP' if node_health_facts.network.nameservers is none
|
|
else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int)
|
|
else 'WARN')),
|
|
'detail': ns_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
# `| default([], true)` only ever fires on the null (unreadable) path, where the
|
|
# counts below are unused because the status is already SKIP.
|
|
ns: "{{ node_health_facts.network.nameservers | default([], true) }}"
|
|
# Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4
|
|
# upstream; IPv6 upstreams do not cover for it unless the node actually has working
|
|
# IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off
|
|
# the IPv4 count and the detail shows the full picture.
|
|
ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}"
|
|
ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}"
|
|
# Where the numbers came from — resolv.conf directly, or resolved behind the stub.
|
|
ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}"
|
|
ns_detail: >-
|
|
{{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason
|
|
| default('the collector recorded no reason', true))
|
|
~ '. The upstream count is UNKNOWN here — not zero, and not one.')
|
|
if node_health_facts.network.nameservers is none
|
|
else ((ns | length) | string ~ ' nameserver(s) — IPv4: '
|
|
~ (ns4 | join(', ') | default('none', true))
|
|
~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true))
|
|
~ (' (read via ' ~ ns_src ~ ')' if ns_src else '')
|
|
~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream
|
|
already broke containerd image pulls and CoreDNS forwarding on dev. The fix
|
|
is the `dns`-tagged block in roles/node_baseline — set this environment
|
|
node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for
|
|
prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate
|
|
if the node has working IPv6 egress, which this check cannot verify.'
|
|
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }}
|
|
|
|
# --- OS / firmware ------------------------------------------------------------------
|
|
- name: Assess — pending reboot
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'os.reboot_required',
|
|
'title': 'Pending reboot flag',
|
|
'status': ('WARN' if node_health_facts.os.reboot_required | bool else 'PASS'),
|
|
'detail': ('/run/reboot-required present — a package upgrade needs a restart'
|
|
if node_health_facts.os.reboot_required | bool
|
|
else 'no reboot flagged') }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Assess — OS, kernel and uptime
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'os.inventory',
|
|
'title': 'OS, kernel and uptime',
|
|
'status': 'INFO',
|
|
'detail': os_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
o: "{{ node_health_facts.os }}"
|
|
os_detail: >-
|
|
{{ (o.pretty_name | default('unknown OS')) ~ ', kernel ' ~ (o.kernel | default('?'))
|
|
~ ', up ' ~ ((o.uptime_seconds | float / 86400) | round(2) | string) ~ ' day(s)' }}
|
|
|
|
- name: Assess — firmware / DMI inventory
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'firmware.inventory',
|
|
'title': 'Chassis, board and BIOS firmware',
|
|
'status': ('INFO' if node_health_facts.dmi.present | bool else 'SKIP'),
|
|
'detail': dmi_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
d: "{{ node_health_facts.dmi }}"
|
|
dmi_detail: >-
|
|
{{ 'no DMI table exposed (typical for some VMs/containers)' if not (d.present | bool)
|
|
else ((d.sys_vendor | default('?')) ~ ' ' ~ (d.product_name | default('?'))
|
|
~ ' / board ' ~ (d.board_name | default('?'))
|
|
~ ' / BIOS ' ~ (d.bios_vendor | default('?')) ~ ' '
|
|
~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
|
|
~ (' — serial numbers not read (root-only DMI fields)'
|
|
if d.product_serial is none else '')) }}
|
|
|
|
# --- EFI Memory Attributes Table ------------------------------------------------------
|
|
# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
|
|
# what follows is only what this check needs to be read correctly.
|
|
#
|
|
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
|
|
# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
|
|
# are involved, and conflating them is what made this check wrong until 2026-09-03:
|
|
#
|
|
# efi_memattr_init() tbl_size = sizeof(*tbl) + num_entries * desc_size, then
|
|
# memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
|
|
# is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
|
|
# NEGATIVE: a garbage product with bit 31 set sign-extends
|
|
# through phys_addr_t and memblock_cap_size() clamps it to
|
|
# PHYS_ADDR_MAX — everything above the table is reserved.
|
|
# A garbage-but-POSITIVE tbl_size reserves a modest, wrong
|
|
# amount and costs nothing measurable.
|
|
# efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
|
|
# THIS is what prints "Failed to map". It fails for ANY
|
|
# sufficiently wrong tbl_size — negative OR merely large.
|
|
#
|
|
# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
|
|
# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
|
|
# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
|
|
# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
|
|
# is the counter-example that falsified the old inference.
|
|
#
|
|
# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
|
|
#
|
|
# "Failed to map ... @ 0x..." the bug FIRED: corrupt table, trusted by the kernel.
|
|
# FAIL when the shared measurement (node_health_mem_shortfall) says memory is
|
|
# catastrophically missing — memory is lost NOW.
|
|
# WARN otherwise — this boot drew a survivable garbage value; the next may not.
|
|
# "Unexpected ... version <N>" version read > 2, so the kernel bailed out before
|
|
# reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
|
|
# safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
|
|
# gates and no draw can reach the fatal arithmetic any more. Same line, opposite
|
|
# meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
|
|
# FW_BUG "Corrupted EFI Memory Attributes Table detected!" the >= 6.13 THIRD gate
|
|
# (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
|
|
# must not be treated as such: it is only reachable on a boot whose garbage leaves the
|
|
# version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
|
|
# line instead. All three prod nodes did exactly that on 2026-09-04.
|
|
#
|
|
# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
|
|
# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
|
|
# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
|
|
# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
|
|
# the draw still happens but every outcome is a rejection, so the same lines mean the
|
|
# validation is working.
|
|
#
|
|
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
|
|
# so it cannot be diagnosed once. It has to be measured on every health run.
|
|
#
|
|
# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
|
|
# node_health_mem_shortfall, computed once further up this file; this check does not measure
|
|
# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
|
|
# one verdict on any host and both checks quote it.
|
|
#
|
|
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
|
|
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
|
|
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
|
|
# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
|
|
# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
|
|
# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
|
|
# log at all, so a node whose log is unreadable is still not unmeasured.
|
|
- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: "{{ node_health_checks + [{
|
|
'id': 'firmware.efi_memattr',
|
|
'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
|
|
'status': memattr_status,
|
|
'detail': memattr_detail }] }}"
|
|
vars:
|
|
ansible_become: false
|
|
k: "{{ node_health_facts.klog }}"
|
|
z: "{{ node_health_facts.memory.zones }}"
|
|
# THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
|
|
# near memory.physical_total. This check contributes the kernel-log facts only.
|
|
ms: "{{ node_health_mem_shortfall }}"
|
|
ma_state: "{{ k.memattr_state | default(none) }}"
|
|
# DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
|
|
# carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
|
|
# THREE checks before it computes tbl_size at all:
|
|
# tbl->version > 2 -> "Unexpected ... table version %d"
|
|
# desc_size implausible -> "Unexpected ... descriptor size %u ..."
|
|
# num_entries > SZ_64K -> FW_BUG "Corrupted EFI Memory Attributes
|
|
# Table detected! ..."
|
|
# ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
|
|
# given boot produces is decided by WHICH field that boot's garbage lands in — it is not
|
|
# a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
|
|
# version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
|
|
# memblock_reserve(). That is the whole difference, and it is why the verdict for
|
|
# `bad_version` below is opposite on the two kernels: same log line, completely different
|
|
# meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
|
|
# something the `version` test can parse.
|
|
memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
|
|
memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
|
|
# `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
|
|
# COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
|
|
# this check and memory.physical_total can no longer disagree about lost memory.
|
|
# `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
|
|
# boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
|
|
# WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
|
|
# forever would make the check unable to ever show the fix landing, which is exactly what
|
|
# it is for.
|
|
memattr_status: >-
|
|
{{ 'SKIP' if not (k.available | bool)
|
|
else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
|
|
else ('WARN' if ma_state == 'failed_to_map'
|
|
else ('PASS' if ma_state == 'corrupt_rejected'
|
|
else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
|
|
if ma_state == 'bad_version'
|
|
else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
|
|
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
|
|
# kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
|
|
# could take effect. Suggesting one would be a fix that quietly does nothing.
|
|
memattr_fix: >-
|
|
FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
|
|
upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
|
|
bogus", first released in v6.13 and absent from v6.12 and from the current
|
|
linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
|
|
There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
|
|
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
|
|
unplugging USB only change where the table lands and may APPEAR to fix it — they are
|
|
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
|
# The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
|
|
# to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
|
|
# kernel log has no part in; the verdict word comes from ms.status, so it always agrees
|
|
# with memory.physical_total. When the measurement is missing it says so, and says why.
|
|
memattr_figures: >-
|
|
{{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
|
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
|
|
~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
|
|
allocator (/proc/zoneinfo)')
|
|
if ms.zones_pct is number
|
|
else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
|
|
reports as installed (/proc/zoneinfo NOT read — '
|
|
~ (z.reason | default('no reason recorded', true)) ~ ')')
|
|
if ms.dimm_pct is number else '') }}
|
|
memattr_corroboration: >-
|
|
{{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
|
|
~ ms.status ~ '): ' ~ memattr_figures
|
|
~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
|
|
else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
|
|
else ' — the normal few percent (kernel image, memmap, firmware
|
|
reservations, graphics stolen memory). NO measurable memory was lost
|
|
on this boot.')))
|
|
if (memattr_figures | trim | length) > 0
|
|
else (' Whether memory was lost on this boot could NOT be measured here: neither
|
|
present-vs-managed pages nor installed DIMM capacity was read — see check
|
|
memory.physical_total for the reasons.') }}
|
|
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
|
|
# against the evidence is not much better than a wrong one.
|
|
memattr_evidence: >-
|
|
{{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
|
|
if (k.memattr_lines | default([]) | length) > 0 else '')
|
|
~ ((' At boot the kernel reported '
|
|
~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
|
~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
|
|
~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
|
|
~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
|
|
if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
|
|
else '') }}
|
|
memattr_detail: >-
|
|
{%- if not (k.available | bool) -%}
|
|
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
|
|
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
|
|
lives only in dmesg.{{ memattr_corroboration }}
|
|
{%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
|
|
THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
|
|
line. The kernel failed to map the EFI Memory Attributes table at
|
|
{{ k.memattr_table_address | default('an unreported address') }}, which proves the
|
|
table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
|
|
from its contents; the measurement below shows that reservation was the catastrophic
|
|
kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
|
|
through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
|
|
table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
|
{%- elif ma_state == 'failed_to_map' -%}
|
|
THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
|
|
loss either. The kernel failed to map the EFI Memory Attributes table at
|
|
{{ k.memattr_table_address | default('an unreported address') }}: the table was
|
|
corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
|
|
WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
|
|
the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
|
|
with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
|
|
tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
|
|
measurable and still fails the later memremap() that printed this line. This boot drew a survivable
|
|
value; the draw is repeated on EVERY boot, at the same address — this platform has
|
|
been seen to lose 13.3 GiB and to keep its full memory at one identical table address
|
|
(infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
|
|
{{ memattr_fix }}
|
|
{%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
|
|
the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
|
|
kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
|
|
VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
|
|
boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
|
|
outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
|
|
efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
|
|
WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
|
|
Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
|
|
(num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
|
|
version field <= 2. Both messages are rejections, and which one a boot prints is the
|
|
lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
|
|
path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
|
|
PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
|
|
{{ memattr_corroboration }}
|
|
{%- elif ma_state == 'bad_version' -%}
|
|
EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
|
|
CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
|
|
version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
|
|
kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
|
|
EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
|
|
tbl_size the garbage produces — nothing measurable if that value is positive,
|
|
everything above the table if it is negative.{{ memattr_evidence }}
|
|
{{ memattr_corroboration }} {{ memattr_fix }}
|
|
{%- elif ma_state == 'corrupt_rejected' -%}
|
|
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
|
|
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
|
|
cannot cost this node memory on this or any future boot; no
|
|
action.{{ memattr_evidence }}{{ memattr_corroboration }}
|
|
{%- elif not (k.boot_covered | bool) -%}
|
|
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
|
|
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
|
|
the ring buffer, which wraps). Every memattr diagnostic is printed within the first
|
|
fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
|
|
pass. Read /var/log/kern.log for this boot, or re-run after a
|
|
reboot.{{ memattr_corroboration }}
|
|
{%- elif k.memattr_published is sameas true -%}
|
|
the firmware publishes an EFI Memory Attributes table at
|
|
{{ k.memattr_table_address | default('an unreported address') }} and the kernel
|
|
accepted it without complaint, in a log that still reaches back to early boot. Not
|
|
affected on this boot.
|
|
{%- elif k.memattr_published is sameas false -%}
|
|
this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
|
|
lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
|
|
guest.
|
|
{%- else -%}
|
|
no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
|
|
so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
|
|
boot, so that absence is measured, not assumed.
|
|
{%- endif -%}
|