home-network-k3s-impl/infra/ansible/roles/node_health/README.md

node_health — read-only node hardware health

Sibling of roles/cluster_verify. cluster_verify answers "is the cluster healthy"; node_health answers "is the hardware under it healthy". Same posture: it inspects and reports, it never changes the node.

Serves R-ENV-01 (you cannot certify a node you have not measured), R-ENV-03 (bare-metal parity with dev) and R-OBS-04 (node-level observability).

# prod (all nodes in the k3s_prod group)
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'

# dev (default target), one node
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1'

Contract

• Read-only and side-effect-free. It reads /sys, /proc, /etc, runs lsblk/lscpu (and lspci only if it happens to be installed), and — only when it can escalate — smartctl. It writes nothing to the node and installs nothing. smartmontools is declared in node_baseline_packages; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is node_baseline's job, never a health check's, never a manual apt-get).
• A skip is honest; a fake green is a defect. Every check is PASS / WARN / FAIL / SKIP / INFO, and a SKIP always carries why it could not be measured. Nothing that was not read is reported as passing. The summary ends with a dedicated "NOT READ (and why) — these are UNKNOWNS, not passes" section.
• Works on bare metal and VMs. Everything absent in a guest (SMART on virtio, DMI, EDAC, coretemp, CPU throttle counters) skips cleanly instead of erroring. Verified against both k3s-prod-1 (HP EliteDesk, bare metal) and k3s-dev-1 (Proxmox VM).
• Never becomes unless it must. Three things need root: SMART, the kernel ring buffer (dmesg, because Debian sets kernel.dmesg_restrict=1) and the SMBIOS DIMM inventory (dmidecode -t 17). Everything else is read unprivileged in one pass, and the privileged second pass re-runs the same collector with --privileged so the parsing has one home. Without escalation those checks SKIP with the exact reason.

What it checks

id	what
access.privilege	whether become works at all (gates the root-only checks)
storage.smart.<disk>	overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — root only. See How the SMART verdict is reached
storage.io_errors	kernel SCSI/ATA ioerr_cnt / iotmo_cnt per disk (no root needed)
storage.capacity, storage.inodes	worst mounted filesystem, space and inodes
storage.layout	physical disks, and whether any device is genuinely free for node_baseline_longhorn_device / node_baseline_etcd_device
memory.available, memory.swap, memory.ecc	headroom, kubelet's swap-off prerequisite and whether it survives a reboot (see Swap off is not the same as swap disabled), EDAC correctable/uncorrectable counters
memory.physical_total	is this machine using the RAM it has? /proc/zoneinfo present-vs-managed pages (no root, metal or VM) and MemTotal vs the DIMM capacity SMBIOS reports (dmidecode -t 17, root, bare metal). Cause-independent by design — see The EFI memattr check
cpu.throttle, cpu.inventory, cpu.virtualisation	thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + /dev/kvm (KubeVirt, R-PLAT-03)
graphics.render_device	is there a usable /dev/dri/renderD* character device — the path Plex hardware-transcodes through. See The render-device check
graphics.drm_driver	which DRM driver is bound to the display device, against node_health_graphics_expected_driver (i915 here)
graphics.media_stack	INFO only: vcs*/vecs* media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why
thermal.max	hottest sensor across thermal zones and hwmon chips
pressure.cpu / .memory / .io	PSI some avg60 — dev learned to care about CPU PSI specifically
network.link, network.errors, network.resolvers	link state/speed, NIC error counters, DNS upstream redundancy per address family
os.reboot_required, os.inventory, firmware.inventory	pending reboot, OS/kernel/uptime, chassis + BIOS
firmware.efi_memattr	did the corrupt-EFI-Memory-Attributes-Table bug fire on this boot, and did it cost this node memory (per the measurement memory.physical_total reports) — root only. See The EFI memattr check

Thresholds live in defaults/main.yml (node_health_thresholds); no verdict is hardcoded in task logic.

Swap off is not the same as swap disabled

SwapTotal == 0 only says nothing has been swapped on yet this boot. It is not evidence that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod nodes came back from a reboot with 977 MiB of swap active (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2 /dev/nvme0n1p4) even though roles/node_baseline had run swapoff -a and commented their /etc/fstab swap line.

The cause is systemd-gpt-auto-generator. It claims any GPT partition whose partition type GUID is 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") straight from the partition table, synthesises a .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. /etc/fstab is never consulted, so neutralising fstab cannot stop it. roles/node_baseline masks swap.target — the one unit every generated .swap unit is WantedBy — and masks the generated units as a second barrier.

So memory.swap reports on both halves, from memory.swap_persistence in the collector (all world-readable, no root):

fact	meaning
swap_target_masked	is /etc/systemd/system/swap.target a symlink to /dev/null?
generated_units	.swap units systemd's generators built for this boot
masked_units	.swap units masked in /etc/systemd/system
unmasked_generated_units	generated_units minus masked_units — what a reboot would bring back
fstab_active_swap_lines	uncommented swap lines left in /etc/fstab

A node with no swap active but an unmasked generated unit and an unmasked swap.target WARNs: it is clean now and will be swapping again after the next reboot. The check used to blame that state on "this node has not been baselined yet", which sent the reader back to re-run the role that had already run.

The render-device check

Plex transcodes on the iGPU through /dev/dri/renderD128. When that node stops existing, nothing fails loudly. Plex falls back to software transcoding by itself; the only symptom is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly that silent degradation.

The state was measured by hand on all three prod nodes on 2026-09-30 and found working. A fact measured once decays silently, so it is re-measured on every health run instead of living in a runbook. Nothing is installed, no module is loaded, no group is changed — the role's read-only contract is unchanged.

fact	where it comes from	root?
/dev/dri contents, ownership, mode, char-vs-not	lstat() on each entry	no
owning group name for the render node's gid	grp.getgrgid()	no
bound DRM driver	/sys/class/drm/card*/device/driver symlink target	no
is the iGPU on the bus at all	/sys/bus/pci/devices/*/class (PCI base class 0x03)	no
media engines (vcs*, vecs*)	/sys/class/drm/card*/engine/*	no
module load state	/proc/modules	no
GuC/HuC	/sys/module/i915/parameters/enable_guc + /proc/cmdline	parameter file is mode 0400

The PCI scan is separate from the DRM scan on purpose

/sys/class/drm/card* exists only once a driver has bound. A machine whose iGPU sits on the bus with nothing attached has no card0 at all — and that is precisely the failure being hunted. So the display device is looked for on the PCI bus independently, from sysfs, which is always there. That is what lets graphics.render_device distinguish

• no GPU in this machine → SKIP (a VM guest; nothing to grade, the same convention thermal.max uses for "no sensor on this hardware"), from
• a GPU is on the bus and there is still no render node → FAIL.

lspci is used only to decorate the PCI entry with a readable name, and it is treated as optional because it is: pciutils is not declared in node_baseline_packages, so nothing in this repo guarantees it. It did turn out to be present on all three prod nodes (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but depending on that would mean depending on an undeclared package. Absent, description is null, the numeric IDs are printed instead, and no verdict changes.

The render node's GID is reported, never asserted

The render device was root:render, gid 992, mode 0660 on all three prod nodes on 2026-09-30. The check does not compare against 992, and must not be changed to. render is a systemd dynamic allocation (systemd-sysusers), not a Debian static gid from base-passwd, so a rebuilt node can legitimately land on a different number. Plex's own container entrypoint stats the device and joins whichever group owns it, so drift genuinely is not a fault. Recording the observed group name and gid in the detail line makes drift visible without making it an error — a hard != 992 would be a false alarm waiting for a reinstall.

What is configurable is the expected DRM driver, node_health_graphics_expected_driver (default i915), so the verdict stays out of task logic. xe bound instead of i915 is a WARN, not a FAIL: the device still works, but the two drivers do not share a userspace media stack and a libva/iHD build that transcodes correctly against one is not automatically right for the other. On these nodes xe.ko.xz exists on disk and is bound to nothing.

Two things that are informational and must stay that way

• GuC/HuC firmware not loaded (i915.enable_guc=0) is a known, accepted state here. It costs low-power H.264 encode quality only; decode and encode both still run on the media engines. Grading it would put a permanent WARN on a chosen configuration.
• No host VAAPI userspace / no vainfo is not a defect. Plex ships its own libva and media driver inside its container, so the host needs neither. Flagging it would invite someone to apt install vainfo on a node to clear a health check — and this role installs nothing.

enable_guc is one of i915's 0400 module parameters, so an unprivileged pass usually cannot read it. It reports as an honest NOT READ — <reason> rather than a fabricated 0, with /proc/cmdline as the world-readable second source for an explicit boot override.

The EFI memattr check

infra/PROD-PROVISIONING.md §0b is authoritative; this is only what the check needs.

HP's R21 firmware publishes a corrupt EFI Memory Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents. Two different kernel functions matter, and the check used to conflate them:

• efi_memattr_init() computes tbl_size = sizeof(*tbl) + num_entries * desc_size — both fields unvalidated — and memblock_reserve()s it. tbl_size is a signed int, so catastrophic loss requires it to be NEGATIVE: bit 31 set sign-extends through phys_addr_t and memblock_cap_size() clamps it to PHYS_ADDR_MAX, reserving everything above the table. A garbage-but-positive tbl_size reserves a modest, wrong amount and costs nothing measurable.
• efi_memattr_apply_permissions() later memremap()s the table. This is what prints "Failed to map", and it fails for any sufficiently wrong tbl_size — negative or merely large.

So Failed to map proves the table was corrupt and trusted by the kernel. It does not prove memory was lost: that depends on the garbage value at that address on that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then (2026-09-03) booted with all 15.1 GiB — same address, same log line. The verdict is therefore based on measured memory, never on the log line alone:

kernel log	measured shortfall	verdict
efi: memattr: Failed to map EFI Memory Attributes table @ 0x…	large (memory.physical_total FAIL)	FAIL — the bug fired and memory is lost now, measured
efi: memattr: Failed to map EFI Memory Attributes table @ 0x…	normal	WARN — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not
efi: memattr: Unexpected EFI Memory Attributes table version <N>, kernel < 6.13	(not consulted)	WARN — the garbage version read outside 0..2 so the kernel bailed out first: exposed, but this boot took the safe path
efi: memattr: Unexpected EFI Memory Attributes table version <N>, kernel >= 6.13	(not consulted)	PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic
[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!	(not consulted)	PASS — kernel >= 6.13 rejected it at the num_entries > 64K gate: this is the fix, working
no memattr line, log reaches back to boot	(not consulted)	PASS
no memattr line, log does not reach back to boot	—	SKIP — the ring buffer wrapped past early boot: unknown, not unaffected
dmesg unreadable	—	SKIP — no root / dmesg_restrict=1: unknown, not unaffected

The WARN is the point on a kernel < 6.13. All three prod nodes publish the same corrupt table and differ only by luck, boot to boot. A node whose kernel just trusted a corrupt table is not passing a health check even when it kept its RAM.

The FW_BUG line is proof, but its ABSENCE proves nothing

efi_memattr_init() on >= 6.13 has three gates, and only the third prints the FW_BUG line:

gate	message	reachable when
tbl->version > 2	Unexpected EFI Memory Attributes table version %d	the garbage lands in the version field
implausible desc_size	Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)	version reads 0..2, desc_size does not
num_entries > SZ_64K	[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)	version and desc_size read plausibly, entry count does not

All three are rejections. Which one a boot prints is decided by which field that boot's garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very often prints the first message, and looking only for the FW_BUG line will make a working fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed the version line, not the FW_BUG one. That is why the verdict keys on the kernel version alongside the log line, and why the check must never demand the FW_BUG string as its evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in the kernel image itself, not a line in dmesg.

The version field is printed with %d — signed. Garbage with bit 31 set is logged negative (version -1569980416, k3s-prod-2, 2026-09-04). The collector regex missed that until 2026-09-04 and such nodes fell through to clean, i.e. a kernel that had just rejected a corrupt table was reported as one that "accepted it without complaint" — a false PASS on the exact bug this check exists to find.

Fix: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports); upstream commit 8fbe4c49c0cc. There is no kernel cmdline workaround — efi_memattr_init() runs before any efi= bail-out could take effect, and rebooting/CMOS/BIOS/USB changes only move where the table lands.

The two memory checks cannot contradict each other

memory.physical_total and firmware.efi_memattr both need the answer to "did this machine lose memory?", and it is measured once — /proc/zoneinfo present-vs-managed pages plus the SMBIOS-vs-MemTotal shortfall — into the shared node_health_mem_shortfall fact in tasks/assess.yml. memory.physical_total is that verdict; firmware.efi_memattr reads it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of its own, so there is exactly one verdict on lost memory per host and both checks quote it. A node whose kernel log cannot be read is still measured by memory.physical_total, and a future memory loss with a different cause is still caught.

Output

1. A human summary printed in the play output.
2. The same summary and a structured JSON record under infra/ansible/_artifacts/node-health/<host>.{txt,json}.

_artifacts/ is in infra/ansible/.gitignore — the same convention as conformance/_artifacts/. Generated output is never committed (it also carries drive serial numbers).

The become subtlety (read before editing)

ansible_become set as an inventory/group var — which is exactly what inventory/group_vars/k3s_prod/vars.yml does — is a connection variable, and connection variables outrank the become: play/task keyword. Verified on ansible-core 2.20: a task saying become: false still tries to sudo and, on prod, dies with Missing sudo password before the module runs. So this role overrides the variable (vars: ansible_become: false) on the unprivileged work, and probes escalation exactly once before attempting SMART.

Equally verified: failed_when: false does not catch a missing-sudo-password error (it is raised by the become plugin before the failure expression is evaluated); ignore_errors: true does, and still populates the registered variable. That is why the escalation probe uses ignore_errors — anything else turns "no vault yet" into a hard, misleading play failure.

How the SMART verdict is reached

SMART raw values are vendor-defined and this role does not judge on them. smartctl(8) is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields has been made entirely vendor-specific."

This is not theoretical. The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 (C400-MTFDDAC512MAM, fw 070H) on reallocated=61440, read as a plain bad-sector count. Micron's own TN-FD-20 SMART Attributes for the C400 SSD defines attribute 5 not as a sector count but as Reallocated Block Count, whose raw is "the grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that multiplier is 256 x 16 = 4096, and 61440 / 4096 = 15 — corroborated on the same drive by attribute 170 Grown_Failing_Block_Ct = 15 and attribute 196 Reallocated_Event_Count = 15. Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a factor of 4096 because it assumed a unit the ATA spec does not guarantee.

What the check judges on instead

Vendor-independent signals only, each of which means the same thing on every drive:

1. The drive's own overall-health self-assessment (smart_status.passed) — FAIL.
2. Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD, taken from smartctl's when_failed field ("now" / "past" / ""). Per smartctl(8), FAILING_NOW means the normalised value is at or below the threshold the manufacturer set for that attribute on that model; a dash means it is OK now and has never failed. This needs no per-vendor knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL; Old_age failing now, or anything that failed in the past → WARN.
3. ATA-standard error counters whose raw meaning is a plain event count everywhere: 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL; 199 CRC (SATA link, not media) → WARN only.
4. NVMe's structured health log — critical_warning, media_errors, percentage_used.
5. Wear from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe percentage_used) — a percentage by definition, and therefore portable where raw wear counts are not.

What is reported but never produces a FAIL

Vendor raw counts are real information and are all printed with their numbers — grown bad-block count, reallocation events (196), erase/program failures, unexpected power losses, and attribute 5's raw shown beside the normalised reading that contradicts it. They can raise a WARN, never a FAIL, because their units are vendor-defined.

The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are matched by the name smartctl's drive database resolved for that model, not by a bare ID: drivedb.h is smartmontools' own per-(model, firmware) translation, so keying off its resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive drivedb does not recognise, the name is Unknown_Attribute, nothing matches, and the field stays null — no claim is made either way.

Two honesty traps this check does not fall into

• Zero is not a clean bill of health. Micron TN-FD-20 states attribute 197 on the C400 "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding; a zero one proves nothing. The role reports the number and does not infer health from it.
• A quiet drive is not necessarily a good drive. overall-health=PASSED with every threshold un-tripped can still sit alongside a third of rated life consumed, grown bad blocks, erase failures and unprotected power losses — which is exactly k3s-prod-1. Those produce a WARN with the numbers, because a check that called that drive "fine" would be worse than the false FAIL it replaced.

Known limits

• Without root: no SMART (wear, power-on hours, error counters, drive temperature) and no DMI serial numbers. On prod that means creating inventory/group_vars/k3s_prod/vault.yml (see that directory's README) and running node-baseline so smartmontools exists — in that order.
• EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so.
• SMART wear and endurance context depends on smartmontools' drivedb.h knowing the drive. On an unrecognised model the vendor attributes are Unknown_Attribute; the portable signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply, but the wear/endurance context SKIPs rather than guessing.
• ioerr_cnt counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs dmesg, which needs root.
• graphics.render_device's FAIL path has never fired against a real node — every prod node has a working render device, and the role is read-only so one cannot be taken away to prove it. The failure branches were exercised against synthetic fact sets only. The same applies to the xe-bound WARN: no node here has xe bound.
• graphics.* proves the render device exists and which driver owns it. It does not prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container, with Plex's own libva, which is out of scope for a node-level check that installs nothing.