home-network-k3s-impl/infra/ansible/roles/node_health

node_health — read-only node hardware health

Sibling of roles/cluster_verify. cluster_verify answers "is the cluster healthy"; node_health answers "is the hardware under it healthy". Same posture: it inspects and reports, it never changes the node.

Serves R-ENV-01 (you cannot certify a node you have not measured), R-ENV-03 (bare-metal parity with dev) and R-OBS-04 (node-level observability).

# prod (all nodes in the k3s_prod group)
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'

# dev (default target), one node
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1'

Contract

• Read-only and side-effect-free. It reads /sys, /proc, /etc, runs lsblk/lscpu, and — only when it can escalate — smartctl. It writes nothing to the node and installs nothing. smartmontools is declared in node_baseline_packages; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is node_baseline's job, never a health check's, never a manual apt-get).
• A skip is honest; a fake green is a defect. Every check is PASS / WARN / FAIL / SKIP / INFO, and a SKIP always carries why it could not be measured. Nothing that was not read is reported as passing. The summary ends with a dedicated "NOT READ (and why) — these are UNKNOWNS, not passes" section.
• Works on bare metal and VMs. Everything absent in a guest (SMART on virtio, DMI, EDAC, coretemp, CPU throttle counters) skips cleanly instead of erroring. Verified against both k3s-prod-1 (HP EliteDesk, bare metal) and k3s-dev-1 (Proxmox VM).
• Never becomes unless it must. Three things need root: SMART, the kernel ring buffer (dmesg, because Debian sets kernel.dmesg_restrict=1) and the SMBIOS DIMM inventory (dmidecode -t 17). Everything else is read unprivileged in one pass, and the privileged second pass re-runs the same collector with --privileged so the parsing has one home. Without escalation those checks SKIP with the exact reason.

What it checks

id	what
access.privilege	whether become works at all (gates the root-only checks)
storage.smart.<disk>	overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — root only. See How the SMART verdict is reached
storage.io_errors	kernel SCSI/ATA ioerr_cnt / iotmo_cnt per disk (no root needed)
storage.capacity, storage.inodes	worst mounted filesystem, space and inodes
storage.layout	physical disks, and whether any device is genuinely free for node_baseline_longhorn_device / node_baseline_etcd_device
memory.available, memory.swap, memory.ecc	headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters
memory.physical_total	is this machine using the RAM it has? /proc/zoneinfo present-vs-managed pages (no root, metal or VM) and MemTotal vs the DIMM capacity SMBIOS reports (dmidecode -t 17, root, bare metal). Cause-independent by design — see The EFI memattr check
cpu.throttle, cpu.inventory, cpu.virtualisation	thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + /dev/kvm (KubeVirt, R-PLAT-03)
thermal.max	hottest sensor across thermal zones and hwmon chips
pressure.cpu / .memory / .io	PSI some avg60 — dev learned to care about CPU PSI specifically
network.link, network.errors, network.resolvers	link state/speed, NIC error counters, DNS upstream redundancy per address family
os.reboot_required, os.inventory, firmware.inventory	pending reboot, OS/kernel/uptime, chassis + BIOS
firmware.efi_memattr	the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — root only. See The EFI memattr check

Thresholds live in defaults/main.yml (node_health_thresholds); no verdict is hardcoded in task logic.

The EFI memattr check

infra/PROD-PROVISIONING.md §0b is authoritative; this is only what the check needs.

HP's R21 firmware publishes a corrupt EFI Memory Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents — drivers/firmware/efi/memattr.c multiplies two never-validated fields (num_entries * desc_size) and memblock_reserve()s the product, i.e. everything from the table address to the top of RAM. Which of two kernel-log lines appears decides whether the node keeps its memory, and it is decided afresh on every boot:

kernel log	meaning	verdict
efi: memattr: Failed to map EFI Memory Attributes table @ 0x…	the garbage version field read <= 2, the kernel trusted the table	FAIL — the node has lost the memory now
efi: memattr: Unexpected EFI Memory Attributes table version <N>	the garbage version read > 2, the kernel bailed out first	WARN — memory intact this boot only; the firmware is still publishing a corrupt table
[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!	kernel >= 6.13 validated and rejected it	PASS — this is the fix, working
no memattr line, log reaches back to boot	no table published (VM/OVMF), or a valid one	PASS
no memattr line, log does not reach back to boot	the ring buffer wrapped past early boot	SKIP — unknown, not unaffected
dmesg unreadable	no root / dmesg_restrict=1	SKIP — unknown, not unaffected

The WARN is the point. k3s-prod-1 and k3s-prod-2 publish the same corrupt table and differ only by luck: k3s-prod-2 itself booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.

Fix: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports); upstream commit 8fbe4c49c0cc. There is no kernel cmdline workaround — efi_memattr_init() runs before any efi= bail-out could take effect, and rebooting/CMOS/BIOS/USB changes only move where the table lands.

memory.physical_total measures the same symptom from /proc/zoneinfo alone, so a node whose kernel log cannot be read is still not unmeasured — and a future memory loss with a different cause is still caught.

Output

1. A human summary printed in the play output.
2. The same summary and a structured JSON record under infra/ansible/_artifacts/node-health/<host>.{txt,json}.

_artifacts/ is in infra/ansible/.gitignore — the same convention as conformance/_artifacts/. Generated output is never committed (it also carries drive serial numbers).

The become subtlety (read before editing)

ansible_become set as an inventory/group var — which is exactly what inventory/group_vars/k3s_prod/vars.yml does — is a connection variable, and connection variables outrank the become: play/task keyword. Verified on ansible-core 2.20: a task saying become: false still tries to sudo and, on prod, dies with Missing sudo password before the module runs. So this role overrides the variable (vars: ansible_become: false) on the unprivileged work, and probes escalation exactly once before attempting SMART.

Equally verified: failed_when: false does not catch a missing-sudo-password error (it is raised by the become plugin before the failure expression is evaluated); ignore_errors: true does, and still populates the registered variable. That is why the escalation probe uses ignore_errors — anything else turns "no vault yet" into a hard, misleading play failure.

How the SMART verdict is reached

SMART raw values are vendor-defined and this role does not judge on them. smartctl(8) is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields has been made entirely vendor-specific."

This is not theoretical. The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 (C400-MTFDDAC512MAM, fw 070H) on reallocated=61440, read as a plain bad-sector count. Micron's own TN-FD-20 SMART Attributes for the C400 SSD defines attribute 5 not as a sector count but as Reallocated Block Count, whose raw is "the grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that multiplier is 256 x 16 = 4096, and 61440 / 4096 = 15 — corroborated on the same drive by attribute 170 Grown_Failing_Block_Ct = 15 and attribute 196 Reallocated_Event_Count = 15. Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a factor of 4096 because it assumed a unit the ATA spec does not guarantee.

What the check judges on instead

Vendor-independent signals only, each of which means the same thing on every drive:

1. The drive's own overall-health self-assessment (smart_status.passed) — FAIL.
2. Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD, taken from smartctl's when_failed field ("now" / "past" / ""). Per smartctl(8), FAILING_NOW means the normalised value is at or below the threshold the manufacturer set for that attribute on that model; a dash means it is OK now and has never failed. This needs no per-vendor knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL; Old_age failing now, or anything that failed in the past → WARN.
3. ATA-standard error counters whose raw meaning is a plain event count everywhere: 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL; 199 CRC (SATA link, not media) → WARN only.
4. NVMe's structured health log — critical_warning, media_errors, percentage_used.
5. Wear from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe percentage_used) — a percentage by definition, and therefore portable where raw wear counts are not.

What is reported but never produces a FAIL

Vendor raw counts are real information and are all printed with their numbers — grown bad-block count, reallocation events (196), erase/program failures, unexpected power losses, and attribute 5's raw shown beside the normalised reading that contradicts it. They can raise a WARN, never a FAIL, because their units are vendor-defined.

The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are matched by the name smartctl's drive database resolved for that model, not by a bare ID: drivedb.h is smartmontools' own per-(model, firmware) translation, so keying off its resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive drivedb does not recognise, the name is Unknown_Attribute, nothing matches, and the field stays null — no claim is made either way.

Two honesty traps this check does not fall into

• Zero is not a clean bill of health. Micron TN-FD-20 states attribute 197 on the C400 "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding; a zero one proves nothing. The role reports the number and does not infer health from it.
• A quiet drive is not necessarily a good drive. overall-health=PASSED with every threshold un-tripped can still sit alongside a third of rated life consumed, grown bad blocks, erase failures and unprotected power losses — which is exactly k3s-prod-1. Those produce a WARN with the numbers, because a check that called that drive "fine" would be worse than the false FAIL it replaced.

Known limits

• Without root: no SMART (wear, power-on hours, error counters, drive temperature) and no DMI serial numbers. On prod that means creating inventory/group_vars/k3s_prod/vault.yml (see that directory's README) and running node-baseline so smartmontools exists — in that order.
• EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so.
• SMART wear and endurance context depends on smartmontools' drivedb.h knowing the drive. On an unrecognised model the vendor attributes are Unknown_Attribute; the portable signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply, but the wear/endurance context SKIPs rather than guessing.
• ioerr_cnt counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs dmesg, which needs root.