home-network-k3s-impl/infra/ansible/roles/node_health

node_health — read-only node hardware health

Sibling of roles/cluster_verify. cluster_verify answers "is the cluster healthy"; node_health answers "is the hardware under it healthy". Same posture: it inspects and reports, it never changes the node.

Serves R-ENV-01 (you cannot certify a node you have not measured), R-ENV-03 (bare-metal parity with dev) and R-OBS-04 (node-level observability).

# prod (all nodes in the k3s_prod group)
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'

# dev (default target), one node
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1'

Contract

• Read-only and side-effect-free. It reads /sys, /proc, /etc, runs lsblk/lscpu, and — only when it can escalate — smartctl. It writes nothing to the node and installs nothing. smartmontools is declared in node_baseline_packages; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is node_baseline's job, never a health check's, never a manual apt-get).
• A skip is honest; a fake green is a defect. Every check is PASS / WARN / FAIL / SKIP / INFO, and a SKIP always carries why it could not be measured. Nothing that was not read is reported as passing. The summary ends with a dedicated "NOT READ (and why) — these are UNKNOWNS, not passes" section.
• Works on bare metal and VMs. Everything absent in a guest (SMART on virtio, DMI, EDAC, coretemp, CPU throttle counters) skips cleanly instead of erroring. Verified against both k3s-prod-1 (HP EliteDesk, bare metal) and k3s-dev-1 (Proxmox VM).
• Never becomes unless it must. Only SMART needs root.

What it checks

id	what
access.privilege	whether become works at all (gates the root-only checks)
storage.smart.<disk>	overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — root only
storage.io_errors	kernel SCSI/ATA ioerr_cnt / iotmo_cnt per disk (no root needed)
storage.capacity, storage.inodes	worst mounted filesystem, space and inodes
storage.layout	physical disks, and whether any device is genuinely free for node_baseline_longhorn_device / node_baseline_etcd_device
memory.available, memory.swap, memory.ecc	headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters
cpu.throttle, cpu.inventory, cpu.virtualisation	thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + /dev/kvm (KubeVirt, R-PLAT-03)
thermal.max	hottest sensor across thermal zones and hwmon chips
pressure.cpu / .memory / .io	PSI some avg60 — dev learned to care about CPU PSI specifically
network.link, network.errors, network.resolvers	link state/speed, NIC error counters, DNS upstream redundancy per address family
os.reboot_required, os.inventory, firmware.inventory	pending reboot, OS/kernel/uptime, chassis + BIOS

Thresholds live in defaults/main.yml (node_health_thresholds); no verdict is hardcoded in task logic.

Output

1. A human summary printed in the play output.
2. The same summary and a structured JSON record under infra/ansible/_artifacts/node-health/<host>.{txt,json}.

_artifacts/ is in infra/ansible/.gitignore — the same convention as conformance/_artifacts/. Generated output is never committed (it also carries drive serial numbers).

The become subtlety (read before editing)

ansible_become set as an inventory/group var — which is exactly what inventory/group_vars/k3s_prod/vars.yml does — is a connection variable, and connection variables outrank the become: play/task keyword. Verified on ansible-core 2.20: a task saying become: false still tries to sudo and, on prod, dies with Missing sudo password before the module runs. So this role overrides the variable (vars: ansible_become: false) on the unprivileged work, and probes escalation exactly once before attempting SMART.

Equally verified: failed_when: false does not catch a missing-sudo-password error (it is raised by the become plugin before the failure expression is evaluated); ignore_errors: true does, and still populates the registered variable. That is why the escalation probe uses ignore_errors — anything else turns "no vault yet" into a hard, misleading play failure.

Known limits

• Without root: no SMART (wear, power-on hours, reallocated/pending sectors, drive temperature) and no DMI serial numbers. On prod that means creating inventory/group_vars/k3s_prod/vault.yml (see that directory's README) and running node-baseline so smartmontools exists — in that order.
• EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so.
• ioerr_cnt counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs dmesg, which needs root.