# `node_health` — read-only node hardware health Sibling of `roles/cluster_verify`. `cluster_verify` answers *"is the cluster healthy"*; `node_health` answers *"is the hardware under it healthy"*. Same posture: it **inspects and reports, it never changes the node**. Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-03** (bare-metal parity with dev) and **R-OBS-04** (node-level observability). ```bash # prod (all nodes in the k3s_prod group) ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod' # dev (default target), one node ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1' ``` ## Contract - **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`, and — only when it can escalate — `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is `node_baseline`'s job, never a health check's, never a manual `apt-get`). - **A skip is honest; a fake green is a defect.** Every check is `PASS` / `WARN` / `FAIL` / `SKIP` / `INFO`, and a `SKIP` always carries *why* it could not be measured. Nothing that was not read is reported as passing. The summary ends with a dedicated "NOT READ (and why) — these are UNKNOWNS, not passes" section. - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI, EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM). - **Never `become`s unless it must.** Only SMART needs root. ## What it checks | id | what | |---|---| | `access.privilege` | whether `become` works at all (gates the root-only checks) | | `storage.smart.` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) | | `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) | | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes | | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` | | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS | Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded in task logic. ## Output 1. A human summary printed in the play output. 2. The same summary and a structured JSON record under `infra/ansible/_artifacts/node-health/.{txt,json}`. `_artifacts/` is in `infra/ansible/.gitignore` — the same convention as `conformance/_artifacts/`. Generated output is never committed (it also carries drive serial numbers). ## The `become` subtlety (read before editing) `ansible_become` set as an inventory/group var — which is exactly what `inventory/group_vars/k3s_prod/vars.yml` does — is a **connection variable**, and connection variables **outrank** the `become:` play/task keyword. Verified on ansible-core 2.20: a task saying `become: false` still tries to sudo and, on prod, dies with `Missing sudo password` before the module runs. So this role overrides the **variable** (`vars: ansible_become: false`) on the unprivileged work, and probes escalation exactly once before attempting SMART. Equally verified: `failed_when: false` does **not** catch a missing-sudo-password error (it is raised by the become plugin before the failure expression is evaluated); `ignore_errors: true` does, and still populates the registered variable. That is why the escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard, misleading play failure. ## How the SMART verdict is reached **SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8) is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields has been made entirely vendor-specific." This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15. Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a factor of 4096 because it assumed a unit the ATA spec does not guarantee. ### What the check judges on instead Vendor-**independent** signals only, each of which means the same thing on every drive: 1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL. 2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's `when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the normalised value is at or below the threshold *the manufacturer set for that attribute on that model*; a dash means it is OK now and has never failed. This needs no per-vendor knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL; Old_age failing now, or anything that failed in the past → WARN. 3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere: 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL; 199 CRC (SATA link, not media) → WARN only. 4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`. 5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe `percentage_used`) — a percentage by definition, and therefore portable where raw wear counts are not. ### What is reported but never produces a FAIL Vendor raw counts are real information and are all printed with their numbers — grown bad-block count, reallocation events (196), erase/program failures, unexpected power losses, and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can raise a **WARN**, never a FAIL, because their units are vendor-defined. The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are matched by **the name smartctl's drive database resolved for that model**, not by a bare ID: `drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field stays `null` — no claim is made either way. ### Two honesty traps this check does not fall into - **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400 "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding; a zero one proves nothing. The role reports the number and does not infer health from it. - **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every threshold un-tripped can still sit alongside a third of rated life consumed, grown bad blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those produce a WARN *with the numbers*, because a check that called that drive "fine" would be worse than the false FAIL it replaced. ## Known limits - Without root: **no SMART** (wear, power-on hours, error counters, drive temperature) and no DMI serial numbers. On prod that means creating `inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running `node-baseline` so `smartmontools` exists — in that order. - EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so. - SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive. On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply, but the wear/endurance context SKIPs rather than guessing. - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs `dmesg`, which needs root.