# `node_health` — read-only node hardware health

Sibling of `roles/cluster_verify`. `cluster_verify` answers *"is the cluster healthy"*;
`node_health` answers *"is the hardware under it healthy"*. Same posture: it **inspects and
reports, it never changes the node**.

Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-03**
(bare-metal parity with dev) and **R-OBS-04** (node-level observability).

```bash
# prod (all nodes in the k3s_prod group)
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'

# dev (default target), one node
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1'
```

## Contract

- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
  and — only when it can escalate — `smartctl`. It writes nothing to the node and
  **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
  absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
  package installation is `node_baseline`'s job, never a health check's, never a manual
  `apt-get`).
- **A skip is honest; a fake green is a defect.** Every check is `PASS` / `WARN` / `FAIL` /
  `SKIP` / `INFO`, and a `SKIP` always carries *why* it could not be measured. Nothing that
  was not read is reported as passing. The summary ends with a dedicated
  "NOT READ (and why) — these are UNKNOWNS, not passes" section.
- **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
  EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
  against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
- **Never `become`s unless it must.** Only SMART needs root.

## What it checks

| id | what |
|---|---|
| `access.privilege` | whether `become` works at all (gates the root-only checks) |
| `storage.smart.<disk>` | overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — **root only** |
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
| `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
| `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
| `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |

Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
in task logic.

## Output

1. A human summary printed in the play output.
2. The same summary and a structured JSON record under
   `infra/ansible/_artifacts/node-health/<host>.{txt,json}`.

`_artifacts/` is in `infra/ansible/.gitignore` — the same convention as
`conformance/_artifacts/`. Generated output is never committed (it also carries drive
serial numbers).

## The `become` subtlety (read before editing)

`ansible_become` set as an inventory/group var — which is exactly what
`inventory/group_vars/k3s_prod/vars.yml` does — is a **connection variable**, and
connection variables **outrank** the `become:` play/task keyword. Verified on
ansible-core 2.20: a task saying `become: false` still tries to sudo and, on prod, dies
with `Missing sudo password` before the module runs. So this role overrides the
**variable** (`vars: ansible_become: false`) on the unprivileged work, and probes
escalation exactly once before attempting SMART.

Equally verified: `failed_when: false` does **not** catch a missing-sudo-password error
(it is raised by the become plugin before the failure expression is evaluated);
`ignore_errors: true` does, and still populates the registered variable. That is why the
escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard,
misleading play failure.

## Known limits

- Without root: **no SMART** (wear, power-on hours, reallocated/pending sectors, drive
  temperature) and no DMI serial numbers. On prod that means creating
  `inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running
  `node-baseline` so `smartmontools` exists — in that order.
- EDAC reports nothing on platforms with no ECC memory controller bound. That is
  "ECC status unknown", not "no errors", and the role says so.
- `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
  unsupported commands during discovery. Attribution needs `dmesg`, which needs root.