# `node_health` — read-only node hardware health Sibling of `roles/cluster_verify`. `cluster_verify` answers *"is the cluster healthy"*; `node_health` answers *"is the hardware under it healthy"*. Same posture: it **inspects and reports, it never changes the node**. Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-03** (bare-metal parity with dev) and **R-OBS-04** (node-level observability). ```bash # prod (all nodes in the k3s_prod group) ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod' # dev (default target), one node ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-health.yml --limit k3s-dev-1' ``` ## Contract - **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu` (and `lspci` only if it happens to be installed), and — only when it can escalate — `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31: package installation is `node_baseline`'s job, never a health check's, never a manual `apt-get`). - **A skip is honest; a fake green is a defect.** Every check is `PASS` / `WARN` / `FAIL` / `SKIP` / `INFO`, and a `SKIP` always carries *why* it could not be measured. Nothing that was not read is reported as passing. The summary ends with a dedicated "NOT READ (and why) — these are UNKNOWNS, not passes" section. - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI, EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM). - **Never `become`s unless it must.** Three things need root: SMART, the kernel ring buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and the privileged second pass re-runs the *same* collector with `--privileged` so the parsing has one home. Without escalation those checks SKIP with the exact reason. ## What it checks | id | what | |---|---| | `access.privilege` | whether `become` works at all (gates the root-only checks) | | `storage.smart.` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) | | `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) | | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes | | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` | | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters | | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | | `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) | | `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) | | `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS | | `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) | Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded in task logic. ### Swap off is not the same as swap disabled `SwapTotal == 0` only says nothing has been swapped on *yet this boot*. It is not evidence that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod nodes came back from a reboot with 977 MiB of swap active (`k3s-prod-1` `/dev/sda4`, `k3s-prod-3` `/dev/sda4`, `k3s-prod-2` `/dev/nvme0n1p4`) even though `roles/node_baseline` had run `swapoff -a` and commented their `/etc/fstab` swap line. The cause is `systemd-gpt-auto-generator`. It claims any GPT partition whose partition **type GUID** is `0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` ("Linux swap") straight from the partition table, synthesises a `.swap` unit into `/run/systemd/generator.late/` and links it into `swap.target.wants/`. `/etc/fstab` is never consulted, so neutralising fstab cannot stop it. `roles/node_baseline` masks `swap.target` — the one unit every generated `.swap` unit is `WantedBy` — and masks the generated units as a second barrier. So `memory.swap` reports on both halves, from `memory.swap_persistence` in the collector (all world-readable, no root): | fact | meaning | |---|---| | `swap_target_masked` | is `/etc/systemd/system/swap.target` a symlink to `/dev/null`? | | `generated_units` | `.swap` units systemd's generators built for this boot | | `masked_units` | `.swap` units masked in `/etc/systemd/system` | | `unmasked_generated_units` | `generated_units` minus `masked_units` — what a reboot would bring back | | `fstab_active_swap_lines` | uncommented `swap` lines left in `/etc/fstab` | A node with no swap active but an unmasked generated unit and an unmasked `swap.target` **WARNs**: it is clean now and will be swapping again after the next reboot. The check used to blame that state on "this node has not been baselined yet", which sent the reader back to re-run the role that had already run. ## The render-device check Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing, nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly that silent degradation. The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A fact measured once decays silently**, so it is re-measured on every health run instead of living in a runbook. Nothing is installed, no module is loaded, no group is changed — the role's read-only contract is unchanged. | fact | where it comes from | root? | |---|---|---| | `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no | | owning group **name** for the render node's gid | `grp.getgrgid()` | no | | bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no | | is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no | | media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no | | module load state | `/proc/modules` | no | | GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** | ### The PCI scan is separate from the DRM scan on purpose `/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on the bus with nothing attached has no `card0` at all — and that is precisely the failure being hunted. So the display device is looked for on the PCI bus independently, from sysfs, which is always there. That is what lets `graphics.render_device` distinguish - *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention `thermal.max` uses for "no sensor on this hardware"), from - *a GPU is on the bus and there is still no render node* → **FAIL**. `lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing in this repo guarantees it. It did turn out to be present on all three prod nodes (`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending on that would mean depending on an undeclared package. Absent, `description` is `null`, the numeric IDs are printed instead, and no verdict changes. ### The render node's GID is reported, never asserted The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on 2026-09-30. The check does **not** compare against `992`, and must not be changed to. `render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own container entrypoint stats the device and joins whichever group owns it, so drift genuinely is not a fault. Recording the observed group name and gid in the detail line makes drift **visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a reinstall. What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver` (default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace media stack and a libva/iHD build that transcodes correctly against one is not automatically right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing. ### Two things that are informational and must stay that way - **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It costs low-power H.264 encode *quality* only; decode and encode both still run on the media engines. Grading it would put a permanent WARN on a chosen configuration. - **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and media driver inside its container, so the host needs neither. Flagging it would invite someone to `apt install vainfo` on a node to clear a health check — and this role installs nothing. `enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually cannot read it. It reports as an honest `NOT READ — ` rather than a fabricated `0`, with `/proc/cmdline` as the world-readable second source for an explicit boot override. ## The EFI memattr check `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13** does not validate it before reserving memory from its contents. **Two different kernel functions matter, and the check used to conflate them:** - `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and costs nothing measurable. - `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or merely large. So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not* prove memory was lost: that depends on the garbage **value** at that address on that boot, not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03) booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on measured memory, never on the log line alone:** | kernel log | measured shortfall | verdict | |---|---|---| | `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured | | `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not | | `efi: memattr: Unexpected EFI Memory Attributes table version `, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path | | `efi: memattr: Unexpected EFI Memory Attributes table version `, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic | | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working | | no memattr line, log reaches back to boot | (not consulted) | PASS | | no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected | | `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected | The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt table is not passing a health check even when it kept its RAM. ### The FW_BUG line is proof, but its ABSENCE proves nothing `efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG line: | gate | message | reachable when | |---|---|---| | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field | | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not | | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not | **All three are rejections.** Which one a boot prints is decided by *which field* that boot's garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very often prints the *first* message, and looking only for the FW_BUG line will make a working fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version alongside the log line, and why the check must never demand the FW_BUG string as its evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in the kernel image itself, not a line in `dmesg`. **The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged *negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected a corrupt table was reported as one that "accepted it without complaint" — a false PASS on the exact bug this check exists to find. **Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from `trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and rebooting/CMOS/BIOS/USB changes only move where the table lands. ### The two memory checks cannot contradict each other `memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in `tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of its own, so there is exactly one verdict on lost memory per host and both checks quote it. A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a future memory loss with a different cause is still caught. ## Output 1. A human summary printed in the play output. 2. The same summary and a structured JSON record under `infra/ansible/_artifacts/node-health/.{txt,json}`. `_artifacts/` is in `infra/ansible/.gitignore` — the same convention as `conformance/_artifacts/`. Generated output is never committed (it also carries drive serial numbers). ## The `become` subtlety (read before editing) `ansible_become` set as an inventory/group var — which is exactly what `inventory/group_vars/k3s_prod/vars.yml` does — is a **connection variable**, and connection variables **outrank** the `become:` play/task keyword. Verified on ansible-core 2.20: a task saying `become: false` still tries to sudo and, on prod, dies with `Missing sudo password` before the module runs. So this role overrides the **variable** (`vars: ansible_become: false`) on the unprivileged work, and probes escalation exactly once before attempting SMART. Equally verified: `failed_when: false` does **not** catch a missing-sudo-password error (it is raised by the become plugin before the failure expression is evaluated); `ignore_errors: true` does, and still populates the registered variable. That is why the escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard, misleading play failure. ## How the SMART verdict is reached **SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8) is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields has been made entirely vendor-specific." This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15. Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a factor of 4096 because it assumed a unit the ATA spec does not guarantee. ### What the check judges on instead Vendor-**independent** signals only, each of which means the same thing on every drive: 1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL. 2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's `when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the normalised value is at or below the threshold *the manufacturer set for that attribute on that model*; a dash means it is OK now and has never failed. This needs no per-vendor knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL; Old_age failing now, or anything that failed in the past → WARN. 3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere: 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL; 199 CRC (SATA link, not media) → WARN only. 4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`. 5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe `percentage_used`) — a percentage by definition, and therefore portable where raw wear counts are not. ### What is reported but never produces a FAIL Vendor raw counts are real information and are all printed with their numbers — grown bad-block count, reallocation events (196), erase/program failures, unexpected power losses, and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can raise a **WARN**, never a FAIL, because their units are vendor-defined. The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are matched by **the name smartctl's drive database resolved for that model**, not by a bare ID: `drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field stays `null` — no claim is made either way. ### Two honesty traps this check does not fall into - **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400 "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding; a zero one proves nothing. The role reports the number and does not infer health from it. - **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every threshold un-tripped can still sit alongside a third of rated life consumed, grown bad blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those produce a WARN *with the numbers*, because a check that called that drive "fine" would be worse than the false FAIL it replaced. ## Known limits - Without root: **no SMART** (wear, power-on hours, error counters, drive temperature) and no DMI serial numbers. On prod that means creating `inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running `node-baseline` so `smartmontools` exists — in that order. - EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so. - SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive. On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply, but the wear/endurance context SKIPs rather than guessing. - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs `dmesg`, which needs root. - `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod node has a working render device, and the role is read-only so one cannot be taken away to prove it. The failure branches were exercised against synthetic fact sets only. The same applies to the `xe`-bound WARN: no node here has `xe` bound. - `graphics.*` proves the render **device** exists and which driver owns it. It does **not** prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container, with Plex's own libva, which is out of scope for a node-level check that installs nothing.