fix(node_health): memory.swap blamed the wrong thing, and could not see the defect

The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.

The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.

The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".

Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
  [PASS] memory.swap — no swap active and none can be activated: swap.target is masked

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 22:35:31 -05:00
1 parent 0e92d258dd
commit 9d00a3fcd3
3 files changed
+175 -8

No files matched your search

+32 -1
View File
@@ -47,7 +47,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
@@ -59,6 +59,37 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
in task logic.
### Swap off is not the same as swap disabled
`SwapTotal == 0` only says nothing has been swapped on *yet this boot*. It is not evidence
that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod
nodes came back from a reboot with 977 MiB of swap active (`k3s-prod-1` `/dev/sda4`,
`k3s-prod-3` `/dev/sda4`, `k3s-prod-2` `/dev/nvme0n1p4`) even though `roles/node_baseline`
had run `swapoff -a` and commented their `/etc/fstab` swap line.
The cause is `systemd-gpt-auto-generator`. It claims any GPT partition whose partition
**type GUID** is `0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` ("Linux swap") straight from the
partition table, synthesises a `.swap` unit into `/run/systemd/generator.late/` and links
it into `swap.target.wants/`. `/etc/fstab` is never consulted, so neutralising fstab
cannot stop it. `roles/node_baseline` masks `swap.target` — the one unit every generated
`.swap` unit is `WantedBy` — and masks the generated units as a second barrier.
So `memory.swap` reports on both halves, from `memory.swap_persistence` in the collector
(all world-readable, no root):
| fact | meaning |
|---|---|
| `swap_target_masked` | is `/etc/systemd/system/swap.target` a symlink to `/dev/null`? |
| `generated_units` | `.swap` units systemd's generators built for this boot |
| `masked_units` | `.swap` units masked in `/etc/systemd/system` |
| `unmasked_generated_units` | `generated_units` minus `masked_units` — what a reboot would bring back |
| `fstab_active_swap_lines` | uncommented `swap` lines left in `/etc/fstab` |
A node with no swap active but an unmasked generated unit and an unmasked `swap.target`
**WARNs**: it is clean now and will be swapping again after the next reboot. The check
used to blame that state on "this node has not been baselined yet", which sent the reader
back to re-run the role that had already run.
## The EFI memattr check