fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
0e92d258dd
commit
9d00a3fcd3
3 files changed
+175
-8
No files matched your search
@@ -47,7 +47,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
|
||||
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
|
||||
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
|
||||
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
|
||||
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
|
||||
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
|
||||
@@ -59,6 +59,37 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
|
||||
in task logic.
|
||||
|
||||
### Swap off is not the same as swap disabled
|
||||
|
||||
`SwapTotal == 0` only says nothing has been swapped on *yet this boot*. It is not evidence
|
||||
that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod
|
||||
nodes came back from a reboot with 977 MiB of swap active (`k3s-prod-1` `/dev/sda4`,
|
||||
`k3s-prod-3` `/dev/sda4`, `k3s-prod-2` `/dev/nvme0n1p4`) even though `roles/node_baseline`
|
||||
had run `swapoff -a` and commented their `/etc/fstab` swap line.
|
||||
|
||||
The cause is `systemd-gpt-auto-generator`. It claims any GPT partition whose partition
|
||||
**type GUID** is `0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` ("Linux swap") straight from the
|
||||
partition table, synthesises a `.swap` unit into `/run/systemd/generator.late/` and links
|
||||
it into `swap.target.wants/`. `/etc/fstab` is never consulted, so neutralising fstab
|
||||
cannot stop it. `roles/node_baseline` masks `swap.target` — the one unit every generated
|
||||
`.swap` unit is `WantedBy` — and masks the generated units as a second barrier.
|
||||
|
||||
So `memory.swap` reports on both halves, from `memory.swap_persistence` in the collector
|
||||
(all world-readable, no root):
|
||||
|
||||
| fact | meaning |
|
||||
|---|---|
|
||||
| `swap_target_masked` | is `/etc/systemd/system/swap.target` a symlink to `/dev/null`? |
|
||||
| `generated_units` | `.swap` units systemd's generators built for this boot |
|
||||
| `masked_units` | `.swap` units masked in `/etc/systemd/system` |
|
||||
| `unmasked_generated_units` | `generated_units` minus `masked_units` — what a reboot would bring back |
|
||||
| `fstab_active_swap_lines` | uncommented `swap` lines left in `/etc/fstab` |
|
||||
|
||||
A node with no swap active but an unmasked generated unit and an unmasked `swap.target`
|
||||
**WARNs**: it is clean now and will be swapping again after the next reboot. The check
|
||||
used to blame that state on "this node has not been baselined yet", which sent the reader
|
||||
back to re-run the role that had already run.
|
||||
|
||||
|
||||
## The EFI memattr check
|
||||
|
||||
|
||||
Reference in new issue
Block a user