fix(node_health): memory.swap blamed the wrong thing, and could not see the defect

The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.

The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.

The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".

Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
  [PASS] memory.swap — no swap active and none can be activated: swap.target is masked

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -301,9 +301,94 @@ def collect_memory():
     # Firmware's own inventory of what is physically installed. Root-only; attempted
     # here anyway so the unprivileged pass records the exact reason it could not read.
     mem["devices"] = collect_memory_devices()
+    # "Is swap off right now?" and "will it still be off after a reboot?" are DIFFERENT
+    # questions, and only the second one is kubelet's actual prerequisite. See below.
+    mem["swap_persistence"] = collect_swap_persistence()
     return mem
 
 
+# --- Will swap still be off after the next reboot? -----------------------------------
+# SwapTotal=0 does not mean swap is DISABLED — it means nothing has been swapped on yet
+# this boot. Measured 2026-09-07 on all three prod nodes: swap was active again after a
+# reboot, 977 MiB each, with /etc/fstab's swap line commented out by node_baseline. The
+# cause is systemd-gpt-auto-generator, which claims any GPT partition typed
+# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") straight from the PARTITION TABLE,
+# synthesises a .swap unit into /run/systemd/generator.late/ and links it into
+# swap.target.wants/. It never reads fstab, so neutralising fstab cannot stop it.
+#
+# roles/node_baseline masks swap.target — the single unit every generated .swap unit is
+# `WantedBy` — and masks the generated units themselves as a second barrier. This
+# collects the facts that let the report tell "off, and staying off" apart from "off
+# until the next reboot", which a /proc/meminfo reading alone can never do.
+#
+# All of it is world-readable (the generator dirs are 0755 root:root), so it works in the
+# UNPRIVILEGED pass: no root, no systemctl, no parsing of systemctl output.
+def _is_systemd_mask(path):
+    """True when `path` is a systemd mask, i.e. a symlink to /dev/null."""
+    try:
+        return os.path.islink(path) and os.path.realpath(path) == "/dev/null"
+    except OSError:
+        return False
+
+
+def collect_swap_persistence():
+    out = {
+        "readable": False,
+        "swap_target_masked": None,
+        # .swap units systemd's generators built for THIS boot (gpt-auto, fstab,
+        # cryptsetup). These are the ones that would be activated again next boot.
+        "generated_units": [],
+        # .swap units masked in /etc/systemd/system — what node_baseline wrote.
+        "masked_units": [],
+        # generated MINUS masked: the units that would still come back. Empty is good.
+        "unmasked_generated_units": [],
+        "fstab_active_swap_lines": None,
+    }
+
+    generated = set()
+    for directory in (
+        "/run/systemd/generator.early",
+        "/run/systemd/generator",
+        "/run/systemd/generator.late",
+    ):
+        try:
+            names = os.listdir(directory)
+        except FileNotFoundError:
+            # A generator produced nothing. That is a real answer, not a read failure:
+            # dev's VMs have no swap partition, so gpt-auto emits no unit at all.
+            continue
+        except OSError as exc:
+            UNREADABLE.append({"path": directory, "reason": "read error: %s" % exc})
+            return out
+        for name in names:
+            if name.endswith(".swap") and os.path.isfile(os.path.join(directory, name)):
+                generated.add(name)
+
+    masked = set()
+    try:
+        for name in os.listdir("/etc/systemd/system"):
+            if name.endswith(".swap") and _is_systemd_mask("/etc/systemd/system/" + name):
+                masked.add(name)
+    except OSError as exc:
+        UNREADABLE.append({"path": "/etc/systemd/system", "reason": "read error: %s" % exc})
+        return out
+
+    out["swap_target_masked"] = _is_systemd_mask("/etc/systemd/system/swap.target")
+    out["generated_units"] = sorted(generated)
+    out["masked_units"] = sorted(masked)
+    out["unmasked_generated_units"] = sorted(generated - masked)
+    out["readable"] = True
+
+    fstab = read_text("/etc/fstab")
+    if fstab is not None:
+        out["fstab_active_swap_lines"] = sum(
+            1
+            for line in fstab.splitlines()
+            if not line.lstrip().startswith("#") and re.search(r"\sswap\s", line)
+        )
+    return out
+
+
 # --- Memory the kernel has but never hands out --------------------------------------
 # /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
 # no DMI. Per zone it reports: