fix(node_health): memory.swap blamed the wrong thing, and could not see the defect

The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.

The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.

The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".

Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
  [PASS] memory.swap — no swap active and none can be activated: swap.target is masked

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/ansible/roles/node_health/tasks/assess.yml

@@ -499,25 +499,76 @@
             if phys_status in ['WARN', 'FAIL'] else '')
          ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
 
-- name: Assess — swap (kubelet requires it off)
+# WHY THIS CHECK IS NOT JUST `SwapTotal == 0`. It used to be, and it read the answer
+# wrong in both directions. It called active swap "this node has not been baselined yet",
+# which was a MIS-DIAGNOSIS — measured 2026-09-07, all three prod nodes had been
+# baselined, had their fstab swap line commented out by node_baseline, and were STILL
+# swapping 977 MiB each after a reboot. Whoever read that WARN would have re-run the very
+# role that had already run, and watched swap come back at the next boot again.
+#
+# The real mechanism is systemd-gpt-auto-generator: it claims any GPT partition typed
+# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, builds a
+# .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It
+# never reads /etc/fstab, so `swapoff -a` + a commented fstab line only holds until the
+# next boot. roles/node_baseline now masks swap.target (the single unit those generated
+# units are `WantedBy`) plus the generated units themselves.
+#
+# The other direction matters just as much: SwapTotal == 0 is NOT proof that swap is
+# disabled — a node freshly `swapoff`-ed reads zero and swaps again on reboot. So this
+# check now asks both questions, and WARNs on a node that is clean right now but has an
+# unmasked generated swap unit waiting for the next boot.
+- name: Assess — swap (kubelet requires it off, and off ACROSS REBOOTS)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
       'id': 'memory.swap',
-      'title': 'Swap disabled (kubelet prerequisite)',
+      'title': 'Swap disabled, and disabled across reboots (kubelet prerequisite)',
       'status': swap_status,
       'detail': swap_detail }] }}"
   vars:
     ansible_become: false
     m: "{{ node_health_facts.memory }}"
+    p: "{{ node_health_facts.memory.swap_persistence | default({}, true) }}"
+    # Durability was actually measurable on this node (the collector could read
+    # /etc/systemd/system and the generator dirs). When false, say so rather than
+    # implying a clean bill from a failed read.
+    swap_durable_known: "{{ (p.readable | default(false)) | bool }}"
+    # Units a reboot would bring back: produced by a generator, not masked, and with
+    # swap.target itself unmasked so something can still pull them in.
+    swap_will_return: >-
+      {{ (swap_durable_known | bool)
+         and not (p.swap_target_masked | default(false) | bool)
+         and (p.unmasked_generated_units | default([], true) | length) > 0 }}
+    swap_units_returning: "{{ p.unmasked_generated_units | default([], true) | join(', ') }}"
+    swap_active_gib: "{{ (m.swap_total_kb | int / 1048576) | round(2) | string }}"
     swap_status: >-
       {{ 'SKIP' if m.swap_total_kb is not number
-         else ('PASS' if m.swap_total_kb | int == 0 else 'WARN') }}
+         else ('WARN' if (m.swap_total_kb | int) > 0
+               else ('WARN' if (swap_will_return | bool) else 'PASS')) }}
+    swap_cause: >-
+      systemd-gpt-auto-generator re-creates a .swap unit for the GPT partition typed
+      0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") at EVERY boot, straight from
+      the partition table and without reading /etc/fstab. This is NOT "the node has not
+      been baselined" — `swapoff -a` and a commented fstab line simply do not survive a
+      reboot. The fix is roles/node_baseline masking swap.target and the generated units:
+      re-run `node-baseline.yml --tags system` against this node.
     swap_detail: >-
       {{ 'swap state unreadable' if m.swap_total_kb is not number
-         else ('no swap active' if m.swap_total_kb | int == 0
-               else ((m.swap_total_kb | int / 1048576) | round(2) | string
-                     ~ ' GiB swap active — roles/node_baseline runs swapoff and neutralises
-                       the fstab entry; this node has not been baselined yet')) }}
+         else ((swap_active_gib ~ ' GiB swap active'
+                ~ (' (' ~ swap_units_returning ~ ')' if swap_units_returning else '')
+                ~ ' — ' ~ swap_cause)
+               if (m.swap_total_kb | int) > 0
+               else ('no swap active, but it would come back at the next boot via '
+                     ~ swap_units_returning ~ ' — ' ~ swap_cause
+                     if (swap_will_return | bool)
+                     else ('no swap active'
+                           ~ (' and none can be activated: swap.target is masked'
+                              if (p.swap_target_masked | default(false) | bool)
+                              else (' and no swap unit exists on this node'
+                                    if (p.generated_units | default([], true) | length) == 0
+                                    else ' and every generated swap unit is masked'))
+                           if (swap_durable_known | bool)
+                           else 'no swap active (durability NOT verified — the collector
+                                 could not read /etc/systemd/system or /run/systemd)'))) }}
 
 - name: Assess — ECC / EDAC correctable errors
   ansible.builtin.set_fact: