fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results

THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.

REFINED MECHANISM — two kernel functions were conflated:
  efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
    (signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
    reserves a modest wrong amount and costs nothing noticeable.
  efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
    'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.

THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.

STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.

SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.

SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.

Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.

Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/PROD-PROVISIONING.md

@@ -168,6 +168,58 @@ The cleanest A/B tell between the two nodes is one line of dmesg:
 Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
 node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
 
+### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost
+
+An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated
+the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss.
+**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03:
+
+```
+MEMATTR=0xbeeb0018                                             <-- the SAME address as both bad boots
+[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018       <-- the SAME line
+MemTotal: 15857732 kB                                          <-- 15.1 GiB of 16 GB. FULL MEMORY.
+```
+
+**Two different kernel functions were being conflated:**
+
+- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be
+  **negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so
+  everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest
+  wrong amount and costs nothing noticeable.
+- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints
+  "Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive.
+
+So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The
+cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB
+twice and 15.1 GiB once.
+
+**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is
+disproven. The MAT address does not determine the outcome; the garbage value at it does.
+
+The health check has been re-based accordingly: the measured shortfall is computed **once** into a
+shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer
+contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot
+survived it"), and FAIL requires a measured shortfall.
+
+### SECOND FAULT — node 2 hangs on a warm reboot, staying powered
+
+On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its
+journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box
+stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the
+operator held the power button and cold-booted it by hand.
+
+**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended
+warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a
+physical visit, and `roles/node_health` cannot see a machine that is off the network.
+
+Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box
+parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs
+eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a
+live availability defect. **Worth disabling on all three nodes regardless.**
+
+Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is
+understood or out-of-band recovery exists.
+
 ### THERE IS NO KERNEL CMDLINE WORKAROUND
 
 `efi_memattr_init()` is called from `efi_config_parse_tables()` under only
@@ -337,6 +389,42 @@ Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.
 **None of this changes the action.** The fix is already known, is not in dispute, and applies to all
 three nodes: kernel ≥ 6.13.
 
+### Reboot survey — 40 boots, run 2026-09-03
+
+`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via
+`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in
+`_artifacts/memattr-survey/`.
+
+| node | bad | n | p(bad) | 95% Clopper–Pearson |
+|---|---:|---:|---:|---|
+| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 |
+| **prod-2** | — | **0** | — | **no data — hung on its first reboot** |
+| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 |
+| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 |
+
+**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged.
+And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node.
+
+What the 40 boots *did* establish:
+
+- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses
+  across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between
+  nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on
+  nodes that only ever drew safe tickets says little about the node that drew a bad one.
+- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once**
+  (37 boots), with both readings present in the sample. Same address, same parse, every time. Combined
+  with the counter-example above, the picture is: the *address* determines what the kernel finds, and the
+  *garbage value there* determines whether it is survivable.
+- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on
+  **all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So
+  the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible
+  writer, but its footprint is machine-specific.
+
+**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy
+is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times
+while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host
+count. It is silent and costs 3× wall clock.
+
 ### What NOT to do
 
 - **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is