fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results

THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.

REFINED MECHANISM — two kernel functions were conflated:
  efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
    (signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
    reserves a modest wrong amount and costs nothing noticeable.
  efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
    'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.

THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.

STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.

SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.

SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.

Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.

Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-03 23:50:59 -05:00
1 parent b19c5db9ae
commit 8e913fdd7b
16 files changed
+2168 -76

No files matched your search

+13
View File
@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP
> ```
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
> ```
> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health
> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
> ```
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
> until `-e memattr_survey_reboots_authorised=true` is passed. See
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
can be mistyped; inventory membership cannot.
```bash
# dev — completely unchanged, no extra vars