home-network-k3s-impl/infra/ansible/roles/memattr_survey

memattr_survey — baseline reboot survey of the EFI Memory Attributes Table lottery

Workstream: WS-TEST / WS-INFRA · Serves: R-ENV-01, R-ENV-03, R-TEST-02 Read first: infra/PROD-PROVISIONING.md §0b — the mechanism is settled there and this role does not re-derive it.

The question

§0b establishes what goes wrong: HP's R21 firmware publishes the EFI Memory Attributes Table into memory the OS classifies as plain System RAM, the boot chain overwrites it, and Linux 6.12 multiplies two unvalidated fields and memblock_reserve()s everything from the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.

Two things §0b leaves unmeasured, and this role measures:

1. Is the boot outcome predicted by where the table landed? Node 2's two observed failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes fleet-wide were at other addresses. If bad outcomes map onto a narrow address range, the causal chain closes without touching hardware or firmware.
2. Is the fault node-2-specific? Nodes 1 and 3 had been observed exactly twice each.

Why 20 rounds

Node 2's failure rate was known only to n = 3 (2 bad of 3) — a 95% Clopper–Pearson interval on p(bad) of ≈ 0.09–0.99. Against a null that loose, a handful of boots is worthless. §0b's power note: detecting elimination needs ~20 boots per arm; detecting a mere odds shift needs ~50. 20 is the floor, not a round number. memattr_survey_rounds is a variable; lowering it below 20 produces an arm that cannot answer the question it was built for, and the analysis will say so.

This is the BASELINE arm

Nothing is changed. No BIOS writes, no kernel change, no package installs, no hardware moves, no node_baseline run. The role installs nothing and configures nothing; its only write to a node is its own append-only ledger under /var/lib. Per §0b the boot chain's own allocation footprint — initrd size, module set, boot device — is part of the causal chain, so anything that perturbs it voids the arm. §6 of the generated report checks that the kernel release and cmdline were constant on every node, for exactly this reason.

Running it

# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml'

# The real thing. Reboots are OPT-IN; the interlock is deliberate.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml \
      -e memattr_survey_reboots_authorised=true'

# Re-analyse existing artifacts, no hardware touched.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml --tags analyse'

Useful overrides: -e memattr_survey_rounds=5, --limit k3s-prod-2, -e memattr_survey_hosts=<group>.

20 rounds × ~90 s ≈ 30–45 minutes. Rounds run in parallel across nodes — they are independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That parallelism depends on strategy: free in the play and on forks ≥ the host count; preflight asserts the latter. See the playbook comment for why the default linear strategy silently serialises a looped include_tasks (it was measured, not assumed).

Safety — verified, not assumed

The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That premise is re-checked on every run, because a document and a machine drift apart:

Check	Failure mode it prevents
Host is in memattr_survey_required_group (default k3s_prod)	A mistyped --limit rebooting the live dev cluster 20 times. Read from inventory membership, which a host pattern cannot fake.
systemctl is-active k3s k3s-agent kubelet containerd docker all inactive	Rebooting a node that is actually serving something.
/etc/rancher/k3s, /var/lib/rancher/k3s, /usr/local/bin/k3s absent	k3s installed but stopped — still a machine with state.
/var/log/journal exists	A volatile journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing.
memattr_survey_reboots_authorised	Reboots at all. Defaults false.

Preflight runs with any_errors_fatal: true, so a violation on any node stops the whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a false start is a rebooted production machine.

Fail-safe and resume

• Per-node isolation. tasks/round.yml wraps each round in block/rescue. A node that does not return within memattr_survey_reboot_timeout (default 600 s, ~6× the observed POST-to-SSH time) sets an abort flag; every later round is skipped for that host only and the others keep collecting. The abort is announced immediately and again in the report — which counts rounds from the artifacts, not from Ansible's host state, so even a node that goes fully unreachable (which rescue cannot catch) still appears as a short arm with a wider interval, never as a silently missing report.
• Resumable. Round numbers come from the on-node ledger, so re-running after an interruption continues where it stopped and can neither lose nor duplicate a round. The boot found on a resume is not re-appended: it is already round N of the interrupted run, and double-counting it would bias the sample toward whichever outcome survived the interruption.
• The abort is durable, not scrollback. <host>.aborted.json records the round, the failing task and the reason, so the report can say why an arm is short instead of leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every run and re-written only if that run aborts. Its absence is not proof of success — a node can be lost abruptly enough that Ansible calls it unreachable before the rescue runs — so completeness is still judged on round count, and the marker only supplies the reason when there is one.
• Analysis survives everything. It is a separate play on localhost, so it runs even if every surveyed node died. With a node down, --tags analyse is the way in: a full run stops at preflight (correctly — any_errors_fatal) on the unreachable host.

Belt and braces on the data

Two independent records, joined on boot_id:

	written by	authoritative for
<host>.rounds.jsonl	the driver, appended + fsynced on the node after each round, and mirrored to the controller after every round	round numbers
<host>.boots.json	one end-of-run pass over the persistent journal, re-deriving every retained boot from scratch	outcomes
<host>.aborted.json	the per-round rescue, when a node stops coming back	why an arm is short

The journal does not care whether the driver was alive: the boot happened, the kernel logged it, journald persisted it. So the harvest can only ever be a superset of the ledger, and a round the driver failed to append still shows up.

Reading the output

Artifacts land in infra/ansible/_artifacts/memattr-survey/ — same convention as roles/node_health and conformance/_artifacts/: generated output, .gitignored, never committed. summary.md is also printed at the end of the run.

The verdict is measured, never inferred from a log line

This is the subtlest part of the design. There are three kernel readings, not two:

kernel_read	What the kernel logged	Memory
valid	nothing at all — it parsed a table it considered valid	intact
bail-version	Unexpected EFI Memory Attributes table version <garbage>	intact — the lucky path, and it looks alarming
failed-map	Failed to map EFI Memory Attributes table @ 0x…	gone

Silence is therefore evidence of health, not a missing measurement — which is why the record carries an explicit memattr_diagnostic_present: false rather than an empty field. And per §0b the Failed to map message is emitted by efi_memattr_apply_permissions(), later than the memblock_reserve() that did the damage: it is a consequence, not the cause. So the verdict comes from MemTotal versus SMBIOS installed size, and kernel_read is recorded alongside it as an independent observation. §5a of the report cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.

Thresholds (files/memattr_probe.py): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything landing between them is reported as indeterminate and excluded from n — a new third thing deserves human eyes, not a coin flip into a bucket.

What is and is not pooled

Boots the survey did not cause are shown in the per-round table for context and excluded from every statistic: pre rows (already in the journal) and round 0 (the boot each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so pooling it would double-count prior evidence into a run whose point is fresh trials).

The statistics

files/memattr_analyse.py, pure stdlib, so it runs in the dev container as-is.

• Clopper–Pearson (exact binomial), by bisection on the exact tails rather than the Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly instead of needing a special case. A bare fraction is what this exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
• The address↔outcome question is tested three ways, because the tempting version is circular:
  • 4b determinism — was any single address seen with both outcomes? One such address falsifies "the address determines the outcome" outright.
  • 4c separation — do the ranges interleave? Reported descriptively and not dressed up as a p-value: the bad set is defined after seeing the data.
  • 4d the pre-registered prediction — §0b, written before this run, names 0xbeeb0018. Fisher exact against this run's boots only. This is the one non-circular test available.
  • 4e address → kernel reading — the fallback that has power when 4a–4d do not. If a run produces no bad boot, verdict has no variance to explain, but kernel_read does (valid vs bail-version), and it sits one step upstream of the outcome in §0b's mechanism. An address that reproduces the same parse boot after boot is the address doing causal work, even when none of those parses was catastrophic.

Files

Path	What it is
tasks/preflight.yml	Safety gate, resume point, baseline probe
tasks/rounds.yml	The loop — resume bounds and the reboot interlock
tasks/round.yml	One round: reboot → measure → append → mirror, with the per-node rescue
tasks/harvest.yml	The authoritative journal pass
tasks/analyse.yml	Controller-side analysis
files/memattr_probe.py	Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable
files/memattr_analyse.py	Clopper–Pearson, Fisher, and the report

What this cannot tell you

It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute the difference between nodes, because node 2 is simultaneously the only single-DIMM node and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled rather than broke. If more is wanted, §0b's recommendation is to swap hardware, not BIOS settings: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this role again on both. None of it changes the action — the fix is a kernel ≥ 6.13 on all three nodes, and that is not in dispute.