# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery **Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02 **Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism is settled there and this role does **not** re-derive it. ## The question §0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it, and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists. Two things §0b leaves unmeasured, and this role measures: 1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes fleet-wide were at other addresses. If bad outcomes map onto a narrow address range, the causal chain closes without touching hardware or firmware. 2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each. ## Why 20 rounds Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds` is a variable; lowering it below 20 produces an arm that cannot answer the question it was built for, and the analysis will say so. ## This is the BASELINE arm **Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware moves, no `node_baseline` run. The role installs nothing and configures nothing; its only write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's own allocation footprint — initrd size, module set, boot device — is *part of the causal chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that the kernel release and cmdline were constant on every node, for exactly this reason. ## Running it ```bash # DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s. ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ ansible-playbook playbooks/memattr-survey.yml' # The real thing. Reboots are OPT-IN; the interlock is deliberate. ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ ansible-playbook playbooks/memattr-survey.yml \ -e memattr_survey_reboots_authorised=true' # Re-analyse existing artifacts, no hardware touched. ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ ansible-playbook playbooks/memattr-survey.yml --tags analyse' ``` Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`, `-e memattr_survey_hosts=`. 20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count; preflight asserts the latter. See the playbook comment for why the default linear strategy silently serialises a looped `include_tasks` (it was measured, not assumed). ## Safety — verified, not assumed The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That premise is re-checked on **every** run, because a document and a machine drift apart: | Check | Failure mode it prevents | |---|---| | Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. | | `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. | | `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. | | `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. | | `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. | Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a false start is a rebooted production machine. ## Fail-safe and resume * **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that host only** and the others keep collecting. The abort is announced immediately and again in the report — which counts rounds from the **artifacts**, not from Ansible's host state, so even a node that goes fully unreachable (which `rescue` cannot catch) still appears as a short arm with a wider interval, never as a silently missing report. * **Resumable.** Round numbers come from the on-node ledger, so re-running after an interruption continues where it stopped and can neither lose nor duplicate a round. The boot found on a resume is *not* re-appended: it is already round N of the interrupted run, and double-counting it would bias the sample toward whichever outcome survived the interruption. * **The abort is durable, not scrollback.** `.aborted.json` records the round, the failing task and the reason, so the report can say *why* an arm is short instead of leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every run and re-written only if that run aborts. Its *absence* is not proof of success — a node can be lost abruptly enough that Ansible calls it unreachable before the rescue runs — so completeness is still judged on round count, and the marker only supplies the reason when there is one. * **Analysis survives everything.** It is a separate play on `localhost`, so it runs even if every surveyed node died. With a node down, `--tags analyse` is the way in: a full run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host. ## Belt and braces on the data Two independent records, joined on `boot_id`: | | written by | authoritative for | |---|---|---| | `.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** | | `.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** | | `.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** | The journal does not care whether the driver was alive: the boot happened, the kernel logged it, journald persisted it. So the harvest can only ever be a *superset* of the ledger, and a round the driver failed to append still shows up. ## Reading the output Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as `roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never committed. `summary.md` is also printed at the end of the run. ### The verdict is measured, never inferred from a log line This is the subtlest part of the design. There are **three** kernel readings, not two: | `kernel_read` | What the kernel logged | Memory | |---|---|---| | `valid` | **nothing at all** — it parsed a table it considered valid | intact | | `bail-version` | `Unexpected EFI Memory Attributes table version ` | intact — the *lucky* path, and it looks alarming | | `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** | Silence is therefore *evidence of health*, not a missing measurement — which is why the record carries an explicit `memattr_diagnostic_present: false` rather than an empty field. And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`, *later* than the `memblock_reserve()` that did the damage: it is a consequence, not the cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and `kernel_read` is recorded alongside it as an independent observation. §5a of the report cross-tabulates the two; if they ever disagree, that disagreement is itself the finding. Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything landing between them is reported as `indeterminate` and excluded from `n` — a new third thing deserves human eyes, not a coin flip into a bucket. ### What is and is not pooled Boots the survey did **not cause** are shown in the per-round table for context and excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so pooling it would double-count prior evidence into a run whose point is fresh trials). ### The statistics `files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is. * **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly instead of needing a special case. A bare fraction is what this exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and utterly different evidence. * **The address↔outcome question is tested three ways**, because the tempting version is circular: * **4b determinism** — was any single address seen with *both* outcomes? One such address falsifies "the address determines the outcome" outright. * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed up as a p-value: the bad set is defined after seeing the data. * **4d the pre-registered prediction** — §0b, written before this run, names `0xbeeb0018`. Fisher exact against this run's boots only. This is the one non-circular test available. * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a run produces no bad boot, `verdict` has no variance to explain, but `kernel_read` does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in §0b's mechanism. An address that reproduces the same *parse* boot after boot is the address doing causal work, even when none of those parses was catastrophic. ## Files | Path | What it is | |---|---| | `tasks/preflight.yml` | Safety gate, resume point, baseline probe | | `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock | | `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue | | `tasks/harvest.yml` | The authoritative journal pass | | `tasks/analyse.yml` | Controller-side analysis | | `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable | | `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report | ## What this cannot tell you It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute the difference between nodes, because node 2 is simultaneously the only single-DIMM node and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all three nodes, and that is not in dispute.