diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 7bad2fe..5fcdeea 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -168,6 +168,58 @@ The cleanest A/B tell between the two nodes is one line of dmesg: Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. +### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost + +An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated +the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss. +**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03: + +``` +MEMATTR=0xbeeb0018 <-- the SAME address as both bad boots +[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018 <-- the SAME line +MemTotal: 15857732 kB <-- 15.1 GiB of 16 GB. FULL MEMORY. +``` + +**Two different kernel functions were being conflated:** + +- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be + **negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so + everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest + wrong amount and costs nothing noticeable. +- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints + "Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive. + +So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The +cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB +twice and 15.1 GiB once. + +**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is +disproven. The MAT address does not determine the outcome; the garbage value at it does. + +The health check has been re-based accordingly: the measured shortfall is computed **once** into a +shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer +contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot +survived it"), and FAIL requires a measured shortfall. + +### SECOND FAULT — node 2 hangs on a warm reboot, staying powered + +On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its +journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box +stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the +operator held the power button and cold-booted it by hand. + +**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended +warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a +physical visit, and `roles/node_health` cannot see a machine that is off the network. + +Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box +parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs +eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a +live availability defect. **Worth disabling on all three nodes regardless.** + +Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is +understood or out-of-band recovery exists. + ### THERE IS NO KERNEL CMDLINE WORKAROUND `efi_memattr_init()` is called from `efi_config_parse_tables()` under only @@ -337,6 +389,42 @@ Fault disappears ⇒ single-DIMM topology. ~1 hour unattended. **None of this changes the action.** The fix is already known, is not in dispute, and applies to all three nodes: kernel ≥ 6.13. +### Reboot survey — 40 boots, run 2026-09-03 + +`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via +`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in +`_artifacts/memattr-survey/`. + +| node | bad | n | p(bad) | 95% Clopper–Pearson | +|---|---:|---:|---:|---| +| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 | +| **prod-2** | — | **0** | — | **no data — hung on its first reboot** | +| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 | +| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 | + +**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged. +And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node. + +What the 40 boots *did* establish: + +- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses + across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between + nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on + nodes that only ever drew safe tickets says little about the node that drew a bad one. +- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once** + (37 boots), with both readings present in the sample. Same address, same parse, every time. Combined + with the counter-example above, the picture is: the *address* determines what the kernel finds, and the + *garbage value there* determines whether it is survivable. +- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on + **all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So + the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible + writer, but its footprint is machine-specific. + +**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy +is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times +while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host +count. It is silent and costs 3× wall clock. + ### What NOT to do - **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is diff --git a/infra/ansible/README.md b/infra/ansible/README.md index b0abb08..12bc695 100644 --- a/infra/ansible/README.md +++ b/infra/ansible/README.md @@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP > ``` > (Both take the same target-group override as the rest — see **Targeting prod** below.) +> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere): +> ``` +> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health +> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey +> ``` +> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so +> until `-e memattr_survey_reboots_authorised=true` is passed. See +> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b. + ## Re-provisioning a node (tofu -replace / fresh OS) A node rebuild wipes **everything on the host filesystem** — including host-level config that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the @@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons: `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the bring-up chain. +`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`) +for the same reason, and adds a second interlock on top: because it **reboots** its targets, the +role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern +can be mistyped; inventory membership cannot. ```bash # dev — completely unchanged, no extra vars diff --git a/infra/ansible/playbooks/memattr-survey.yml b/infra/ansible/playbooks/memattr-survey.yml new file mode 100644 index 0000000..23d1e8f --- /dev/null +++ b/infra/ansible/playbooks/memattr-survey.yml @@ -0,0 +1,137 @@ +--- +# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01, +# R-ENV-03, R-TEST-02). +# +# WHAT QUESTION THIS ANSWERS +# infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the +# EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and +# Linux 6.12 reserves everything from the table address to the top of RAM without +# validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this +# playbook measures them: +# +# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed +# failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes +# fleet-wide were at other addresses. If bad outcomes map onto a narrow address +# range, the causal chain closes with nothing touched. +# 2. Is the fault node-2-specific, or does ANY node reach the bad path given enough +# boots? Nodes 1 and 3 have each been observed exactly twice. +# +# §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate +# is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against +# a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20 +# boots per arm; detecting a mere odds shift needs ~50. +# +# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no +# package installs, no hardware moves, no node_baseline run. Its job is to establish real +# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap, +# a Fast Boot flip, a 6.13 kernel) compares against it. +# +# HOW TO RUN IT +# Dry run first — every stage except the reboots, so the pipeline is proven before 45 +# minutes of wall clock are committed: +# +# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ +# ansible-playbook playbooks/memattr-survey.yml' +# +# The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate): +# +# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ +# ansible-playbook playbooks/memattr-survey.yml \ +# -e memattr_survey_reboots_authorised=true' +# +# Fewer rounds, or one node: +# -e memattr_survey_rounds=5 --limit k3s-prod-2 +# +# RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier +# rounds are never lost or double-counted. Analysis alone, over existing artifacts: +# ansible-playbook playbooks/memattr-survey.yml --tags analyse +# +# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml. +# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is +# not a member of memattr_survey_required_group — the dev cluster is live, and a survey +# that reboots it 20 times because of a mistyped --limit is exactly the accident worth +# engineering against. +# +# HOW TO READ THE OUTPUT +# infra/ansible/_artifacts/memattr-survey/ (gitignored, regenerated every run) +# .rounds.jsonl one JSON line per round, mirrored off the node after EVERY round +# .boots.json every boot the persistent journal holds, re-derived in one pass +# at the end — the AUTHORITATIVE record +# summary.md the analysis, also printed at the end of the run +# See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot +# with NO memattr log line is a healthy third state and not a missing measurement. + +# ===================================================================================== +# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook +# before a single reboot. That is the right asymmetry: the cost of a false stop is a +# re-run, the cost of a false start is rebooting a machine that was doing something. +# ===================================================================================== +- name: memattr survey — preflight (refuse to reboot anything that is not idle prod) + hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}" + gather_facts: false + become: true + any_errors_fatal: true + tags: [survey] + tasks: + - name: Preflight and baseline probe + ansible.builtin.include_role: + name: memattr_survey + tasks_from: preflight + +# ===================================================================================== +# PLAY 2 — THE SURVEY. +# +# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only +# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others +# keep collecting. +# +# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not +# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands +# that into one include PER (host, item), appended per host: all 20 of node 1's rounds, +# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those +# blocks belongs to exactly one host, so the other two sit idle through it and the survey +# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are +# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever +# appeared in the output. `free` lets each host walk its own task list at its own pace, +# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots. +# +# Consequences worth knowing before editing: output INTERLEAVES between hosts (every +# progress line is host-tagged for exactly this reason), and `run_once` is unreliable +# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent +# directory create on every host instead. Concurrency is also capped by `forks` +# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather +# than letting a bigger fleet silently serialise again. +# ===================================================================================== +- name: memattr survey — reboot rounds and journal harvest + hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}" + gather_facts: false + become: true + strategy: free + tags: [survey] + tasks: + - name: Run the reboot rounds + ansible.builtin.include_role: + name: memattr_survey + tasks_from: rounds + + - name: Harvest every retained boot from the persistent journal + ansible.builtin.include_role: + name: memattr_survey + tasks_from: harvest + +# ===================================================================================== +# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs +# even if every surveyed node died, because the artifacts are already on disk and the +# analysis counts rounds from THOSE rather than from Ansible's host state. A node that +# vanished shows up as a short arm with a wider interval, not as a missing report. +# ===================================================================================== +- name: memattr survey — analyse + hosts: localhost + gather_facts: false + become: false + tags: [analyse] + tasks: + - name: Produce the analysis + ansible.builtin.include_role: + name: memattr_survey + tasks_from: analyse diff --git a/infra/ansible/roles/memattr_survey/README.md b/infra/ansible/roles/memattr_survey/README.md new file mode 100644 index 0000000..c8a61fe --- /dev/null +++ b/infra/ansible/roles/memattr_survey/README.md @@ -0,0 +1,201 @@ +# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery + +**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02 +**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism +is settled there and this role does **not** re-derive it. + +## The question + +§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes +Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it, +and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from +the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists. + +Two things §0b leaves unmeasured, and this role measures: + +1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed + failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes + fleet-wide were at other addresses. If bad outcomes map onto a narrow address range, + the causal chain closes without touching hardware or firmware. +2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each. + +## Why 20 rounds + +Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson +interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is +worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a +mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds` +is a variable; lowering it below 20 produces an arm that cannot answer the question it was +built for, and the analysis will say so. + +## This is the BASELINE arm + +**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware +moves, no `node_baseline` run. The role installs nothing and configures nothing; its only +write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's +own allocation footprint — initrd size, module set, boot device — is *part of the causal +chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that +the kernel release and cmdline were constant on every node, for exactly this reason. + +## Running it + +```bash +# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s. +./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ + ansible-playbook playbooks/memattr-survey.yml' + +# The real thing. Reboots are OPT-IN; the interlock is deliberate. +./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ + ansible-playbook playbooks/memattr-survey.yml \ + -e memattr_survey_reboots_authorised=true' + +# Re-analyse existing artifacts, no hardware touched. +./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ + ansible-playbook playbooks/memattr-survey.yml --tags analyse' +``` + +Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`, +`-e memattr_survey_hosts=`. + +20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are +independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That +parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count; +preflight asserts the latter. See the playbook comment for why the default linear +strategy silently serialises a looped `include_tasks` (it was measured, not assumed). + +## Safety — verified, not assumed + +The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That +premise is re-checked on **every** run, because a document and a machine drift apart: + +| Check | Failure mode it prevents | +|---|---| +| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. | +| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. | +| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. | +| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. | +| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. | + +Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the +whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a +false start is a rebooted production machine. + +## Fail-safe and resume + +* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node + that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the + observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that + host only** and the others keep collecting. The abort is announced immediately and again + in the report — which counts rounds from the **artifacts**, not from Ansible's host + state, so even a node that goes fully unreachable (which `rescue` cannot catch) still + appears as a short arm with a wider interval, never as a silently missing report. +* **Resumable.** Round numbers come from the on-node ledger, so re-running after an + interruption continues where it stopped and can neither lose nor duplicate a round. The + boot found on a resume is *not* re-appended: it is already round N of the interrupted + run, and double-counting it would bias the sample toward whichever outcome survived the + interruption. +* **The abort is durable, not scrollback.** `.aborted.json` records the round, the + failing task and the reason, so the report can say *why* an arm is short instead of + leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every + run and re-written only if that run aborts. Its *absence* is not proof of success — a + node can be lost abruptly enough that Ansible calls it unreachable before the rescue + runs — so completeness is still judged on round count, and the marker only supplies the + reason when there is one. +* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even + if every surveyed node died. With a node down, `--tags analyse` is the way in: a full + run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host. + +## Belt and braces on the data + +Two independent records, joined on `boot_id`: + +| | written by | authoritative for | +|---|---|---| +| `.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** | +| `.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** | +| `.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** | + +The journal does not care whether the driver was alive: the boot happened, the kernel +logged it, journald persisted it. So the harvest can only ever be a *superset* of the +ledger, and a round the driver failed to append still shows up. + +## Reading the output + +Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as +`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never +committed. `summary.md` is also printed at the end of the run. + +### The verdict is measured, never inferred from a log line + +This is the subtlest part of the design. There are **three** kernel readings, not two: + +| `kernel_read` | What the kernel logged | Memory | +|---|---|---| +| `valid` | **nothing at all** — it parsed a table it considered valid | intact | +| `bail-version` | `Unexpected EFI Memory Attributes table version ` | intact — the *lucky* path, and it looks alarming | +| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** | + +Silence is therefore *evidence of health*, not a missing measurement — which is why the +record carries an explicit `memattr_diagnostic_present: false` rather than an empty field. +And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`, +*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the +cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and +`kernel_read` is recorded alongside it as an independent observation. §5a of the report +cross-tabulates the two; if they ever disagree, that disagreement is itself the finding. + +Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed +clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything +landing between them is reported as `indeterminate` and excluded from `n` — a new third +thing deserves human eyes, not a coin flip into a bucket. + +### What is and is not pooled + +Boots the survey did **not cause** are shown in the per-round table for context and +excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot +each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so +pooling it would double-count prior evidence into a run whose point is fresh trials). + +### The statistics + +`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is. + +* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the + Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly + instead of needing a special case. A bare fraction is what this exercise exists to stop + reporting: 2/3 and 13/20 are the same fraction and utterly different evidence. +* **The address↔outcome question is tested three ways**, because the tempting version is + circular: + * **4b determinism** — was any single address seen with *both* outcomes? One such + address falsifies "the address determines the outcome" outright. + * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed + up as a p-value: the bad set is defined after seeing the data. + * **4d the pre-registered prediction** — §0b, written before this run, names + `0xbeeb0018`. Fisher exact against this run's boots only. This is the one + non-circular test available. + * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a + run produces no bad boot, `verdict` has no variance to explain, but `kernel_read` + does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in + §0b's mechanism. An address that reproduces the same *parse* boot after boot is the + address doing causal work, even when none of those parses was catastrophic. + +## Files + +| Path | What it is | +|---|---| +| `tasks/preflight.yml` | Safety gate, resume point, baseline probe | +| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock | +| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue | +| `tasks/harvest.yml` | The authoritative journal pass | +| `tasks/analyse.yml` | Controller-side analysis | +| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable | +| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report | + +## What this cannot tell you + +It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute +the difference between nodes, because node 2 is simultaneously the only single-DIMM node +and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled +rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS +settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this +role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all +three nodes, and that is not in dispute. diff --git a/infra/ansible/roles/memattr_survey/defaults/main.yml b/infra/ansible/roles/memattr_survey/defaults/main.yml new file mode 100644 index 0000000..32f49e0 --- /dev/null +++ b/infra/ansible/roles/memattr_survey/defaults/main.yml @@ -0,0 +1,92 @@ +--- +# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery +# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is +# settled and is NOT re-derived here. +# +# THE QUESTION THIS ROLE ANSWERS +# §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI +# Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured: +# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's +# observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed +# successes fleet-wide were at other addresses. If bad outcomes map onto a narrow +# address range, the causal chain is closed with no hardware or firmware touched. +# 2. Is the fault node-2-specific, or does any node reach the bad path given enough +# boots? Nodes 1 and 3 have each been observed exactly twice. +# +# WHY 20 ROUNDS AND NOT FIVE +# Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson +# interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of +# boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault +# needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor, +# not a round number. +# +# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED. +# No BIOS writes, no kernel change, no package installs, no hardware moves. The role +# installs nothing and configures nothing; its only write to a node is its own append- +# only ledger under /var/lib. Anything that alters the boot chain's allocation +# footprint — and per §0b that includes the initrd, the module set and the boot device — +# changes the variable under test and voids the arm. + +# --- Sample size -------------------------------------------------------------------- +# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent +# machines), so 20 rounds costs the wall clock of 20 reboots, not 60. +memattr_survey_rounds: 20 + +# --- Safety interlock ---------------------------------------------------------------- +# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety +# preflight, a baseline probe of the current boot, the journal harvest and the analysis. +# That is a genuinely useful dry run and it is also how you prove the pipeline works +# before committing 45 minutes of wall clock to it. +memattr_survey_reboots_authorised: false + +# The role refuses to run against a host that is not in this group. The prod boxes carry +# no k3s and nothing is running on them, which is the entire reason rebooting them is +# sanctioned; the dev cluster is live and must never be caught by a stray --limit. +memattr_survey_required_group: k3s_prod + +# Unit names whose presence or activity means something IS running on this node. Any hit +# aborts the whole playbook before the first reboot (see tasks/preflight.yml). +memattr_survey_forbidden_units: + - k3s + - k3s-agent + - kubelet + - containerd + - docker +# Paths whose existence means k3s has been installed here, even if not currently running. +memattr_survey_forbidden_paths: + - /etc/rancher/k3s + - /var/lib/rancher/k3s + - /usr/local/bin/k3s + +# --- Reboot behaviour ----------------------------------------------------------------- +# A node that does not answer within this is declared lost: the role stops rebooting THAT +# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the +# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a +# slow boot. +memattr_survey_reboot_timeout: 600 +memattr_survey_connect_timeout: 20 +# Let journald settle before reading the new boot's kernel log. The kernel log itself is +# complete long before SSH is up; this is purely margin. +memattr_survey_post_reboot_delay: 15 + +# --- On-node ledger (belt) ------------------------------------------------------------- +# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if +# the controller dies. This is what makes the survey resumable: re-running skips rounds +# already recorded here rather than starting over. +memattr_survey_state_dir: /var/lib/memattr-survey +memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl" + +# --- Controller artifacts (braces) ------------------------------------------------------ +# Same convention as roles/node_health and conformance/_artifacts/: generated output under +# a `_artifacts/` directory that .gitignore excludes, never committed. +# .rounds.jsonl the per-round ledger, re-fetched AFTER EVERY ROUND so a crash +# mid-survey still leaves every completed round on the controller +# .boots.json the journal harvest — every retained boot re-derived in one +# pass. THIS is the authoritative record; the ledger is the backup. +# summary.md the analysis (also printed) +memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey" +memattr_survey_write_artifacts: true + +# Print each round's one-line result as it happens. A 45-minute unattended run with no +# output is indistinguishable from a hung one. +memattr_survey_show_progress: true diff --git a/infra/ansible/roles/memattr_survey/files/memattr_analyse.py b/infra/ansible/roles/memattr_survey/files/memattr_analyse.py new file mode 100644 index 0000000..9e361df --- /dev/null +++ b/infra/ansible/roles/memattr_survey/files/memattr_analyse.py @@ -0,0 +1,521 @@ +#!/usr/bin/env python3 +"""memattr_analyse.py — turn a memattr survey's artifacts into the answer. + +Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads + .boots.json the journal harvest — authoritative, one entry per retained boot + .rounds.jsonl the per-round ledger — authoritative for ROUND NUMBERS +and joins them on boot_id. + +WHAT IT ANSWERS, in order: + + 1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated + loudly and up front, because every number below it is weaker than it looks. + + 2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing + this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and + completely different evidence. Clopper-Pearson is the exact (conservative) binomial + interval — no normal approximation, correct at 0/n and n/n where the approximation is + degenerate and where a clean arm is most likely to land. + + 3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three + ways, deliberately, because the tempting version of this test is circular: + (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such + address falsifies "the address determines the outcome" outright, and no amount + of correlation elsewhere repairs it. + (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported + as sorted min/max per outcome plus the count of good addresses falling inside + the bad range. This is descriptive; it is NOT dressed up as a p-value, because + the bad set was chosen after seeing the data. + (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts + that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots + this run observed, which is the one non-circular test available. + + 4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory + ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell. + +Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is. +""" + +from __future__ import annotations + +import argparse +import glob +import json +import math +import os +import sys +from collections import Counter, defaultdict + +# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad +# both times, while all five successes fleet-wide were at other addresses. +PREREGISTERED_BAD_ADDRESS = "0xbeeb0018" + + +# --- Exact binomial interval ---------------------------------------------------------- +def binom_tail_ge(n: int, k: int, p: float) -> float: + """P(X >= k) for X ~ Bin(n, p).""" + if k <= 0: + return 1.0 + return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1)) + + +def binom_tail_le(n: int, k: int, p: float) -> float: + """P(X <= k) for X ~ Bin(n, p).""" + if k >= n: + return 1.0 + return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1)) + + +def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]: + """Exact 95% (by default) binomial confidence interval for k successes in n trials. + + Solved by bisection on the exact tail probabilities rather than via the Beta/F + quantile identity, so it needs nothing outside the standard library and the + boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing: + lower bound solves P(X >= k | p) = alpha/2, and is 0 when k = 0 + upper bound solves P(X <= k | p) = alpha/2, and is 1 when k = n + """ + if n == 0: + return (0.0, 1.0) + + def bisect(fn, target, lo, hi): + for _ in range(200): + mid = (lo + hi) / 2 + if fn(mid) < target: + lo = mid + else: + hi = mid + return (lo + hi) / 2 + + lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0) + upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0) + return (lower, upper) + + +def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float: + """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only. + + Used ONLY for the pre-registered prediction (address == the §0b address vs outcome), + never for a bad set defined after the fact — that would be a garden of forking paths + with a p-value stapled on. + """ + n = a + b + c + d + if n == 0: + return 1.0 + row1, col1 = a + b, a + c + + def prob(x): + return ( + math.comb(row1, x) + * math.comb(n - row1, col1 - x) + / math.comb(n, col1) + ) + + observed = prob(a) + lo = max(0, col1 - (n - row1)) + hi = min(row1, col1) + # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating + # float noise in the equality case. + return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9))) + + +# --- Loading --------------------------------------------------------------------------- +def load_host(art_dir: str, host: str) -> dict: + """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers).""" + with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle: + harvest = json.load(handle) + + rounds_by_boot: dict[str, int] = {} + ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl") + ledger: list[dict] = [] + if os.path.exists(ledger_path): + with open(ledger_path, encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + ledger.append(record) + if record.get("boot_id") and isinstance(record.get("round"), int): + rounds_by_boot[record["boot_id"]] = record["round"] + + boots = [] + for boot in harvest.get("boots", []): + entry = dict(boot) + entry["round"] = rounds_by_boot.get(boot.get("boot_id")) + boots.append(entry) + boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0))) + + # Written by the per-round rescue when a node stops coming back. Its absence is not + # proof of success — a node can be lost so abruptly that Ansible calls it unreachable + # before the rescue runs — so completeness is still judged on round COUNT, and this + # only supplies the reason when one exists. + abort = None + abort_path = os.path.join(art_dir, f"{host}.aborted.json") + if os.path.exists(abort_path): + try: + with open(abort_path, encoding="utf-8") as handle: + abort = json.load(handle) + except (OSError, json.JSONDecodeError): + abort = {"reason": "abort marker present but unreadable"} + + return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort} + + +def discover_hosts(art_dir: str) -> list[str]: + return sorted( + os.path.basename(p)[: -len(".boots.json")] + for p in glob.glob(os.path.join(art_dir, "*.boots.json")) + ) + + +# --- Rendering --------------------------------------------------------------------------- +def gib(kb) -> str: + return "?" if kb is None else f"{kb / 1048576:.2f}" + + +def pct(x) -> str: + return "?" if x is None else f"{100 * x:.0f}%" + + +def render(hosts: list[dict], expected_rounds: int) -> str: + out: list[str] = [] + w = out.append + + all_boots = [b for h in hosts for b in h["boots"]] + # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately: + # * pre-existing journal boots (round None) — observed under an unknown selection + # process, and already counted once in §0b's table; + # * round 0, the boot each node was found sitting on — it is a boot nobody chose, + # but for node 2 it is literally §0b's "boot 0" row, so pooling it would double- + # count prior evidence into a run whose entire point is fresh, independent trials. + # Both are still SHOWN in the per-round table, because context is not contamination. + surveyed = [b for b in all_boots if b.get("round") not in (None, 0)] + + w("# EFI Memory Attributes Table — baseline reboot survey") + w("") + w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no") + w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this") + w("measures p(bad) per node and tests whether the outcome is predicted by the address") + w("the firmware published the table at.") + w("") + + # --- 1. Completeness ----------------------------------------------------------------- + w("## 1. Completeness — read this before any number below it") + w("") + w("| node | rounds recorded | target | journal boots harvested | short? |") + w("|---|---:|---:|---:|---|") + short_arms = [] + for h in hosts: + rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)] + n = len(rounds) + is_short = n < expected_rounds + if is_short: + short_arms.append((h["host"], n)) + w( + f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | " + f"{'**YES — arm incomplete**' if is_short else 'no'} |" + ) + w("") + if short_arms: + w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms)) + w("> Every interval below is correspondingly wider than planned.") + w("") + for h in hosts: + if h.get("abort"): + a = h["abort"] + w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of " + f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}") + w(f"> (failing task: {a.get('failed_task', 'unknown')})") + w("") + + # --- 2. Per-round table --------------------------------------------------------------- + w("## 2. Per-round results") + w("") + for h in hosts: + installed = h["harvest"].get("smbios_installed_kb") + w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)") + w("") + w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |") + w("|---:|---|---|---|---:|---:|---:|---|") + for b in h["boots"]: + rnd = b.get("round") + label = "pre" if rnd is None else str(rnd) + usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb") + w( + f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | " + f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | " + f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | " + f"{gib(usable)} | {pct(b.get('usable_fraction'))} | " + f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |" + ) + w("") + w("`round` = survey round. `pre` = a boot already in the journal before the survey;") + w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED") + w("from every statistic below. `kernel read`: `valid` = table") + w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage") + w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =") + w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from") + w("the presence of a log line.") + w("") + + # --- 3. p(bad) ------------------------------------------------------------------------- + w("## 3. p(bad) per node — 95% Clopper-Pearson") + w("") + w("| node | bad | n | p(bad) | 95% CI | reading |") + w("|---|---:|---:|---:|---|---|") + for h in hosts: + rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)] + n = len(rounds) + bad = sum(1 for b in rounds if b.get("verdict") == "bad") + lo, hi = clopper_pearson(bad, n) + if n == 0: + reading = "no data" + elif bad == 0: + reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}" + elif bad == n: + reading = f"bad every time in {n}; p(bad) at least {lo:.2f}" + else: + reading = f"p(bad) between {lo:.2f} and {hi:.2f}" + w( + f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | " + f"{lo:.2f} – {hi:.2f} | {reading} |" + ) + w("") + indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")] + if indet: + w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are") + w("> reported as `indeterminate` rather than forced into a bucket. They are") + w("> excluded from n above. Inspect them by hand.") + w("") + + # --- 4. Address vs outcome ----------------------------------------------------------- + w("## 4. Is the outcome predicted by where the table landed?") + w("") + by_addr: dict[str, Counter] = defaultdict(Counter) + addr_hosts: dict[str, set] = defaultdict(set) + for b in surveyed: + addr = b.get("memattr_address") or "none" + by_addr[addr][b.get("verdict")] += 1 + addr_hosts[addr].add(b.get("host") or "?") + + w("### 4a. Every address observed in the survey") + w("") + w("| MEMATTR | good | bad | other | nodes | verdict for this address |") + w("|---|---:|---:|---:|---|---|") + ambiguous = [] + for addr in sorted(by_addr, key=lambda a: (a == "none", a)): + counts = by_addr[addr] + good, bad = counts.get("good", 0), counts.get("bad", 0) + other = sum(v for k, v in counts.items() if k not in ("good", "bad")) + if good and bad: + note = "**BOTH — address does not determine outcome**" + ambiguous.append(addr) + elif bad: + note = "always bad" + elif good: + note = "always good" + else: + note = "—" + w( + f"| `{addr}` | {good} | {bad} | {other} | " + f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |" + ) + w("") + + w("### 4b. Determinism") + w("") + if not by_addr: + w("No surveyed boots — nothing to test.") + elif ambiguous: + w(f"**FALSIFIED for {len(ambiguous)} address(es):** " + + ", ".join(f"`{a}`" for a in ambiguous) + + " was seen with BOTH outcomes. The address alone cannot determine the outcome.") + else: + w("**Not falsified.** No address in this survey was ever seen with both outcomes —") + w("consistent with the address determining the outcome, though with this many") + w("distinct addresses most were seen only once, which makes the test weak.") + singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1) + w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)") + w("") + + w("### 4c. Separation of the address ranges") + w("") + bad_addrs = sorted({b["memattr_address"] for b in surveyed + if b.get("verdict") == "bad" and b.get("memattr_address")}) + good_addrs = sorted({b["memattr_address"] for b in surveyed + if b.get("verdict") == "good" and b.get("memattr_address")}) + if bad_addrs and good_addrs: + blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16) + inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi] + w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`") + w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`") + w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**" + + (" — the ranges interleave, so 'a narrow bad range' does not hold" + if inside else " — the ranges are disjoint")) + w("") + w("Descriptive only. The bad set was defined after seeing the data, so this is a") + w("picture, not a test.") + elif not bad_addrs: + w("No bad boot was observed in this survey, so there is no bad range to separate.") + else: + w("No good boot was observed in this survey.") + w("") + + w("### 4d. The pre-registered prediction") + w("") + w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the") + w("bad address (both prior failures were there; all five prior successes elsewhere).") + w("Tested against this survey's boots only:") + w("") + a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad") + b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good") + c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad") + d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good") + w(f"| | bad | good |") + w(f"|---|---:|---:|") + w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |") + w(f"| elsewhere | {c} | {d} |") + w("") + if a + b_ == 0: + w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the") + w("prediction had no opportunity to be right or wrong. That is itself informative:") + w("the address moves every boot, so a specific address is not a reproducible handle.") + else: + p = fisher_exact_2x2(a, b_, c, d) + w(f"Fisher exact (two-sided) p = **{p:.4g}**" + + (" — the prediction holds." if p < 0.05 and a and not b_ + else " — not significant at 0.05." if p >= 0.05 + else " — significant, but read the cell counts before believing it.")) + w("") + + w("### 4e. Does the address predict how the KERNEL READ the table?") + w("") + w("The strongest test this data can support when no bad boot occurred. `verdict` has") + w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both") + w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:") + w("the kernel reads whatever bytes happen to be at the address the firmware published.") + w("If the address determines the reading, the address is doing causal work, one step") + w("short of the catastrophic path.") + w("") + reads_by_addr: dict[str, set] = defaultdict(set) + n_by_addr: Counter = Counter() + for b in surveyed: + addr = b.get("memattr_address") or "none" + reads_by_addr[addr].add(b.get("kernel_read")) + n_by_addr[addr] += 1 + repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1} + inconsistent = {a: r for a, r in repeated.items() if len(r) > 1} + distinct_reads = {r for rs in reads_by_addr.values() for r in rs} + w(f"- distinct addresses: **{len(reads_by_addr)}**, of which " + f"**{len(repeated)}** were observed more than once") + w(f"- distinct kernel readings across the survey: " + + ", ".join(f"`{r}`" for r in sorted(distinct_reads))) + w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**") + w("") + if len(distinct_reads) < 2: + w("Only one reading occurred in the whole survey, so there is nothing to predict.") + elif inconsistent: + w("**The address does NOT determine the reading:** " + + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items())) + elif not repeated: + w("No address recurred, so the test could not be run.") + else: + w(f"**The address determines the reading, on every address observed more than once**") + w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both") + w("readings present in the sample). This is direct evidence for the causal chain") + w("§0b describes: the same address reproduces the same parse, boot after boot.") + w("It does NOT establish which addresses are catastrophic — no bad boot occurred —") + w("only that WHERE the table lands is what decides what the kernel finds there.") + w("") + w("| MEMATTR | boots | kernel reading(s) |") + w("|---|---:|---|") + for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)): + marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else "" + w(f"| `{addr}` | {n_by_addr[addr]} | " + f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |") + w("") + + # --- 5. Corroboration ------------------------------------------------------------------- + w("## 5. Corroborating evidence") + w("") + w("### 5a. Kernel's own reading vs the measured outcome") + w("") + cross: dict[tuple, int] = Counter() + for b in surveyed: + cross[(b.get("kernel_read"), b.get("verdict"))] += 1 + w("| kernel read | verdict | boots |") + w("|---|---|---:|") + for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))): + w(f"| {read} | {verdict} | {count} |") + w("") + disagreements = [ + (read, verdict, count) + for (read, verdict), count in cross.items() + if (read == "failed-map") != (verdict == "bad") + ] + if disagreements: + w("> **The log line and the measurement DISAGREE** in " + f"{sum(c for _, _, c in disagreements)} boot(s): " + + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements) + + ". That is a finding in its own right — §0b's model says `failed-map` and only") + w("> `failed-map` costs memory.") + else: + w("`failed-map` and only `failed-map` cost memory, in every boot observed —") + w("exactly as §0b's mechanism predicts, and independent of the address question.") + w("") + + w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)") + w("") + w("| verdict | NODE_DATA end address | boots |") + w("|---|---|---:|") + nd: Counter = Counter() + for b in surveyed: + line = b.get("node_data_line") or "" + end = line.split("-")[-1].rstrip("]") if "-" in line else "?" + nd[(b.get("verdict"), end)] += 1 + for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])): + w(f"| {verdict} | `{end}` | {count} |") + w("") + + # --- 6. Invariants ------------------------------------------------------------------ + w("## 6. Nothing under test changed") + w("") + w("| node | kernels seen | distinct cmdlines | SMBIOS installed |") + w("|---|---|---:|---|") + for h in hosts: + kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")}) + cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")} + w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | " + f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |") + w("") + w("More than one kernel or more than one cmdline on a node would void that node's arm.") + w("") + return "\n".join(out) + "\n" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--artifact-dir", required=True) + parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted") + parser.add_argument("--out", default=None, help="also write the report here") + args = parser.parse_args() + + names = discover_hosts(args.artifact_dir) + if not names: + print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr) + return 1 + hosts = [load_host(args.artifact_dir, name) for name in names] + report = render(hosts, args.rounds) + if args.out: + with open(args.out, "w", encoding="utf-8") as handle: + handle.write(report) + sys.stdout.write(report) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/infra/ansible/roles/memattr_survey/files/memattr_probe.py b/infra/ansible/roles/memattr_survey/files/memattr_probe.py new file mode 100644 index 0000000..aa2567e --- /dev/null +++ b/infra/ansible/roles/memattr_survey/files/memattr_probe.py @@ -0,0 +1,489 @@ +#!/usr/bin/env python3 +"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node. + +Part of roles/memattr_survey (infra/ansible). See that role's README.md for the +question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism. + +WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS + The record for one boot needs six sources cross-referenced (kernel log config-table + line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log + `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in + Jinja across six registered results is unreadable and would put the classification + logic in a template instead of in one testable place. It also has to be exactly the + same logic for a live boot and for a boot harvested out of the journal, which a + single --all-boots mode gives for free. + +MODES + (default) one JSON record for the CURRENT boot, on stdout + --all-boots one JSON object with a `boots` list: EVERY boot the persistent + journal still holds, re-derived from scratch. This is the + AUTHORITATIVE record — it survives the driver script dying mid-run. + --status {"completed_rounds": N, "records": M} from the on-node ledger, so the + playbook can resume without re-deriving anything. + + --round N stamp the record with the survey round number + --append PATH ALSO append the record to PATH as one JSON line (the on-node ledger), + created 0644 with parents. Appended with O_APPEND in one write() so a + concurrent reader never sees a half line. + +VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE + A node whose kernel reads a VALID table logs no memattr diagnostic at all and is + perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a + scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map" + — costs memory, and even that message is emitted by a LATER function than the one + that did the damage. So the verdict comes from usable RAM versus SMBIOS installed + RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If + the two ever disagree, that disagreement is itself the finding. + +EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field +this script could not read comes back as null with a reason in `notes`, never as an +empty string and never as a hard failure. A survey that aborts on one unreadable field +loses the other 19 rounds. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import time + +SCHEMA = "memattr-survey/1" + +# --- Verdict thresholds ------------------------------------------------------------- +# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code, +# reserved firmware regions and struct page overhead account for the rest); the bad path +# yields 0.15-0.17 (everything above the table address is reserved). Two orders of +# separation, so the thresholds are nowhere near either cluster. Anything landing in the +# gap is deliberately NOT called: it is a new third thing and wants human eyes. +VERDICT_GOOD_MIN_FRACTION = 0.85 +VERDICT_BAD_MAX_FRACTION = 0.60 + +# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)` +RE_MEMORY_LINE = re.compile( + r"^Memory:\s+(?P\d+)K/(?P\d+)K available\b.*?" + r"(?:(?P\d+)K reserved)?" +) +# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...` +RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P0x[0-9a-fA-F]+)") +# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` +RE_TABLE_VERSION = re.compile(r"table version\s+(?P-?\d+)") +# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell: +# on the bad path pgdat is forced immediately BELOW the table address instead of landing +# at the top of RAM. Captured because it is free and it is independent evidence. +RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P[0-9a-fx]+-[0-9a-fx]+)\]") +RE_CMDLINE = re.compile(r"^Command line:\s*(?P.*)$") +# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed` +RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P\d+)\s+(?P[kKmMgGtT]B)\s*$") + +DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024} + + +def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]: + """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent.""" + try: + proc = subprocess.run( + argv, capture_output=True, text=True, timeout=timeout, check=False + ) + except FileNotFoundError: + return 127, "", f"{argv[0]}: not found" + except subprocess.TimeoutExpired: + return 124, "", f"{' '.join(argv)}: timed out after {timeout}s" + except OSError as exc: # pragma: no cover - defensive + return 126, "", f"{' '.join(argv)}: {exc}" + return proc.returncode, proc.stdout, proc.stderr + + +def read_text(path: str) -> str | None: + try: + with open(path, "r", encoding="utf-8", errors="replace") as handle: + return handle.read() + except OSError: + return None + + +def kernel_log(boot: str) -> tuple[list[str], str | None]: + """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID). + + `-o cat` is deliberate: it yields the message VERBATIM with no host/facility + decoration, which is what the survey wants to record, and it is stable across + systemd versions. `-k` restricts to _TRANSPORT=kernel. + """ + rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180) + if rc != 0: + return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}" + return out.splitlines(), None + + +def list_boots() -> tuple[list[dict], str | None]: + """Every boot the journal still holds, oldest first. + + Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls + back to parsing the table, because a survey that cannot enumerate boots has lost + its authoritative record and must say so rather than silently reporting one boot. + """ + rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120) + if rc == 0: + try: + parsed = json.loads(out) + except json.JSONDecodeError as exc: + return [], f"--list-boots -o json unparseable: {exc}" + boots = [ + { + "index": int(entry.get("index", 0)), + "boot_id": entry.get("boot_id"), + "first_entry_usec": entry.get("first_entry"), + "last_entry_usec": entry.get("last_entry"), + } + for entry in parsed + ] + return sorted(boots, key=lambda b: b["index"]), None + + rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120) + if rc != 0: + return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}" + boots = [] + for line in out.splitlines(): + fields = line.split() + if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]): + continue # header row + boots.append( + { + "index": int(fields[0]), + "boot_id": fields[1], + "first_entry_usec": None, + "last_entry_usec": None, + } + ) + return sorted(boots, key=lambda b: b["index"]), None + + +def journal_storage() -> dict: + """Is the journal PERSISTENT? The harvest is only authoritative if it is. + + /var/log/journal existing is what makes journald write to disk; /run/log/journal is + the volatile fallback and would mean every reboot erases the record. + """ + return { + "var_log_journal": os.path.isdir("/var/log/journal"), + "run_log_journal": os.path.isdir("/run/log/journal"), + "persistent": os.path.isdir("/var/log/journal"), + } + + +def smbios_installed_kb() -> tuple[int | None, str | None]: + """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict. + + Deliberately not /proc/meminfo (that is the number under test) and not dmesg's + `Memory: x/yK` denominator (which is already post-e820 and post-reservation). + Hardware does not change between boots, so one read applies to harvested boots too. + """ + rc, out, err = run(["dmidecode", "-t", "17"], timeout=60) + if rc != 0: + return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}" + total = 0 + seen = 0 + for line in out.splitlines(): + match = RE_DMI_SIZE.match(line) + if not match: + continue # "No Module Installed" and every non-Size line + total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()] + seen += 1 + if seen == 0: + return None, "dmidecode -t 17 reported no populated memory devices" + return total, None + + +def meminfo_total_kb() -> tuple[int | None, str | None]: + text = read_text("/proc/meminfo") + if text is None: + return None, "/proc/meminfo unreadable" + for line in text.splitlines(): + if line.startswith("MemTotal:"): + return int(line.split()[1]), None + return None, "/proc/meminfo has no MemTotal" + + +def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]: + """What did the kernel make of the table? THREE outcomes, not two (§0b). + + Returns (classification, table_version_if_reported). + + no-table firmware published no MEMATTR pointer at all (the dev nodes) + valid a MEMATTR pointer AND no diagnostic => the kernel parsed a table it + considered valid. This is SILENCE AS EVIDENCE, and it is a healthy + state, not a missing measurement. + bail-version garbage version > 2 => kernel warned and bailed. Memory intact. + The LUCKY path, and it looks alarming in the log. + failed-map the catastrophic path. The message comes from + efi_memattr_apply_permissions() and is therefore PROOF the bogus + memblock_reserve() already happened upstream of it. + bogus-size a kernel >= 6.13 rejecting the table. Should never appear on 6.12; + if it does, the kernel under test changed and the arm is void. + """ + version = None + for line in diagnostics: + match = RE_TABLE_VERSION.search(line) + if match: + version = int(match.group("version")) + joined = " | ".join(diagnostics) + if "Failed to map" in joined: + return "failed-map", version + if "Corrupted EFI Memory Attributes Table" in joined: + return "bogus-size", version + if "Unexpected EFI Memory Attributes table version" in joined: + return "bail-version", version + if diagnostics: + return "diagnostic-unrecognised", version + if memattr_addr is None: + return "no-table", version + return "valid", version + + +def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]: + if usable_kb is None or not installed_kb: + return "unknown", None + fraction = usable_kb / installed_kb + if fraction >= VERDICT_GOOD_MIN_FRACTION: + return "good", fraction + if fraction <= VERDICT_BAD_MAX_FRACTION: + return "bad", fraction + return "indeterminate", fraction + + +def build_record( + *, + boot_selector: str, + boot_meta: dict | None, + installed_kb, + installed_note, + round_number, + live: bool, +) -> dict: + notes: list[str] = [] + if installed_note: + notes.append(installed_note) + + lines, log_note = kernel_log(boot_selector) + if log_note: + notes.append(log_note) + + config_table_line = None + memattr_addr = None + diagnostics: list[str] = [] + memory_line = None + node_data_line = None + cmdline = None + + for line in lines: + stripped = line.strip() + if memattr_addr is None: + match = RE_MEMATTR_ADDR.search(stripped) + if match and stripped.startswith("efi:"): + memattr_addr = match.group("addr").lower() + config_table_line = stripped + # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the + # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix + # change upstream cannot silently drop the line. + if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped: + if stripped != config_table_line: + diagnostics.append(stripped) + if memory_line is None and stripped.startswith("Memory:"): + if RE_MEMORY_LINE.match(stripped): + memory_line = stripped + if node_data_line is None: + match = RE_NODE_DATA.match(stripped) + if match: + node_data_line = stripped + if cmdline is None: + match = RE_CMDLINE.match(stripped) + if match: + cmdline = match.group("cmdline").strip() + + available_kb = present_kb = reserved_kb = None + if memory_line: + match = RE_MEMORY_LINE.match(memory_line) + available_kb = int(match.group("available")) + present_kb = int(match.group("present")) + reserved = match.group("reserved") + reserved_kb = int(reserved) if reserved else None + else: + notes.append("no `Memory: .../... available` line found in this boot's kernel log") + + kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics) + + # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the + # equivalent measurement is the kernel's own `Memory: available` figure, which + # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so + # nobody has to guess later. + mem_total_kb, meminfo_note = (None, None) + if live: + mem_total_kb, meminfo_note = meminfo_total_kb() + if meminfo_note: + notes.append(meminfo_note) + usable_kb = mem_total_kb if mem_total_kb is not None else available_kb + basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available" + verdict, fraction = derive_verdict(usable_kb, installed_kb) + + record = { + "schema": SCHEMA, + "round": round_number, + "host": os.uname().nodename, + "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "recorded_at_epoch": int(time.time()), + "live_boot": live, + "boot_index": (boot_meta or {}).get("index"), + "boot_id": (boot_meta or {}).get("boot_id"), + "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"), + "kernel_release": os.uname().release, + "kernel_cmdline": cmdline, + # --- the independent variable ------------------------------------------------- + "efi_config_table_line": config_table_line, + "memattr_address": memattr_addr, + "memattr_diagnostics": diagnostics, + "memattr_diagnostic_count": len(diagnostics), + # An explicit marker, never an empty field: "the kernel said nothing" is a + # measurement (it read a VALID table), not a gap in the data. + "memattr_diagnostic_present": bool(diagnostics), + "memattr_table_version": table_version, + "kernel_read": kernel_read, + # --- the dependent variable --------------------------------------------------- + "memory_line": memory_line, + "memory_available_kb": available_kb, + "memory_present_kb": present_kb, + "memory_reserved_kb": reserved_kb, + "mem_total_kb": mem_total_kb, + "smbios_installed_kb": installed_kb, + "usable_kb": usable_kb, + "usable_fraction": round(fraction, 5) if fraction is not None else None, + "verdict": verdict, + "verdict_basis": f"{basis} vs SMBIOS installed", + # --- corroborating evidence --------------------------------------------------- + "node_data_line": node_data_line, + "notes": notes, + } + return record + + +def append_record(path: str, record: dict) -> str | None: + """One O_APPEND write of one line, so a reader never sees a torn record.""" + try: + parent = os.path.dirname(path) or "." + os.makedirs(parent, mode=0o755, exist_ok=True) + payload = (json.dumps(record, sort_keys=True) + "\n").encode() + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644) + try: + os.write(fd, payload) + os.fsync(fd) # the point of the ledger is surviving the next reboot + finally: + os.close(fd) + except OSError as exc: + return f"could not append to {path}: {exc}" + return None + + +def ledger_status(path: str) -> dict: + """Resume support: how far did a previous run get? + + Round numbers come from the ledger itself rather than a line count, so the + baseline round 0 record and any hand-added line cannot shift the numbering. + """ + rounds: list[int] = [] + records = 0 + malformed = 0 + if os.path.exists(path): + with open(path, "r", encoding="utf-8", errors="replace") as handle: + for line in handle: + line = line.strip() + if not line: + continue + records += 1 + try: + parsed = json.loads(line) + except json.JSONDecodeError: + malformed += 1 + continue + value = parsed.get("round") + if isinstance(value, int): + rounds.append(value) + return { + "schema": SCHEMA, + "ledger": path, + "records": records, + "malformed_records": malformed, + "completed_rounds": max(rounds) if rounds else 0, + "rounds_recorded": sorted(rounds), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--round", type=int, default=None, help="survey round number to stamp") + parser.add_argument("--append", default=None, help="on-node ledger to append to") + parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot") + parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state") + args = parser.parse_args() + + if args.status: + print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True)) + return 0 + + installed_kb, installed_note = smbios_installed_kb() + boots, boots_note = list_boots() + storage = journal_storage() + + if args.all_boots: + records = [] + for boot in boots: + # Address boots by BOOT ID, not by relative index: indices shift under the + # harvester's feet if journald rotates mid-harvest, and the ID is what the + # per-round ledger recorded, so the two records join cleanly. + selector = boot["boot_id"] or str(boot["index"]) + records.append( + build_record( + boot_selector=selector, + boot_meta=boot, + installed_kb=installed_kb, + installed_note=installed_note, + round_number=None, + live=(boot["index"] == 0), + ) + ) + payload = { + "schema": SCHEMA, + "host": os.uname().nodename, + "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "journal_storage": storage, + "smbios_installed_kb": installed_kb, + "boots_listed": len(boots), + "notes": [n for n in [boots_note, installed_note] if n], + "boots": records, + } + print(json.dumps(payload, indent=2, sort_keys=True)) + return 0 + + current = next((b for b in boots if b["index"] == 0), None) + record = build_record( + boot_selector="0", + boot_meta=current, + installed_kb=installed_kb, + installed_note=installed_note, + round_number=args.round, + live=True, + ) + record["journal_storage"] = storage + if boots_note: + record["notes"].append(boots_note) + if args.append: + note = append_record(args.append, record) + if note: + record["notes"].append(note) + print(json.dumps(record, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/infra/ansible/roles/memattr_survey/tasks/analyse.yml b/infra/ansible/roles/memattr_survey/tasks/analyse.yml new file mode 100644 index 0000000..12a69c7 --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/analyse.yml @@ -0,0 +1,42 @@ +--- +# Controller-side analysis. Runs on localhost over the artifact directory, so it is +# independent of whether any surveyed node is still alive — and so it can be re-run on +# its own (`--tags analyse`) without touching hardware. +# +# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and +# Fisher belong somewhere testable, and a confidence interval assembled out of template +# filters is a confidence interval nobody will ever check. + +- name: Locate the survey artifacts + ansible.builtin.find: + paths: "{{ memattr_survey_artifact_dir }}" + patterns: "*.boots.json" + register: memattr_survey_artifacts + +- name: Report that there is nothing to analyse + ansible.builtin.debug: + msg: >- + No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a + harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true. + when: memattr_survey_artifacts.matched | int == 0 + +- name: Analyse the survey + ansible.builtin.command: + argv: + - /usr/bin/env + - python3 + - "{{ role_path }}/files/memattr_analyse.py" + - "--artifact-dir" + - "{{ memattr_survey_artifact_dir }}" + - "--rounds" + - "{{ memattr_survey_rounds }}" + - "--out" + - "{{ memattr_survey_artifact_dir }}/summary.md" + register: memattr_survey_analysis + changed_when: false + when: memattr_survey_artifacts.matched | int > 0 + +- name: Show the analysis + ansible.builtin.debug: + msg: "{{ memattr_survey_analysis.stdout.split('\n') }}" + when: memattr_survey_analysis is not skipped diff --git a/infra/ansible/roles/memattr_survey/tasks/harvest.yml b/infra/ansible/roles/memattr_survey/tasks/harvest.yml new file mode 100644 index 0000000..ba9be9e --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/harvest.yml @@ -0,0 +1,66 @@ +--- +# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from +# scratch in one pass by the same code that produced the per-round ledger. +# +# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and +# the append, that round exists on the machine but not in the ledger. The journal does not +# care whether the driver was alive — the boot happened, the kernel logged it, journald +# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis +# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and +# any boot the ledger missed still shows up. +# +# This runs even for a node that aborted mid-survey, which is exactly when it matters most. + +- name: Harvest every retained boot from the persistent journal + ansible.builtin.script: + cmd: memattr_probe.py --all-boots + executable: /usr/bin/python3 + register: memattr_survey_harvest_raw + changed_when: false # strictly a read of the journal + +- name: Parse the harvest + ansible.builtin.set_fact: + memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}" + +- name: Ensure the artifact directory exists (on the controller) + ansible.builtin.file: + path: "{{ memattr_survey_artifact_dir }}" + state: directory + mode: "0755" + delegate_to: localhost + become: false + # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and + # run_once is unreliable there. `state: directory` is idempotent, so three hosts each + # asserting the directory exists is cheaper than a conditional that might not fire. + changed_when: false + when: memattr_survey_write_artifacts | bool + +- name: Write the harvested boot record + ansible.builtin.copy: + dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json" + mode: "0644" + content: "{{ memattr_survey_harvest | to_nice_json }}" + delegate_to: localhost + become: false + changed_when: false + when: memattr_survey_write_artifacts | bool + +- name: Mirror the final ledger to the controller + ansible.builtin.fetch: + src: "{{ memattr_survey_state_file }}" + dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl" + flat: true + changed_when: false + when: memattr_survey_write_artifacts | bool + +- name: Harvest summary + ansible.builtin.debug: + msg: >- + {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots + from the journal + ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad, + {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good, + {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other) + {%- if memattr_survey_aborted | default(false) | bool %} + — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }}) + {%- endif %} diff --git a/infra/ansible/roles/memattr_survey/tasks/main.yml b/infra/ansible/roles/memattr_survey/tasks/main.yml new file mode 100644 index 0000000..def0d5c --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/main.yml @@ -0,0 +1,23 @@ +--- +# memattr_survey — see defaults/main.yml for the question, README.md for how to run and +# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures. +# +# ORDER OF OPERATIONS +# 1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no +# persistent journal; read the resume point; record the boot we found it on. +# 2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe. +# 3. harvest.yml — re-derive every retained boot from the journal (authoritative). +# +# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe +# node stops everything) while the rounds are not (one dead node stops only itself), and +# so that the controller-side analysis runs even if every node dies. This file is the +# whole-role entry point kept for completeness; the playbook calls the stages directly. + +- name: Preflight and baseline + ansible.builtin.include_tasks: preflight.yml + +- name: Run the survey rounds + ansible.builtin.include_tasks: rounds.yml + +- name: Harvest the authoritative record + ansible.builtin.include_tasks: harvest.yml diff --git a/infra/ansible/roles/memattr_survey/tasks/preflight.yml b/infra/ansible/roles/memattr_survey/tasks/preflight.yml new file mode 100644 index 0000000..cf5f044 --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/preflight.yml @@ -0,0 +1,149 @@ +--- +# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml), +# so a violation on ANY node stops the entire playbook before a single reboot — the +# correct posture when the failure mode is "rebooted a machine that was doing something". +# +# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no +# k3s and runs nothing. That premise is checked here on every run rather than trusted from +# a document, because the document and the machine drift independently. + +- name: Assert the target host is in the prod group + # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied + # command line cannot slip past: membership is read from the inventory, not the pattern. + ansible.builtin.assert: + that: + - memattr_survey_required_group in (group_names | default([])) + fail_msg: >- + REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of + '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}). + This role reboots its targets. Point it at prod, or override + memattr_survey_required_group deliberately. + quiet: true + +- name: Check whether any cluster or container runtime unit is active + # `is-active` returns non-zero for inactive/unknown units, which is the expected answer + # here, so the rc is data and not a failure. + ansible.builtin.command: + argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}" + register: memattr_survey_units + changed_when: false + failed_when: false + +- name: Check whether k3s has been installed on this node + ansible.builtin.stat: + path: "{{ item }}" + loop: "{{ memattr_survey_forbidden_paths }}" + register: memattr_survey_paths + changed_when: false + +- name: Assert nothing is running on this node + ansible.builtin.assert: + that: + - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0 + - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0 + fail_msg: >- + REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle. + units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }}; + present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}. + The survey is only sanctioned because these boxes carry no k3s and run nothing. + quiet: true + +- name: Clear any abort marker from a previous run + # A marker left by a run that has since been resumed successfully would make this run's + # report describe the LAST failure rather than this one. The marker means "this run + # aborted", so it is cleared at the start of every run and re-written only if it does. + ansible.builtin.file: + path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json" + state: absent + delegate_to: localhost + become: false + changed_when: false + when: memattr_survey_write_artifacts | bool + +- name: Assert forks allows every node to be surveyed concurrently + # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a + # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes + # a multiple of the wall clock it should. Cheap to satisfy (`-f `), so assert rather + # than discover it three hours later. + ansible.builtin.assert: + that: + - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length) + fail_msg: >- + forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets + {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with + -f {{ ansible_play_hosts_all | length }} (or higher). + quiet: true + run_once: true + +- name: Read the node's resume state and baseline boot + # One call does double duty: it reports how far a previous run got (so this one resumes) + # and, with --round 0, records the boot we found the node on. Round 0 is a free data + # point — it is a boot nobody chose, so it is not selected for by the survey. + ansible.builtin.script: + cmd: >- + memattr_probe.py --status {{ memattr_survey_state_file | quote }} + executable: /usr/bin/python3 + register: memattr_survey_status_raw + changed_when: false + +- name: Record the resume point + ansible.builtin.set_fact: + memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}" + memattr_survey_aborted: false + memattr_survey_abort_reason: "" + +- name: Assert the probe schema is the one this role understands + ansible.builtin.assert: + that: + - memattr_survey_status.schema == 'memattr-survey/1' + fail_msg: >- + Probe returned schema {{ memattr_survey_status.schema | default('') }}; + this role understands memattr-survey/1. Role and probe are out of step. + quiet: true + +- name: Probe the boot the node is currently on + # On a FIRST run this is stamped round 0 and appended: the boot the node was already + # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended, + # because that boot is already round N of the interrupted run and double-counting it + # would bias the sample toward whatever outcome happened to survive the interruption. + ansible.builtin.script: + cmd: >- + memattr_probe.py + {% if memattr_survey_status.completed_rounds | int == 0 %} + --round 0 --append {{ memattr_survey_state_file | quote }} + {% endif %} + executable: /usr/bin/python3 + register: memattr_survey_baseline_raw + # The append is the experiment's own ledger, not a change to the node under test: this + # role reports `changed` only for things that could affect a boot, and a file written + # under /var/lib long after the kernel parsed the table cannot. + changed_when: false + +- name: Parse the baseline record + ansible.builtin.set_fact: + memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}" + +- name: Assert the journal is PERSISTENT + # The end-of-run harvest is the authoritative record precisely because it re-derives + # every boot from the journal. On a volatile journal (/run/log/journal only) each + # reboot erases the evidence and the harvest would silently return one boot — a survey + # that looks complete and holds nothing. Verified, not assumed. + ansible.builtin.assert: + that: + - memattr_survey_baseline.journal_storage.persistent | bool + fail_msg: >- + {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing). + Every reboot would erase the record this survey depends on. Refusing to run. + quiet: true + +- name: Report the starting state + ansible.builtin.debug: + msg: >- + {{ inventory_hostname }}: idle, prod, persistent journal. + kernel {{ memattr_survey_baseline.kernel_release | default('?') }}, + installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB. + baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }} + kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }} + verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}. + resuming from round {{ memattr_survey_status.completed_rounds }} + of {{ memattr_survey_rounds }}. diff --git a/infra/ansible/roles/memattr_survey/tasks/round.yml b/infra/ansible/roles/memattr_survey/tasks/round.yml new file mode 100644 index 0000000..128c78b --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/round.yml @@ -0,0 +1,98 @@ +--- +# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice. +# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`. +# +# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the +# survey: on any failure this host sets memattr_survey_aborted and every later round is +# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for +# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported +# loudly here AND again in the final analysis, which counts rounds from the artifacts +# rather than from Ansible's host state — so a node that vanishes so hard that it is +# marked unreachable (which rescue cannot catch) still shows up as a short arm. + +- name: "Round {{ memattr_survey_round }} — survey step" + block: + - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return" + ansible.builtin.reboot: + reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}" + connect_timeout: "{{ memattr_survey_connect_timeout | int }}" + post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}" + msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}" + + - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger" + ansible.builtin.script: + cmd: >- + memattr_probe.py --round {{ memattr_survey_round }} + --append {{ memattr_survey_state_file | quote }} + executable: /usr/bin/python3 + register: memattr_survey_round_raw + changed_when: false # see the note in preflight.yml — the ledger is not the node + + - name: "Round {{ memattr_survey_round }} — parse the record" + ansible.builtin.set_fact: + memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}" + + - name: "Round {{ memattr_survey_round }} — result" + ansible.builtin.debug: + msg: >- + [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}] + MEMATTR={{ memattr_survey_record.memattr_address | default('none') }} + read={{ memattr_survey_record.kernel_read }} + {%- if memattr_survey_record.memattr_table_version is not none %} + (version {{ memattr_survey_record.memattr_table_version }}) + {%- endif %} + MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G + of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G + => {{ memattr_survey_record.verdict | upper }} + boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }} + when: memattr_survey_show_progress | bool + + # BRACES. The on-node ledger already survives a reboot; re-fetching it every round + # means the controller also holds every completed round if the DRIVER dies, without + # waiting for the end-of-run harvest. One tiny file per host, per-host destination + # paths, so three concurrent fetches cannot collide. + - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller" + ansible.builtin.fetch: + src: "{{ memattr_survey_state_file }}" + dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl" + flat: true + changed_when: false + when: memattr_survey_write_artifacts | bool + + rescue: + - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others" + ansible.builtin.set_fact: + memattr_survey_aborted: true + memattr_survey_abort_reason: >- + round {{ memattr_survey_round }} failed: + {{ ansible_failed_result.msg | default('unknown error') | trim }} + + # The abort must survive the run, not live only in terminal scrollback. Without + # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads + # like a targeting mistake instead of a node that failed to come back. + - name: "Round {{ memattr_survey_round }} — record the abort as an artifact" + ansible.builtin.copy: + dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json" + mode: "0644" + content: "{{ memattr_survey_abort_record | to_nice_json }}" + vars: + memattr_survey_abort_record: + schema: "memattr-survey/1" + host: "{{ inventory_hostname }}" + aborted_at_round: "{{ memattr_survey_round }}" + target_rounds: "{{ memattr_survey_rounds }}" + reason: "{{ memattr_survey_abort_reason | trim }}" + failed_task: "{{ ansible_failed_task.name | default('unknown') }}" + delegate_to: localhost + become: false + changed_when: false + when: memattr_survey_write_artifacts | bool + + - name: "Round {{ memattr_survey_round }} — announce the abort" + ansible.builtin.debug: + msg: >- + *** SURVEY ABORTED ON {{ inventory_hostname }} at round + {{ memattr_survey_round }}/{{ memattr_survey_rounds }}: + {{ memattr_survey_abort_reason }} + *** The other nodes continue. This node's arm is SHORT — the analysis will + say so, and its p(bad) interval will be correspondingly wider. diff --git a/infra/ansible/roles/memattr_survey/tasks/rounds.yml b/infra/ansible/roles/memattr_survey/tasks/rounds.yml new file mode 100644 index 0000000..015ffaf --- /dev/null +++ b/infra/ansible/roles/memattr_survey/tasks/rounds.yml @@ -0,0 +1,40 @@ +--- +# The loop itself, kept separate from round.yml so the loop bounds (which encode the +# RESUME logic) are readable in one place. +# +# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a +# re-run after an interruption continues where it stopped and can never lose or duplicate +# an earlier round. Already at or past the target count => nothing to do, and the run +# collapses to a harvest. +# +# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which +# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest, +# analysis) still runs, so a dry run exercises the entire pipeline without touching a +# single machine. + +- name: Announce the plan + ansible.builtin.debug: + msg: >- + {{ inventory_hostname }}: + {%- if not (memattr_survey_reboots_authorised | bool) %} + DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm). + {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %} + already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do. + {%- else %} + rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }} + to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots). + {%- endif %} + +- name: Survey round + ansible.builtin.include_tasks: round.yml + loop: >- + {{ range(memattr_survey_status.completed_rounds | int + 1, + memattr_survey_rounds | int + 1) | list }} + loop_control: + loop_var: memattr_survey_round + label: "{{ inventory_hostname }} round {{ memattr_survey_round }}" + # Evaluated per iteration, so the abort flag set by a rescue in one round skips every + # round after it on THIS host and no others. + when: + - memattr_survey_reboots_authorised | bool + - not (memattr_survey_aborted | bool) diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index 8b8a6bb..a67d59f 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -54,7 +54,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS | -| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) | +| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) | Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded in task logic. @@ -65,33 +65,54 @@ in task logic. `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13** -does not validate it before reserving memory from its contents — -`drivers/firmware/efi/memattr.c` multiplies two never-validated fields -(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the -table address to the top of RAM. Which of two kernel-log lines appears decides whether the -node keeps its memory, and it is **decided afresh on every boot**: +does not validate it before reserving memory from its contents. **Two different kernel +functions matter, and the check used to conflate them:** -| kernel log | meaning | verdict | +- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both + fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so + **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through + `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything + above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and + costs nothing measurable. +- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints + "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or + merely large. + +So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not* +prove memory was lost: that depends on the garbage **value** at that address on that boot, +not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03) +booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on +measured memory, never on the log line alone:** + +| kernel log | measured shortfall | verdict | |---|---|---| -| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* | -| `efi: memattr: Unexpected EFI Memory Attributes table version ` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table | -| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working | -| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS | -| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected | -| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected | +| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured | +| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not | +| `efi: memattr: Unexpected EFI Memory Attributes table version ` | (not consulted) | **WARN** — the garbage `version` read `> 2` so the kernel bailed out first: exposed, but this boot took the safe path | +| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 validated and rejected it: this *is* the fix, working | +| no memattr line, log reaches back to boot | (not consulted) | PASS | +| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected | +| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected | -The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and -differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before -it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing. +The WARN is the point. All three prod nodes publish the same corrupt table and differ only +by **luck**, boot to boot. A node whose kernel just trusted a corrupt table is not passing a +health check even when it kept its RAM. **Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from `trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and rebooting/CMOS/BIOS/USB changes only move where the table lands. -`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node -whose kernel log cannot be read is still not unmeasured — and a future memory loss with a -different cause is still caught. +### The two memory checks cannot contradict each other + +`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine +lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus +the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in +`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads +it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of +its own, so there is exactly one verdict on lost memory per host and both checks quote it. +A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a +future memory loss with a different cause is still caught. ## Output diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index 9f2c62b..d5a6f22 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -468,19 +468,27 @@ def collect_memory_devices(): # write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes # Table, and Linux < 6.13 does not validate it before reserving memory from its contents # (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s -# the product). The kernel prints exactly one of two lines, and WHICH one decides whether -# the node keeps its RAM: +# the product). The kernel prints one of two lines: # -# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read -# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW. +# "Failed to map EFI Memory Attributes table @ 0x..." the garbage version read <= 2, so +# the kernel TRUSTED the table and reserved memory from it. This is emitted later, by +# efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently +# wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove +# memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to +# PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive +# tbl_size costs nothing measurable. Whether memory actually went missing is decided +# by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB +# twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line. # "Unexpected EFI Memory Attributes table version " the garbage version read > 2, so # the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the # firmware is still publishing a corrupt table and the outcome is decided afresh by -# where the table lands on every boot. That is luck, not immunity. +# what the garbage reads on every boot. That is luck, not immunity. # -# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots -# before it lost 13.3 GiB), which is precisely why it is detected on every health run -# instead of diagnosed once. +# This collector reports the LINES and the addresses; it never rules on memory. Classifying +# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall. +# +# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every +# health run instead of diagnosed once. # # dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on # every node here). The unprivileged pass therefore records WHY it could not read, and diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 7553959..d8fcefa 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -387,18 +387,28 @@ ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available (' ~ avail_pct | string ~ '%)') }} -# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the -# one cause we have actually hit here; this one asks the blunter question — "is this -# machine using the RAM it has?" — and answers it from two sources that share NOTHING -# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some -# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml. -- name: Assess — usable memory vs the memory this machine physically has +# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ---------------------------- +# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly +# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this +# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one +# from the presence of a kernel-log line — and they duly contradicted each other on the same +# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @ +# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was +# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The +# measurement was right; the inference from the log line was wrong — the memattr block below +# has the refined mechanism. +# +# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one +# place and both checks read this fact: neither derives a shortfall of its own, and the +# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot +# disagree about whether memory was lost — there is only one verdict on that question, and +# it is this one. Thresholds and their measured baselines: defaults/main.yml. +- name: Assess — measure memory the machine has and cannot use (once, shared by two checks) ansible.builtin.set_fact: - node_health_checks: "{{ node_health_checks + [{ - 'id': 'memory.physical_total', - 'title': 'Usable memory vs installed/present memory (independent of the kernel log)', - 'status': phys_status, - 'detail': phys_detail }] }}" + node_health_mem_shortfall: "{{ { + 'zones_pct': z_pct, 'zones_status': z_status, + 'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb, + 'status': worst_status } }}" vars: ansible_become: false th: "{{ node_health_thresholds }}" @@ -430,11 +440,36 @@ # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two # SKIPs must never add up to a PASS. - phys_status: >- + worst_status: >- {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last) | default('SKIP', true) }} - # What the kernel log says, IF it was read — quoted here only as attribution for a - # shortfall this check measured on its own. The verdict above never depends on it. + +# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling +# firmware.efi_memattr names the one cause we have actually hit here; this one asks the +# blunter question — "is this machine using the RAM it has?" — and answers it from two +# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor +# losing memory for some other reason still shows up. +- name: Assess — usable memory vs the memory this machine physically has + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'memory.physical_total', + 'title': 'Usable memory vs installed/present memory (independent of the kernel log)', + 'status': phys_status, + 'detail': phys_detail }] }}" + vars: + ansible_become: false + ms: "{{ node_health_mem_shortfall }}" + z: "{{ node_health_facts.memory.zones }}" + dev: "{{ node_health_facts.memory.devices }}" + m: "{{ node_health_facts.memory }}" + # Straight from the shared measurement — this check re-derives nothing. + z_pct: "{{ ms.zones_pct }}" + inst_kb: "{{ ms.installed_kb }}" + d_pct: "{{ ms.dimm_pct }}" + phys_status: "{{ ms.status }}" + # What the kernel log says, IF it was read — quoted here only as attribution for the + # measured shortfall. The verdict above is the shared measurement and never depends on + # the log; firmware.efi_memattr reads that same verdict, so the two always agree. phys_cause: >- {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b') @@ -757,24 +792,49 @@ # what follows is only what this check needs to be read correctly. # # HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does -# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c -# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s -# the product — everything from the table address to the top of RAM. The kernel prints -# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field -# happens to read > 2: +# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS +# are involved, and conflating them is what made this check wrong until 2026-09-03: # -# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel -# TRUSTED the table. The node has lost the memory. FAIL. -# "Unexpected EFI Memory Attributes table version " version read > 2, the kernel -# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a -# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS: -# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself -# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one -# unlucky boot from losing 83% of its RAM is not passing a health check. +# efi_memattr_init() tbl_size = sizeof(*tbl) + num_entries * desc_size, then +# memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size +# is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE +# NEGATIVE: a garbage product with bit 31 set sign-extends +# through phys_addr_t and memblock_cap_size() clamps it to +# PHYS_ADDR_MAX — everything above the table is reserved. +# A garbage-but-POSITIVE tbl_size reserves a modest, wrong +# amount and costs nothing measurable. +# efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and +# THIS is what prints "Failed to map". It fails for ANY +# sufficiently wrong tbl_size — negative OR merely large. +# +# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT +# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on +# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on +# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot +# is the counter-example that falsified the old inference. +# +# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE: +# +# "Failed to map ... @ 0x..." the bug FIRED: corrupt table, trusted by the kernel. +# FAIL when the shared measurement (node_health_mem_shortfall) says memory is +# catastrophically missing — memory is lost NOW. +# WARN otherwise — this boot drew a survivable garbage value; the next may not. +# "Unexpected ... version " version read > 2, so the kernel bailed out before +# reserving anything. WARN: exposed, and this boot took the safe path. +# +# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out, +# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the +# measurement separates the last two. Never PASS on either line: the firmware is publishing +# a corrupt table and the draw is repeated on every boot. # # WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC, # so it cannot be diagnosed once. It has to be measured on every health run. # +# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact, +# node_health_mem_shortfall, computed once further up this file; this check does not measure +# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly +# one verdict on any host and both checks quote it. +# # THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected # ONLY if the kernel log was actually read AND still reaches back to early boot — every # memattr diagnostic is printed in the first fraction of a second, and the ring buffer @@ -793,11 +853,17 @@ ansible_become: false k: "{{ node_health_facts.klog }}" z: "{{ node_health_facts.memory.zones }}" + # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed + # near memory.physical_total. This check contributes the kernel-log facts only. + ms: "{{ node_health_mem_shortfall }}" ma_state: "{{ k.memattr_state | default(none) }}" + # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it + # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why + # this check and memory.physical_total can no longer disagree about lost memory. memattr_status: >- {{ 'SKIP' if not (k.available | bool) - else ('FAIL' if ma_state == 'failed_to_map' - else ('WARN' if ma_state == 'bad_version' + else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL') + else ('WARN' if ma_state in ['failed_to_map', 'bad_version'] else ('PASS' if ma_state == 'corrupt_rejected' else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }} # One remediation, quoted by both findings. NOTE what it must NOT say: there is no @@ -812,14 +878,32 @@ bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or unplugging USB only change where the table lands and may APPEAR to fix it — they are a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b. - # The symptom, measured independently of this log (see memory.physical_total). - memattr_corroboration: >- - {{ (' Measured independently of this log: MemTotal ' - ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' + # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed + # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the + # kernel log has no part in; the verdict word comes from ms.status, so it always agrees + # with memory.physical_total. When the measurement is missing it says so, and says why. + memattr_figures: >- + {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. ' - ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away - from the allocator (/proc/zoneinfo) — check memory.physical_total.') - if ((z.available | bool) and z.unmanaged_pct is number) else '' }} + ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the + allocator (/proc/zoneinfo)') + if ms.zones_pct is number + else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS + reports as installed (/proc/zoneinfo NOT read — ' + ~ (z.reason | default('no reason recorded', true)) ~ ')') + if ms.dimm_pct is number else '') }} + memattr_corroboration: >- + {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict ' + ~ ms.status ~ '): ' ~ memattr_figures + ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL' + else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN' + else ' — the normal few percent (kernel image, memmap, firmware + reservations, graphics stolen memory). NO measurable memory was lost + on this boot.'))) + if (memattr_figures | trim | length) > 0 + else (' Whether memory was lost on this boot could NOT be measured here: neither + present-vs-managed pages nor installed DIMM capacity was read — see check + memory.physical_total for the reasons.') }} # The kernel log lines themselves, verbatim. A verdict the operator cannot check # against the evidence is not much better than a wrong one. memattr_evidence: >- @@ -837,23 +921,43 @@ the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}. Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic lives only in dmesg.{{ memattr_corroboration }} + {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%} + THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log + line. The kernel failed to map the EFI Memory Attributes table at + {{ k.memattr_table_address | default('an unreported address') }}, which proves the + table was corrupt AND that this kernel trusted it far enough to memblock_reserve() + from its contents; the measurement below shows that reservation was the catastrophic + kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended + through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the + table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }} {%- elif ma_state == 'failed_to_map' -%} - THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory - Attributes table at {{ k.memattr_table_address | default('an unreported address') }} - and reserved memory from the corrupt table anyway — everything from that address - upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }} - {%- elif ma_state == 'bad_version' -%} - EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table - here too; this boot survived only because its garbage version field read - {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out - before reserving anything. The outcome is re-decided on EVERY boot: if that field - reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right - now and this node is one unlucky boot from the failure state.{{ memattr_evidence }} + THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory + loss either. The kernel failed to map the EFI Memory Attributes table at + {{ k.memattr_table_address | default('an unreported address') }}: the table was + corrupt and this kernel trusted it far enough to memblock_reserve() from its contents. + WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by + the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size + with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage + tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing + measurable and still fails the later memremap() that printed this line. This boot drew a survivable + value; the draw is repeated on EVERY boot, at the same address — this platform has + been seen to lose 13.3 GiB and to keep its full memory at one identical table address + (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }} + {%- elif ma_state == 'bad_version' -%} + EXPOSED, not immune, though this boot took the safe path. The firmware publishes a + CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage + version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the + kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on + EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever + tbl_size the garbage produces — nothing measurable if that value is positive, + everything above the table if it is negative.{{ memattr_evidence }} + {{ memattr_corroboration }} {{ memattr_fix }} {%- elif ma_state == 'corrupt_rejected' -%} the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel - validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is - intact and stays intact across reboots; no action.{{ memattr_evidence }} + validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug + cannot cost this node memory on this or any future boot; no + action.{{ memattr_evidence }}{{ memattr_corroboration }} {%- elif not (k.boot_covered | bool) -%} NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in