Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results

Browse Source 
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.

REFINED MECHANISM — two kernel functions were conflated:
  efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
    (signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
    reserves a modest wrong amount and costs nothing noticeable.
  efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
    'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.

THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.

STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.

SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.

SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.

Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.

Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-03 23:50:59 -05:00
1 parent b19c5db9ae
commit 8e913fdd7b
16 files changed
+2168 -76
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/PROD-PROVISIONING.md
+88
View File
@@ -168,6 +168,58 @@ The cleanest A/B tell between the two nodes is one line of dmesg:
Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost
An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated
the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss.
**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03:
```
MEMATTR=0xbeeb0018 <-- the SAME address as both bad boots
[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018 <-- the SAME line
MemTotal: 15857732 kB <-- 15.1 GiB of 16 GB. FULL MEMORY.
```
**Two different kernel functions were being conflated:**
- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be
**negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so
everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest
wrong amount and costs nothing noticeable.
- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints
"Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive.
So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The
cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB
twice and 15.1 GiB once.
**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is
disproven. The MAT address does not determine the outcome; the garbage value at it does.
The health check has been re-based accordingly: the measured shortfall is computed **once** into a
shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer
contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot
survived it"), and FAIL requires a measured shortfall.
### SECOND FAULT — node 2 hangs on a warm reboot, staying powered
On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its
journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box
stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the
operator held the power button and cold-booted it by hand.
**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended
warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a
physical visit, and `roles/node_health` cannot see a machine that is off the network.
Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box
parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs
eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a
live availability defect. **Worth disabling on all three nodes regardless.**
Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is
understood or out-of-band recovery exists.
### THERE IS NO KERNEL CMDLINE WORKAROUND
`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
@@ -337,6 +389,42 @@ Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.
**None of this changes the action.** The fix is already known, is not in dispute, and applies to all
three nodes: kernel ≥ 6.13.
### Reboot survey — 40 boots, run 2026-09-03
`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via
`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in
`_artifacts/memattr-survey/`.
| node | bad | n | p(bad) | 95% Clopper–Pearson |
|---|---:|---:|---:|---|
| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 |
| **prod-2** | — | **0** | — | **no data — hung on its first reboot** |
| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 |
| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 |
**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged.
And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node.
What the 40 boots *did* establish:
- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses
across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between
nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on
nodes that only ever drew safe tickets says little about the node that drew a bad one.
- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once**
(37 boots), with both readings present in the sample. Same address, same parse, every time. Combined
with the counter-example above, the picture is: the *address* determines what the kernel finds, and the
*garbage value there* determines whether it is survivable.
- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on
**all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So
the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible
writer, but its footprint is machine-specific.
**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy
is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times
while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host
count. It is silent and costs 3× wall clock.
### What NOT to do
- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
infra/ansible/README.md
+13
View File
@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP
> ```
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
> ```
> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health
> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
> ```
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
> until `-e memattr_survey_reboots_authorised=true` is passed. See
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
can be mistyped; inventory membership cannot.
```bash
# dev — completely unchanged, no extra vars
infra/ansible/playbooks/memattr-survey.yml
+137
View File
@@ -0,0 +1,137 @@
---
# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
# R-ENV-03, R-TEST-02).
#
# WHAT QUESTION THIS ANSWERS
# infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
# EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
# Linux 6.12 reserves everything from the table address to the top of RAM without
# validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
# playbook measures them:
#
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
# failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
# fleet-wide were at other addresses. If bad outcomes map onto a narrow address
# range, the causal chain closes with nothing touched.
# 2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
# boots? Nodes 1 and 3 have each been observed exactly twice.
#
# §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
# is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
# a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
# boots per arm; detecting a mere odds shift needs ~50.
#
# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
# package installs, no hardware moves, no node_baseline run. Its job is to establish real
# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
# a Fast Boot flip, a 6.13 kernel) compares against it.
#
# HOW TO RUN IT
# Dry run first — every stage except the reboots, so the pipeline is proven before 45
# minutes of wall clock are committed:
#
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
# ansible-playbook playbooks/memattr-survey.yml'
#
# The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
#
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
# ansible-playbook playbooks/memattr-survey.yml \
# -e memattr_survey_reboots_authorised=true'
#
# Fewer rounds, or one node:
# -e memattr_survey_rounds=5 --limit k3s-prod-2
#
# RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
# rounds are never lost or double-counted. Analysis alone, over existing artifacts:
# ansible-playbook playbooks/memattr-survey.yml --tags analyse
#
# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
# engineering against.
#
# HOW TO READ THE OUTPUT
# infra/ansible/_artifacts/memattr-survey/ (gitignored, regenerated every run)
# <host>.rounds.jsonl one JSON line per round, mirrored off the node after EVERY round
# <host>.boots.json every boot the persistent journal holds, re-derived in one pass
# at the end — the AUTHORITATIVE record
# summary.md the analysis, also printed at the end of the run
# See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
# with NO memattr log line is a healthy third state and not a missing measurement.
# =====================================================================================
# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
# before a single reboot. That is the right asymmetry: the cost of a false stop is a
# re-run, the cost of a false start is rebooting a machine that was doing something.
# =====================================================================================
- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
gather_facts: false
become: true
any_errors_fatal: true
tags: [survey]
tasks:
- name: Preflight and baseline probe
ansible.builtin.include_role:
name: memattr_survey
tasks_from: preflight
# =====================================================================================
# PLAY 2 — THE SURVEY.
#
# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
# keep collecting.
#
# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
# blocks belongs to exactly one host, so the other two sit idle through it and the survey
# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
# appeared in the output. `free` lets each host walk its own task list at its own pace,
# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
#
# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
# directory create on every host instead. Concurrency is also capped by `forks`
# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
# than letting a bigger fleet silently serialise again.
# =====================================================================================
- name: memattr survey — reboot rounds and journal harvest
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
gather_facts: false
become: true
strategy: free
tags: [survey]
tasks:
- name: Run the reboot rounds
ansible.builtin.include_role:
name: memattr_survey
tasks_from: rounds
- name: Harvest every retained boot from the persistent journal
ansible.builtin.include_role:
name: memattr_survey
tasks_from: harvest
# =====================================================================================
# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
# even if every surveyed node died, because the artifacts are already on disk and the
# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
# vanished shows up as a short arm with a wider interval, not as a missing report.
# =====================================================================================
- name: memattr survey — analyse
hosts: localhost
gather_facts: false
become: false
tags: [analyse]
tasks:
- name: Produce the analysis
ansible.builtin.include_role:
name: memattr_survey
tasks_from: analyse
infra/ansible/roles/memattr_survey/README.md
+201
View File
@@ -0,0 +1,201 @@
# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
is settled there and this role does **not** re-derive it.
## The question
§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
Two things §0b leaves unmeasured, and this role measures:
1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
the causal chain closes without touching hardware or firmware.
2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
## Why 20 rounds
Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
is a variable; lowering it below 20 produces an arm that cannot answer the question it was
built for, and the analysis will say so.
## This is the BASELINE arm
**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
own allocation footprint — initrd size, module set, boot device — is *part of the causal
chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
the kernel release and cmdline were constant on every node, for exactly this reason.
## Running it
```bash
# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
ansible-playbook playbooks/memattr-survey.yml'
# The real thing. Reboots are OPT-IN; the interlock is deliberate.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
ansible-playbook playbooks/memattr-survey.yml \
-e memattr_survey_reboots_authorised=true'
# Re-analyse existing artifacts, no hardware touched.
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
ansible-playbook playbooks/memattr-survey.yml --tags analyse'
```
Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
`-e memattr_survey_hosts=<group>`.
20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
preflight asserts the latter. See the playbook comment for why the default linear
strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
## Safety — verified, not assumed
The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
premise is re-checked on **every** run, because a document and a machine drift apart:
| Check | Failure mode it prevents |
|---|---|
| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
false start is a rebooted production machine.
## Fail-safe and resume
* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
host only** and the others keep collecting. The abort is announced immediately and again
in the report — which counts rounds from the **artifacts**, not from Ansible's host
state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
appears as a short arm with a wider interval, never as a silently missing report.
* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
interruption continues where it stopped and can neither lose nor duplicate a round. The
boot found on a resume is *not* re-appended: it is already round N of the interrupted
run, and double-counting it would bias the sample toward whichever outcome survived the
interruption.
* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
failing task and the reason, so the report can say *why* an arm is short instead of
leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
run and re-written only if that run aborts. Its *absence* is not proof of success — a
node can be lost abruptly enough that Ansible calls it unreachable before the rescue
runs — so completeness is still judged on round count, and the marker only supplies the
reason when there is one.
* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
## Belt and braces on the data
Two independent records, joined on `boot_id`:
| | written by | authoritative for |
|---|---|---|
| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
The journal does not care whether the driver was alive: the boot happened, the kernel
logged it, journald persisted it. So the harvest can only ever be a *superset* of the
ledger, and a round the driver failed to append still shows up.
## Reading the output
Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
committed. `summary.md` is also printed at the end of the run.
### The verdict is measured, never inferred from a log line
This is the subtlest part of the design. There are **three** kernel readings, not two:
| `kernel_read` | What the kernel logged | Memory |
|---|---|---|
| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
Silence is therefore *evidence of health*, not a missing measurement — which is why the
record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
`kernel_read` is recorded alongside it as an independent observation. §5a of the report
cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
landing between them is reported as `indeterminate` and excluded from `n` — a new third
thing deserves human eyes, not a coin flip into a bucket.
### What is and is not pooled
Boots the survey did **not cause** are shown in the per-round table for context and
excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
pooling it would double-count prior evidence into a run whose point is fresh trials).
### The statistics
`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
instead of needing a special case. A bare fraction is what this exercise exists to stop
reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
* **The address↔outcome question is tested three ways**, because the tempting version is
circular:
* **4b determinism** — was any single address seen with *both* outcomes? One such
address falsifies "the address determines the outcome" outright.
* **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
up as a p-value: the bad set is defined after seeing the data.
* **4d the pre-registered prediction** — §0b, written before this run, names
`0xbeeb0018`. Fisher exact against this run's boots only. This is the one
non-circular test available.
* **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
§0b's mechanism. An address that reproduces the same *parse* boot after boot is the
address doing causal work, even when none of those parses was catastrophic.
## Files
| Path | What it is |
|---|---|
| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
| `tasks/harvest.yml` | The authoritative journal pass |
| `tasks/analyse.yml` | Controller-side analysis |
| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
## What this cannot tell you
It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
the difference between nodes, because node 2 is simultaneously the only single-DIMM node
and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
three nodes, and that is not in dispute.
infra/ansible/roles/memattr_survey/defaults/main.yml
+92
View File
@@ -0,0 +1,92 @@
---
# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
# settled and is NOT re-derived here.
#
# THE QUESTION THIS ROLE ANSWERS
# §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
# Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
# observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
# successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
# address range, the causal chain is closed with no hardware or firmware touched.
# 2. Is the fault node-2-specific, or does any node reach the bad path given enough
# boots? Nodes 1 and 3 have each been observed exactly twice.
#
# WHY 20 ROUNDS AND NOT FIVE
# Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
# interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
# boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
# needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
# not a round number.
#
# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
# No BIOS writes, no kernel change, no package installs, no hardware moves. The role
# installs nothing and configures nothing; its only write to a node is its own append-
# only ledger under /var/lib. Anything that alters the boot chain's allocation
# footprint — and per §0b that includes the initrd, the module set and the boot device —
# changes the variable under test and voids the arm.
# --- Sample size --------------------------------------------------------------------
# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
memattr_survey_rounds: 20
# --- Safety interlock ----------------------------------------------------------------
# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
# That is a genuinely useful dry run and it is also how you prove the pipeline works
# before committing 45 minutes of wall clock to it.
memattr_survey_reboots_authorised: false
# The role refuses to run against a host that is not in this group. The prod boxes carry
# no k3s and nothing is running on them, which is the entire reason rebooting them is
# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
memattr_survey_required_group: k3s_prod
# Unit names whose presence or activity means something IS running on this node. Any hit
# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
memattr_survey_forbidden_units:
- k3s
- k3s-agent
- kubelet
- containerd
- docker
# Paths whose existence means k3s has been installed here, even if not currently running.
memattr_survey_forbidden_paths:
- /etc/rancher/k3s
- /var/lib/rancher/k3s
- /usr/local/bin/k3s
# --- Reboot behaviour -----------------------------------------------------------------
# A node that does not answer within this is declared lost: the role stops rebooting THAT
# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
# slow boot.
memattr_survey_reboot_timeout: 600
memattr_survey_connect_timeout: 20
# Let journald settle before reading the new boot's kernel log. The kernel log itself is
# complete long before SSH is up; this is purely margin.
memattr_survey_post_reboot_delay: 15
# --- On-node ledger (belt) -------------------------------------------------------------
# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
# the controller dies. This is what makes the survey resumable: re-running skips rounds
# already recorded here rather than starting over.
memattr_survey_state_dir: /var/lib/memattr-survey
memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
# --- Controller artifacts (braces) ------------------------------------------------------
# Same convention as roles/node_health and conformance/_artifacts/: generated output under
# a `_artifacts/` directory that .gitignore excludes, never committed.
# <host>.rounds.jsonl the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
# mid-survey still leaves every completed round on the controller
# <host>.boots.json the journal harvest — every retained boot re-derived in one
# pass. THIS is the authoritative record; the ledger is the backup.
# summary.md the analysis (also printed)
memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
memattr_survey_write_artifacts: true
# Print each round's one-line result as it happens. A 45-minute unattended run with no
# output is indistinguishable from a hung one.
memattr_survey_show_progress: true
infra/ansible/roles/memattr_survey/files/memattr_analyse.py
+521
View File
@@ -0,0 +1,521 @@
#!/usr/bin/env python3
"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
<host>.boots.json the journal harvest — authoritative, one entry per retained boot
<host>.rounds.jsonl the per-round ledger — authoritative for ROUND NUMBERS
and joins them on boot_id.
WHAT IT ANSWERS, in order:
1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
loudly and up front, because every number below it is weaker than it looks.
2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
completely different evidence. Clopper-Pearson is the exact (conservative) binomial
interval — no normal approximation, correct at 0/n and n/n where the approximation is
degenerate and where a clean arm is most likely to land.
3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
ways, deliberately, because the tempting version of this test is circular:
(a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
address falsifies "the address determines the outcome" outright, and no amount
of correlation elsewhere repairs it.
(b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
as sorted min/max per outcome plus the count of good addresses falling inside
the bad range. This is descriptive; it is NOT dressed up as a p-value, because
the bad set was chosen after seeing the data.
(c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
this run observed, which is the one non-circular test available.
4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
"""
from __future__ import annotations
import argparse
import glob
import json
import math
import os
import sys
from collections import Counter, defaultdict
# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
# both times, while all five successes fleet-wide were at other addresses.
PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
# --- Exact binomial interval ----------------------------------------------------------
def binom_tail_ge(n: int, k: int, p: float) -> float:
"""P(X >= k) for X ~ Bin(n, p)."""
if k <= 0:
return 1.0
return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
def binom_tail_le(n: int, k: int, p: float) -> float:
"""P(X <= k) for X ~ Bin(n, p)."""
if k >= n:
return 1.0
return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
"""Exact 95% (by default) binomial confidence interval for k successes in n trials.
Solved by bisection on the exact tail probabilities rather than via the Beta/F
quantile identity, so it needs nothing outside the standard library and the
boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
lower bound solves P(X >= k | p) = alpha/2, and is 0 when k = 0
upper bound solves P(X <= k | p) = alpha/2, and is 1 when k = n
"""
if n == 0:
return (0.0, 1.0)
def bisect(fn, target, lo, hi):
for _ in range(200):
mid = (lo + hi) / 2
if fn(mid) < target:
lo = mid
else:
hi = mid
return (lo + hi) / 2
lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
return (lower, upper)
def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
"""Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
never for a bad set defined after the fact — that would be a garden of forking paths
with a p-value stapled on.
"""
n = a + b + c + d
if n == 0:
return 1.0
row1, col1 = a + b, a + c
def prob(x):
return (
math.comb(row1, x)
* math.comb(n - row1, col1 - x)
/ math.comb(n, col1)
)
observed = prob(a)
lo = max(0, col1 - (n - row1))
hi = min(row1, col1)
# 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
# float noise in the equality case.
return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
# --- Loading ---------------------------------------------------------------------------
def load_host(art_dir: str, host: str) -> dict:
"""Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
harvest = json.load(handle)
rounds_by_boot: dict[str, int] = {}
ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
ledger: list[dict] = []
if os.path.exists(ledger_path):
with open(ledger_path, encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
ledger.append(record)
if record.get("boot_id") and isinstance(record.get("round"), int):
rounds_by_boot[record["boot_id"]] = record["round"]
boots = []
for boot in harvest.get("boots", []):
entry = dict(boot)
entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
boots.append(entry)
boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
# Written by the per-round rescue when a node stops coming back. Its absence is not
# proof of success — a node can be lost so abruptly that Ansible calls it unreachable
# before the rescue runs — so completeness is still judged on round COUNT, and this
# only supplies the reason when one exists.
abort = None
abort_path = os.path.join(art_dir, f"{host}.aborted.json")
if os.path.exists(abort_path):
try:
with open(abort_path, encoding="utf-8") as handle:
abort = json.load(handle)
except (OSError, json.JSONDecodeError):
abort = {"reason": "abort marker present but unreadable"}
return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
def discover_hosts(art_dir: str) -> list[str]:
return sorted(
os.path.basename(p)[: -len(".boots.json")]
for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
)
# --- Rendering ---------------------------------------------------------------------------
def gib(kb) -> str:
return "?" if kb is None else f"{kb / 1048576:.2f}"
def pct(x) -> str:
return "?" if x is None else f"{100 * x:.0f}%"
def render(hosts: list[dict], expected_rounds: int) -> str:
out: list[str] = []
w = out.append
all_boots = [b for h in hosts for b in h["boots"]]
# Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
# * pre-existing journal boots (round None) — observed under an unknown selection
# process, and already counted once in §0b's table;
# * round 0, the boot each node was found sitting on — it is a boot nobody chose,
# but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
# count prior evidence into a run whose entire point is fresh, independent trials.
# Both are still SHOWN in the per-round table, because context is not contamination.
surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
w("# EFI Memory Attributes Table — baseline reboot survey")
w("")
w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
w("measures p(bad) per node and tests whether the outcome is predicted by the address")
w("the firmware published the table at.")
w("")
# --- 1. Completeness -----------------------------------------------------------------
w("## 1. Completeness — read this before any number below it")
w("")
w("| node | rounds recorded | target | journal boots harvested | short? |")
w("|---|---:|---:|---:|---|")
short_arms = []
for h in hosts:
rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
n = len(rounds)
is_short = n < expected_rounds
if is_short:
short_arms.append((h["host"], n))
w(
f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
f"{'**YES — arm incomplete**' if is_short else 'no'} |"
)
w("")
if short_arms:
w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
w("> Every interval below is correspondingly wider than planned.")
w("")
for h in hosts:
if h.get("abort"):
a = h["abort"]
w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
w(f"> (failing task: {a.get('failed_task', 'unknown')})")
w("")
# --- 2. Per-round table ---------------------------------------------------------------
w("## 2. Per-round results")
w("")
for h in hosts:
installed = h["harvest"].get("smbios_installed_kb")
w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
w("")
w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
w("|---:|---|---|---|---:|---:|---:|---|")
for b in h["boots"]:
rnd = b.get("round")
label = "pre" if rnd is None else str(rnd)
usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
w(
f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
)
w("")
w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
w("from every statistic below. `kernel read`: `valid` = table")
w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
w("the presence of a log line.")
w("")
# --- 3. p(bad) -------------------------------------------------------------------------
w("## 3. p(bad) per node — 95% Clopper-Pearson")
w("")
w("| node | bad | n | p(bad) | 95% CI | reading |")
w("|---|---:|---:|---:|---|---|")
for h in hosts:
rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
n = len(rounds)
bad = sum(1 for b in rounds if b.get("verdict") == "bad")
lo, hi = clopper_pearson(bad, n)
if n == 0:
reading = "no data"
elif bad == 0:
reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
elif bad == n:
reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
else:
reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
w(
f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
f"{lo:.2f} – {hi:.2f} | {reading} |"
)
w("")
indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
if indet:
w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
w("> reported as `indeterminate` rather than forced into a bucket. They are")
w("> excluded from n above. Inspect them by hand.")
w("")
# --- 4. Address vs outcome -----------------------------------------------------------
w("## 4. Is the outcome predicted by where the table landed?")
w("")
by_addr: dict[str, Counter] = defaultdict(Counter)
addr_hosts: dict[str, set] = defaultdict(set)
for b in surveyed:
addr = b.get("memattr_address") or "none"
by_addr[addr][b.get("verdict")] += 1
addr_hosts[addr].add(b.get("host") or "?")
w("### 4a. Every address observed in the survey")
w("")
w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
w("|---|---:|---:|---:|---|---|")
ambiguous = []
for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
counts = by_addr[addr]
good, bad = counts.get("good", 0), counts.get("bad", 0)
other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
if good and bad:
note = "**BOTH — address does not determine outcome**"
ambiguous.append(addr)
elif bad:
note = "always bad"
elif good:
note = "always good"
else:
note = "—"
w(
f"| `{addr}` | {good} | {bad} | {other} | "
f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
)
w("")
w("### 4b. Determinism")
w("")
if not by_addr:
w("No surveyed boots — nothing to test.")
elif ambiguous:
w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+ ", ".join(f"`{a}`" for a in ambiguous)
+ " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
else:
w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
w("consistent with the address determining the outcome, though with this many")
w("distinct addresses most were seen only once, which makes the test weak.")
singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
w("")
w("### 4c. Separation of the address ranges")
w("")
bad_addrs = sorted({b["memattr_address"] for b in surveyed
if b.get("verdict") == "bad" and b.get("memattr_address")})
good_addrs = sorted({b["memattr_address"] for b in surveyed
if b.get("verdict") == "good" and b.get("memattr_address")})
if bad_addrs and good_addrs:
blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+ (" — the ranges interleave, so 'a narrow bad range' does not hold"
if inside else " — the ranges are disjoint"))
w("")
w("Descriptive only. The bad set was defined after seeing the data, so this is a")
w("picture, not a test.")
elif not bad_addrs:
w("No bad boot was observed in this survey, so there is no bad range to separate.")
else:
w("No good boot was observed in this survey.")
w("")
w("### 4d. The pre-registered prediction")
w("")
w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
w("bad address (both prior failures were there; all five prior successes elsewhere).")
w("Tested against this survey's boots only:")
w("")
a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
w(f"| | bad | good |")
w(f"|---|---:|---:|")
w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
w(f"| elsewhere | {c} | {d} |")
w("")
if a + b_ == 0:
w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
w("prediction had no opportunity to be right or wrong. That is itself informative:")
w("the address moves every boot, so a specific address is not a reproducible handle.")
else:
p = fisher_exact_2x2(a, b_, c, d)
w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+ (" — the prediction holds." if p < 0.05 and a and not b_
else " — not significant at 0.05." if p >= 0.05
else " — significant, but read the cell counts before believing it."))
w("")
w("### 4e. Does the address predict how the KERNEL READ the table?")
w("")
w("The strongest test this data can support when no bad boot occurred. `verdict` has")
w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
w("the kernel reads whatever bytes happen to be at the address the firmware published.")
w("If the address determines the reading, the address is doing causal work, one step")
w("short of the catastrophic path.")
w("")
reads_by_addr: dict[str, set] = defaultdict(set)
n_by_addr: Counter = Counter()
for b in surveyed:
addr = b.get("memattr_address") or "none"
reads_by_addr[addr].add(b.get("kernel_read"))
n_by_addr[addr] += 1
repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
f"**{len(repeated)}** were observed more than once")
w(f"- distinct kernel readings across the survey: "
+ ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
w("")
if len(distinct_reads) < 2:
w("Only one reading occurred in the whole survey, so there is nothing to predict.")
elif inconsistent:
w("**The address does NOT determine the reading:** "
+ "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
elif not repeated:
w("No address recurred, so the test could not be run.")
else:
w(f"**The address determines the reading, on every address observed more than once**")
w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
w("readings present in the sample). This is direct evidence for the causal chain")
w("§0b describes: the same address reproduces the same parse, boot after boot.")
w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
w("only that WHERE the table lands is what decides what the kernel finds there.")
w("")
w("| MEMATTR | boots | kernel reading(s) |")
w("|---|---:|---|")
for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
w(f"| `{addr}` | {n_by_addr[addr]} | "
f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
w("")
# --- 5. Corroboration -------------------------------------------------------------------
w("## 5. Corroborating evidence")
w("")
w("### 5a. Kernel's own reading vs the measured outcome")
w("")
cross: dict[tuple, int] = Counter()
for b in surveyed:
cross[(b.get("kernel_read"), b.get("verdict"))] += 1
w("| kernel read | verdict | boots |")
w("|---|---|---:|")
for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
w(f"| {read} | {verdict} | {count} |")
w("")
disagreements = [
(read, verdict, count)
for (read, verdict), count in cross.items()
if (read == "failed-map") != (verdict == "bad")
]
if disagreements:
w("> **The log line and the measurement DISAGREE** in "
f"{sum(c for _, _, c in disagreements)} boot(s): "
+ "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+ ". That is a finding in its own right — §0b's model says `failed-map` and only")
w("> `failed-map` costs memory.")
else:
w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
w("exactly as §0b's mechanism predicts, and independent of the address question.")
w("")
w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
w("")
w("| verdict | NODE_DATA end address | boots |")
w("|---|---|---:|")
nd: Counter = Counter()
for b in surveyed:
line = b.get("node_data_line") or ""
end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
nd[(b.get("verdict"), end)] += 1
for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
w(f"| {verdict} | `{end}` | {count} |")
w("")
# --- 6. Invariants ------------------------------------------------------------------
w("## 6. Nothing under test changed")
w("")
w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
w("|---|---|---:|---|")
for h in hosts:
kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
w("")
w("More than one kernel or more than one cmdline on a node would void that node's arm.")
w("")
return "\n".join(out) + "\n"
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--artifact-dir", required=True)
parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
parser.add_argument("--out", default=None, help="also write the report here")
args = parser.parse_args()
names = discover_hosts(args.artifact_dir)
if not names:
print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
return 1
hosts = [load_host(args.artifact_dir, name) for name in names]
report = render(hosts, args.rounds)
if args.out:
with open(args.out, "w", encoding="utf-8") as handle:
handle.write(report)
sys.stdout.write(report)
return 0
if __name__ == "__main__":
sys.exit(main())
infra/ansible/roles/memattr_survey/files/memattr_probe.py
+489
View File
@@ -0,0 +1,489 @@
#!/usr/bin/env python3
"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
The record for one boot needs six sources cross-referenced (kernel log config-table
line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
`NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
Jinja across six registered results is unreadable and would put the classification
logic in a template instead of in one testable place. It also has to be exactly the
same logic for a live boot and for a boot harvested out of the journal, which a
single --all-boots mode gives for free.
MODES
(default) one JSON record for the CURRENT boot, on stdout
--all-boots one JSON object with a `boots` list: EVERY boot the persistent
journal still holds, re-derived from scratch. This is the
AUTHORITATIVE record — it survives the driver script dying mid-run.
--status {"completed_rounds": N, "records": M} from the on-node ledger, so the
playbook can resume without re-deriving anything.
--round N stamp the record with the survey round number
--append PATH ALSO append the record to PATH as one JSON line (the on-node ledger),
created 0644 with parents. Appended with O_APPEND in one write() so a
concurrent reader never sees a half line.
VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
— costs memory, and even that message is emitted by a LATER function than the one
that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
the two ever disagree, that disagreement is itself the finding.
EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
this script could not read comes back as null with a reason in `notes`, never as an
empty string and never as a hard failure. A survey that aborts on one unreadable field
loses the other 19 rounds.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import time
SCHEMA = "memattr-survey/1"
# --- Verdict thresholds -------------------------------------------------------------
# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
# reserved firmware regions and struct page overhead account for the rest); the bad path
# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
# separation, so the thresholds are nowhere near either cluster. Anything landing in the
# gap is deliberately NOT called: it is a new third thing and wants human eyes.
VERDICT_GOOD_MIN_FRACTION = 0.85
VERDICT_BAD_MAX_FRACTION = 0.60
# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
RE_MEMORY_LINE = re.compile(
r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
r"(?:(?P<reserved>\d+)K reserved)?"
)
# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
# on the bad path pgdat is forced immediately BELOW the table address instead of landing
# at the top of RAM. Captured because it is free and it is independent evidence.
RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
"""Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
try:
proc = subprocess.run(
argv, capture_output=True, text=True, timeout=timeout, check=False
)
except FileNotFoundError:
return 127, "", f"{argv[0]}: not found"
except subprocess.TimeoutExpired:
return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
except OSError as exc: # pragma: no cover - defensive
return 126, "", f"{' '.join(argv)}: {exc}"
return proc.returncode, proc.stdout, proc.stderr
def read_text(path: str) -> str | None:
try:
with open(path, "r", encoding="utf-8", errors="replace") as handle:
return handle.read()
except OSError:
return None
def kernel_log(boot: str) -> tuple[list[str], str | None]:
"""Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
`-o cat` is deliberate: it yields the message VERBATIM with no host/facility
decoration, which is what the survey wants to record, and it is stable across
systemd versions. `-k` restricts to _TRANSPORT=kernel.
"""
rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
if rc != 0:
return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
return out.splitlines(), None
def list_boots() -> tuple[list[dict], str | None]:
"""Every boot the journal still holds, oldest first.
Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
back to parsing the table, because a survey that cannot enumerate boots has lost
its authoritative record and must say so rather than silently reporting one boot.
"""
rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
if rc == 0:
try:
parsed = json.loads(out)
except json.JSONDecodeError as exc:
return [], f"--list-boots -o json unparseable: {exc}"
boots = [
{
"index": int(entry.get("index", 0)),
"boot_id": entry.get("boot_id"),
"first_entry_usec": entry.get("first_entry"),
"last_entry_usec": entry.get("last_entry"),
}
for entry in parsed
]
return sorted(boots, key=lambda b: b["index"]), None
rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
if rc != 0:
return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
boots = []
for line in out.splitlines():
fields = line.split()
if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
continue # header row
boots.append(
{
"index": int(fields[0]),
"boot_id": fields[1],
"first_entry_usec": None,
"last_entry_usec": None,
}
)
return sorted(boots, key=lambda b: b["index"]), None
def journal_storage() -> dict:
"""Is the journal PERSISTENT? The harvest is only authoritative if it is.
/var/log/journal existing is what makes journald write to disk; /run/log/journal is
the volatile fallback and would mean every reboot erases the record.
"""
return {
"var_log_journal": os.path.isdir("/var/log/journal"),
"run_log_journal": os.path.isdir("/run/log/journal"),
"persistent": os.path.isdir("/var/log/journal"),
}
def smbios_installed_kb() -> tuple[int | None, str | None]:
"""Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
`Memory: x/yK` denominator (which is already post-e820 and post-reservation).
Hardware does not change between boots, so one read applies to harvested boots too.
"""
rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
if rc != 0:
return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
total = 0
seen = 0
for line in out.splitlines():
match = RE_DMI_SIZE.match(line)
if not match:
continue # "No Module Installed" and every non-Size line
total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
seen += 1
if seen == 0:
return None, "dmidecode -t 17 reported no populated memory devices"
return total, None
def meminfo_total_kb() -> tuple[int | None, str | None]:
text = read_text("/proc/meminfo")
if text is None:
return None, "/proc/meminfo unreadable"
for line in text.splitlines():
if line.startswith("MemTotal:"):
return int(line.split()[1]), None
return None, "/proc/meminfo has no MemTotal"
def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
"""What did the kernel make of the table? THREE outcomes, not two (§0b).
Returns (classification, table_version_if_reported).
no-table firmware published no MEMATTR pointer at all (the dev nodes)
valid a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
state, not a missing measurement.
bail-version garbage version > 2 => kernel warned and bailed. Memory intact.
The LUCKY path, and it looks alarming in the log.
failed-map the catastrophic path. The message comes from
efi_memattr_apply_permissions() and is therefore PROOF the bogus
memblock_reserve() already happened upstream of it.
bogus-size a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
if it does, the kernel under test changed and the arm is void.
"""
version = None
for line in diagnostics:
match = RE_TABLE_VERSION.search(line)
if match:
version = int(match.group("version"))
joined = " | ".join(diagnostics)
if "Failed to map" in joined:
return "failed-map", version
if "Corrupted EFI Memory Attributes Table" in joined:
return "bogus-size", version
if "Unexpected EFI Memory Attributes table version" in joined:
return "bail-version", version
if diagnostics:
return "diagnostic-unrecognised", version
if memattr_addr is None:
return "no-table", version
return "valid", version
def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
if usable_kb is None or not installed_kb:
return "unknown", None
fraction = usable_kb / installed_kb
if fraction >= VERDICT_GOOD_MIN_FRACTION:
return "good", fraction
if fraction <= VERDICT_BAD_MAX_FRACTION:
return "bad", fraction
return "indeterminate", fraction
def build_record(
*,
boot_selector: str,
boot_meta: dict | None,
installed_kb,
installed_note,
round_number,
live: bool,
) -> dict:
notes: list[str] = []
if installed_note:
notes.append(installed_note)
lines, log_note = kernel_log(boot_selector)
if log_note:
notes.append(log_note)
config_table_line = None
memattr_addr = None
diagnostics: list[str] = []
memory_line = None
node_data_line = None
cmdline = None
for line in lines:
stripped = line.strip()
if memattr_addr is None:
match = RE_MEMATTR_ADDR.search(stripped)
if match and stripped.startswith("efi:"):
memattr_addr = match.group("addr").lower()
config_table_line = stripped
# Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
# kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
# change upstream cannot silently drop the line.
if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
if stripped != config_table_line:
diagnostics.append(stripped)
if memory_line is None and stripped.startswith("Memory:"):
if RE_MEMORY_LINE.match(stripped):
memory_line = stripped
if node_data_line is None:
match = RE_NODE_DATA.match(stripped)
if match:
node_data_line = stripped
if cmdline is None:
match = RE_CMDLINE.match(stripped)
if match:
cmdline = match.group("cmdline").strip()
available_kb = present_kb = reserved_kb = None
if memory_line:
match = RE_MEMORY_LINE.match(memory_line)
available_kb = int(match.group("available"))
present_kb = int(match.group("present"))
reserved = match.group("reserved")
reserved_kb = int(reserved) if reserved else None
else:
notes.append("no `Memory: .../... available` line found in this boot's kernel log")
kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
# /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
# equivalent measurement is the kernel's own `Memory: available` figure, which
# tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
# nobody has to guess later.
mem_total_kb, meminfo_note = (None, None)
if live:
mem_total_kb, meminfo_note = meminfo_total_kb()
if meminfo_note:
notes.append(meminfo_note)
usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
verdict, fraction = derive_verdict(usable_kb, installed_kb)
record = {
"schema": SCHEMA,
"round": round_number,
"host": os.uname().nodename,
"recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
"recorded_at_epoch": int(time.time()),
"live_boot": live,
"boot_index": (boot_meta or {}).get("index"),
"boot_id": (boot_meta or {}).get("boot_id"),
"boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
"kernel_release": os.uname().release,
"kernel_cmdline": cmdline,
# --- the independent variable -------------------------------------------------
"efi_config_table_line": config_table_line,
"memattr_address": memattr_addr,
"memattr_diagnostics": diagnostics,
"memattr_diagnostic_count": len(diagnostics),
# An explicit marker, never an empty field: "the kernel said nothing" is a
# measurement (it read a VALID table), not a gap in the data.
"memattr_diagnostic_present": bool(diagnostics),
"memattr_table_version": table_version,
"kernel_read": kernel_read,
# --- the dependent variable ---------------------------------------------------
"memory_line": memory_line,
"memory_available_kb": available_kb,
"memory_present_kb": present_kb,
"memory_reserved_kb": reserved_kb,
"mem_total_kb": mem_total_kb,
"smbios_installed_kb": installed_kb,
"usable_kb": usable_kb,
"usable_fraction": round(fraction, 5) if fraction is not None else None,
"verdict": verdict,
"verdict_basis": f"{basis} vs SMBIOS installed",
# --- corroborating evidence ---------------------------------------------------
"node_data_line": node_data_line,
"notes": notes,
}
return record
def append_record(path: str, record: dict) -> str | None:
"""One O_APPEND write of one line, so a reader never sees a torn record."""
try:
parent = os.path.dirname(path) or "."
os.makedirs(parent, mode=0o755, exist_ok=True)
payload = (json.dumps(record, sort_keys=True) + "\n").encode()
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
try:
os.write(fd, payload)
os.fsync(fd) # the point of the ledger is surviving the next reboot
finally:
os.close(fd)
except OSError as exc:
return f"could not append to {path}: {exc}"
return None
def ledger_status(path: str) -> dict:
"""Resume support: how far did a previous run get?
Round numbers come from the ledger itself rather than a line count, so the
baseline round 0 record and any hand-added line cannot shift the numbering.
"""
rounds: list[int] = []
records = 0
malformed = 0
if os.path.exists(path):
with open(path, "r", encoding="utf-8", errors="replace") as handle:
for line in handle:
line = line.strip()
if not line:
continue
records += 1
try:
parsed = json.loads(line)
except json.JSONDecodeError:
malformed += 1
continue
value = parsed.get("round")
if isinstance(value, int):
rounds.append(value)
return {
"schema": SCHEMA,
"ledger": path,
"records": records,
"malformed_records": malformed,
"completed_rounds": max(rounds) if rounds else 0,
"rounds_recorded": sorted(rounds),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
parser.add_argument("--append", default=None, help="on-node ledger to append to")
parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
args = parser.parse_args()
if args.status:
print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
return 0
installed_kb, installed_note = smbios_installed_kb()
boots, boots_note = list_boots()
storage = journal_storage()
if args.all_boots:
records = []
for boot in boots:
# Address boots by BOOT ID, not by relative index: indices shift under the
# harvester's feet if journald rotates mid-harvest, and the ID is what the
# per-round ledger recorded, so the two records join cleanly.
selector = boot["boot_id"] or str(boot["index"])
records.append(
build_record(
boot_selector=selector,
boot_meta=boot,
installed_kb=installed_kb,
installed_note=installed_note,
round_number=None,
live=(boot["index"] == 0),
)
)
payload = {
"schema": SCHEMA,
"host": os.uname().nodename,
"harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
"journal_storage": storage,
"smbios_installed_kb": installed_kb,
"boots_listed": len(boots),
"notes": [n for n in [boots_note, installed_note] if n],
"boots": records,
}
print(json.dumps(payload, indent=2, sort_keys=True))
return 0
current = next((b for b in boots if b["index"] == 0), None)
record = build_record(
boot_selector="0",
boot_meta=current,
installed_kb=installed_kb,
installed_note=installed_note,
round_number=args.round,
live=True,
)
record["journal_storage"] = storage
if boots_note:
record["notes"].append(boots_note)
if args.append:
note = append_record(args.append, record)
if note:
record["notes"].append(note)
print(json.dumps(record, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
infra/ansible/roles/memattr_survey/tasks/analyse.yml
+42
View File
@@ -0,0 +1,42 @@
---
# Controller-side analysis. Runs on localhost over the artifact directory, so it is
# independent of whether any surveyed node is still alive — and so it can be re-run on
# its own (`--tags analyse`) without touching hardware.
#
# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
# Fisher belong somewhere testable, and a confidence interval assembled out of template
# filters is a confidence interval nobody will ever check.
- name: Locate the survey artifacts
ansible.builtin.find:
paths: "{{ memattr_survey_artifact_dir }}"
patterns: "*.boots.json"
register: memattr_survey_artifacts
- name: Report that there is nothing to analyse
ansible.builtin.debug:
msg: >-
No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
when: memattr_survey_artifacts.matched | int == 0
- name: Analyse the survey
ansible.builtin.command:
argv:
- /usr/bin/env
- python3
- "{{ role_path }}/files/memattr_analyse.py"
- "--artifact-dir"
- "{{ memattr_survey_artifact_dir }}"
- "--rounds"
- "{{ memattr_survey_rounds }}"
- "--out"
- "{{ memattr_survey_artifact_dir }}/summary.md"
register: memattr_survey_analysis
changed_when: false
when: memattr_survey_artifacts.matched | int > 0
- name: Show the analysis
ansible.builtin.debug:
msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
when: memattr_survey_analysis is not skipped
infra/ansible/roles/memattr_survey/tasks/harvest.yml
+66
View File
@@ -0,0 +1,66 @@
---
# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
# scratch in one pass by the same code that produced the per-round ledger.
#
# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
# the append, that round exists on the machine but not in the ledger. The journal does not
# care whether the driver was alive — the boot happened, the kernel logged it, journald
# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
# any boot the ledger missed still shows up.
#
# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
- name: Harvest every retained boot from the persistent journal
ansible.builtin.script:
cmd: memattr_probe.py --all-boots
executable: /usr/bin/python3
register: memattr_survey_harvest_raw
changed_when: false # strictly a read of the journal
- name: Parse the harvest
ansible.builtin.set_fact:
memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
- name: Ensure the artifact directory exists (on the controller)
ansible.builtin.file:
path: "{{ memattr_survey_artifact_dir }}"
state: directory
mode: "0755"
delegate_to: localhost
become: false
# NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
# run_once is unreliable there. `state: directory` is idempotent, so three hosts each
# asserting the directory exists is cheaper than a conditional that might not fire.
changed_when: false
when: memattr_survey_write_artifacts | bool
- name: Write the harvested boot record
ansible.builtin.copy:
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
mode: "0644"
content: "{{ memattr_survey_harvest | to_nice_json }}"
delegate_to: localhost
become: false
changed_when: false
when: memattr_survey_write_artifacts | bool
- name: Mirror the final ledger to the controller
ansible.builtin.fetch:
src: "{{ memattr_survey_state_file }}"
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
flat: true
changed_when: false
when: memattr_survey_write_artifacts | bool
- name: Harvest summary
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
from the journal
({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
{{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
{{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
{%- if memattr_survey_aborted | default(false) | bool %}
— NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
{%- endif %}
infra/ansible/roles/memattr_survey/tasks/main.yml
+23
View File
@@ -0,0 +1,23 @@
---
# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
#
# ORDER OF OPERATIONS
# 1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
# persistent journal; read the resume point; record the boot we found it on.
# 2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
# 3. harvest.yml — re-derive every retained boot from the journal (authoritative).
#
# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
# node stops everything) while the rounds are not (one dead node stops only itself), and
# so that the controller-side analysis runs even if every node dies. This file is the
# whole-role entry point kept for completeness; the playbook calls the stages directly.
- name: Preflight and baseline
ansible.builtin.include_tasks: preflight.yml
- name: Run the survey rounds
ansible.builtin.include_tasks: rounds.yml
- name: Harvest the authoritative record
ansible.builtin.include_tasks: harvest.yml
infra/ansible/roles/memattr_survey/tasks/preflight.yml
+149
View File
@@ -0,0 +1,149 @@
---
# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
# so a violation on ANY node stops the entire playbook before a single reboot — the
# correct posture when the failure mode is "rebooted a machine that was doing something".
#
# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
# k3s and runs nothing. That premise is checked here on every run rather than trusted from
# a document, because the document and the machine drift independently.
- name: Assert the target host is in the prod group
# The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
# command line cannot slip past: membership is read from the inventory, not the pattern.
ansible.builtin.assert:
that:
- memattr_survey_required_group in (group_names | default([]))
fail_msg: >-
REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
'{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
This role reboots its targets. Point it at prod, or override
memattr_survey_required_group deliberately.
quiet: true
- name: Check whether any cluster or container runtime unit is active
# `is-active` returns non-zero for inactive/unknown units, which is the expected answer
# here, so the rc is data and not a failure.
ansible.builtin.command:
argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
register: memattr_survey_units
changed_when: false
failed_when: false
- name: Check whether k3s has been installed on this node
ansible.builtin.stat:
path: "{{ item }}"
loop: "{{ memattr_survey_forbidden_paths }}"
register: memattr_survey_paths
changed_when: false
- name: Assert nothing is running on this node
ansible.builtin.assert:
that:
- memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
- memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
fail_msg: >-
REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
The survey is only sanctioned because these boxes carry no k3s and run nothing.
quiet: true
- name: Clear any abort marker from a previous run
# A marker left by a run that has since been resumed successfully would make this run's
# report describe the LAST failure rather than this one. The marker means "this run
# aborted", so it is cleared at the start of every run and re-written only if it does.
ansible.builtin.file:
path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
state: absent
delegate_to: localhost
become: false
changed_when: false
when: memattr_survey_write_artifacts | bool
- name: Assert forks allows every node to be surveyed concurrently
# The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
# hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
# a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
# than discover it three hours later.
ansible.builtin.assert:
that:
- (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
fail_msg: >-
forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
{{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
-f {{ ansible_play_hosts_all | length }} (or higher).
quiet: true
run_once: true
- name: Read the node's resume state and baseline boot
# One call does double duty: it reports how far a previous run got (so this one resumes)
# and, with --round 0, records the boot we found the node on. Round 0 is a free data
# point — it is a boot nobody chose, so it is not selected for by the survey.
ansible.builtin.script:
cmd: >-
memattr_probe.py --status {{ memattr_survey_state_file | quote }}
executable: /usr/bin/python3
register: memattr_survey_status_raw
changed_when: false
- name: Record the resume point
ansible.builtin.set_fact:
memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
memattr_survey_aborted: false
memattr_survey_abort_reason: ""
- name: Assert the probe schema is the one this role understands
ansible.builtin.assert:
that:
- memattr_survey_status.schema == 'memattr-survey/1'
fail_msg: >-
Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
this role understands memattr-survey/1. Role and probe are out of step.
quiet: true
- name: Probe the boot the node is currently on
# On a FIRST run this is stamped round 0 and appended: the boot the node was already
# sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
# because that boot is already round N of the interrupted run and double-counting it
# would bias the sample toward whatever outcome happened to survive the interruption.
ansible.builtin.script:
cmd: >-
memattr_probe.py
{% if memattr_survey_status.completed_rounds | int == 0 %}
--round 0 --append {{ memattr_survey_state_file | quote }}
{% endif %}
executable: /usr/bin/python3
register: memattr_survey_baseline_raw
# The append is the experiment's own ledger, not a change to the node under test: this
# role reports `changed` only for things that could affect a boot, and a file written
# under /var/lib long after the kernel parsed the table cannot.
changed_when: false
- name: Parse the baseline record
ansible.builtin.set_fact:
memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
- name: Assert the journal is PERSISTENT
# The end-of-run harvest is the authoritative record precisely because it re-derives
# every boot from the journal. On a volatile journal (/run/log/journal only) each
# reboot erases the evidence and the harvest would silently return one boot — a survey
# that looks complete and holds nothing. Verified, not assumed.
ansible.builtin.assert:
that:
- memattr_survey_baseline.journal_storage.persistent | bool
fail_msg: >-
{{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
Every reboot would erase the record this survey depends on. Refusing to run.
quiet: true
- name: Report the starting state
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }}: idle, prod, persistent journal.
kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
resuming from round {{ memattr_survey_status.completed_rounds }}
of {{ memattr_survey_rounds }}.
infra/ansible/roles/memattr_survey/tasks/round.yml
+98
View File
@@ -0,0 +1,98 @@
---
# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
#
# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
# survey: on any failure this host sets memattr_survey_aborted and every later round is
# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
# loudly here AND again in the final analysis, which counts rounds from the artifacts
# rather than from Ansible's host state — so a node that vanishes so hard that it is
# marked unreachable (which rescue cannot catch) still shows up as a short arm.
- name: "Round {{ memattr_survey_round }} — survey step"
block:
- name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
ansible.builtin.reboot:
reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
- name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
ansible.builtin.script:
cmd: >-
memattr_probe.py --round {{ memattr_survey_round }}
--append {{ memattr_survey_state_file | quote }}
executable: /usr/bin/python3
register: memattr_survey_round_raw
changed_when: false # see the note in preflight.yml — the ledger is not the node
- name: "Round {{ memattr_survey_round }} — parse the record"
ansible.builtin.set_fact:
memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
- name: "Round {{ memattr_survey_round }} — result"
ansible.builtin.debug:
msg: >-
[{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
read={{ memattr_survey_record.kernel_read }}
{%- if memattr_survey_record.memattr_table_version is not none %}
(version {{ memattr_survey_record.memattr_table_version }})
{%- endif %}
MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
=> {{ memattr_survey_record.verdict | upper }}
boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
when: memattr_survey_show_progress | bool
# BRACES. The on-node ledger already survives a reboot; re-fetching it every round
# means the controller also holds every completed round if the DRIVER dies, without
# waiting for the end-of-run harvest. One tiny file per host, per-host destination
# paths, so three concurrent fetches cannot collide.
- name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
ansible.builtin.fetch:
src: "{{ memattr_survey_state_file }}"
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
flat: true
changed_when: false
when: memattr_survey_write_artifacts | bool
rescue:
- name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
ansible.builtin.set_fact:
memattr_survey_aborted: true
memattr_survey_abort_reason: >-
round {{ memattr_survey_round }} failed:
{{ ansible_failed_result.msg | default('unknown error') | trim }}
# The abort must survive the run, not live only in terminal scrollback. Without
# this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
# like a targeting mistake instead of a node that failed to come back.
- name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
ansible.builtin.copy:
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
mode: "0644"
content: "{{ memattr_survey_abort_record | to_nice_json }}"
vars:
memattr_survey_abort_record:
schema: "memattr-survey/1"
host: "{{ inventory_hostname }}"
aborted_at_round: "{{ memattr_survey_round }}"
target_rounds: "{{ memattr_survey_rounds }}"
reason: "{{ memattr_survey_abort_reason | trim }}"
failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
delegate_to: localhost
become: false
changed_when: false
when: memattr_survey_write_artifacts | bool
- name: "Round {{ memattr_survey_round }} — announce the abort"
ansible.builtin.debug:
msg: >-
*** SURVEY ABORTED ON {{ inventory_hostname }} at round
{{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
{{ memattr_survey_abort_reason }}
*** The other nodes continue. This node's arm is SHORT — the analysis will
say so, and its p(bad) interval will be correspondingly wider.
infra/ansible/roles/memattr_survey/tasks/rounds.yml
+40
View File
@@ -0,0 +1,40 @@
---
# The loop itself, kept separate from round.yml so the loop bounds (which encode the
# RESUME logic) are readable in one place.
#
# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
# re-run after an interruption continues where it stopped and can never lose or duplicate
# an earlier round. Already at or past the target count => nothing to do, and the run
# collapses to a harvest.
#
# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
# analysis) still runs, so a dry run exercises the entire pipeline without touching a
# single machine.
- name: Announce the plan
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }}:
{%- if not (memattr_survey_reboots_authorised | bool) %}
DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
{%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
{%- else %}
rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
{%- endif %}
- name: Survey round
ansible.builtin.include_tasks: round.yml
loop: >-
{{ range(memattr_survey_status.completed_rounds | int + 1,
memattr_survey_rounds | int + 1) | list }}
loop_control:
loop_var: memattr_survey_round
label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
# Evaluated per iteration, so the abort flag set by a rescue in one round skips every
# round after it on THIS host and no others.
when:
- memattr_survey_reboots_authorised | bool
- not (memattr_survey_aborted | bool)
infra/ansible/roles/node_health/README.md
+40 -19
View File
@@ -54,7 +54,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
| `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
| `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
| `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
in task logic.
@@ -65,33 +65,54 @@ in task logic.
`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
does not validate it before reserving memory from its contents —
`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
table address to the top of RAM. Which of two kernel-log lines appears decides whether the
node keeps its memory, and it is **decided afresh on every boot**:
does not validate it before reserving memory from its contents. **Two different kernel
functions matter, and the check used to conflate them:**
| kernel log | meaning | verdict |
- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
**catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
`phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
costs nothing measurable.
- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
"Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
merely large.
So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
prove memory was lost: that depends on the garbage **value** at that address on that boot,
not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
measured memory, never on the log line alone:**
| kernel log | measured shortfall | verdict |
|---|---|---|
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | (not consulted) | **WARN** — the garbage `version` read `> 2` so the kernel bailed out first: exposed, but this boot took the safe path |
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 validated and rejected it: this *is* the fix, working |
| no memattr line, log reaches back to boot | (not consulted) | PASS |
| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
The WARN is the point. All three prod nodes publish the same corrupt table and differ only
by **luck**, boot to boot. A node whose kernel just trusted a corrupt table is not passing a
health check even when it kept its RAM.
**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
rebooting/CMOS/BIOS/USB changes only move where the table lands.
`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
different cause is still caught.
### The two memory checks cannot contradict each other
`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
its own, so there is exactly one verdict on lost memory per host and both checks quote it.
A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
future memory loss with a different cause is still caught.
## Output
infra/ansible/roles/node_health/files/node_health_collect.py
+16 -8
View File
@@ -468,19 +468,27 @@ def collect_memory_devices():
# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
# the node keeps its RAM:
# the product). The kernel prints one of two lines:
#
# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read
# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
# "Failed to map EFI Memory Attributes table @ 0x..." the garbage version read <= 2, so
# the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
# efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
# wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
# memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
# PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
# tbl_size costs nothing measurable. Whether memory actually went missing is decided
# by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
# twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
# "Unexpected EFI Memory Attributes table version <N>" the garbage version read > 2, so
# the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
# firmware is still publishing a corrupt table and the outcome is decided afresh by
# where the table lands on every boot. That is luck, not immunity.
# what the garbage reads on every boot. That is luck, not immunity.
#
# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
# before it lost 13.3 GiB), which is precisely why it is detected on every health run
# instead of diagnosed once.
# This collector reports the LINES and the addresses; it never rules on memory. Classifying
# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
#
# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
# health run instead of diagnosed once.
#
# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
# every node here). The unprivileged pass therefore records WHY it could not read, and
infra/ansible/roles/node_health/tasks/assess.yml
+153 -49
View File
@@ -387,18 +387,28 @@
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
~ avail_pct | string ~ '%)') }}
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
# one cause we have actually hit here; this one asks the blunter question — "is this
# machine using the RAM it has?" — and answers it from two sources that share NOTHING
# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
- name: Assess — usable memory vs the memory this machine physically has
# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
# from the presence of a kernel-log line — and they duly contradicted each other on the same
# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
# measurement was right; the inference from the log line was wrong — the memattr block below
# has the refined mechanism.
#
# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
# place and both checks read this fact: neither derives a shortfall of its own, and the
# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
# disagree about whether memory was lost — there is only one verdict on that question, and
# it is this one. Thresholds and their measured baselines: defaults/main.yml.
- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'memory.physical_total',
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
'status': phys_status,
'detail': phys_detail }] }}"
node_health_mem_shortfall: "{{ {
'zones_pct': z_pct, 'zones_status': z_status,
'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
'status': worst_status } }}"
vars:
ansible_become: false
th: "{{ node_health_thresholds }}"
@@ -430,11 +440,36 @@
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
# SKIPs must never add up to a PASS.
phys_status: >-
worst_status: >-
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
| default('SKIP', true) }}
# What the kernel log says, IF it was read — quoted here only as attribution for a
# shortfall this check measured on its own. The verdict above never depends on it.
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
# blunter question — "is this machine using the RAM it has?" — and answers it from two
# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
# losing memory for some other reason still shows up.
- name: Assess — usable memory vs the memory this machine physically has
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'memory.physical_total',
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
'status': phys_status,
'detail': phys_detail }] }}"
vars:
ansible_become: false
ms: "{{ node_health_mem_shortfall }}"
z: "{{ node_health_facts.memory.zones }}"
dev: "{{ node_health_facts.memory.devices }}"
m: "{{ node_health_facts.memory }}"
# Straight from the shared measurement — this check re-derives nothing.
z_pct: "{{ ms.zones_pct }}"
inst_kb: "{{ ms.installed_kb }}"
d_pct: "{{ ms.dimm_pct }}"
phys_status: "{{ ms.status }}"
# What the kernel log says, IF it was read — quoted here only as attribution for the
# measured shortfall. The verdict above is the shared measurement and never depends on
# the log; firmware.efi_memattr reads that same verdict, so the two always agree.
phys_cause: >-
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
@@ -757,24 +792,49 @@
# what follows is only what this check needs to be read correctly.
#
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
# the product — everything from the table address to the top of RAM. The kernel prints
# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
# happens to read > 2:
# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
# are involved, and conflating them is what made this check wrong until 2026-09-03:
#
# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel
# TRUSTED the table. The node has lost the memory. FAIL.
# "Unexpected EFI Memory Attributes table version <N>" version read > 2, the kernel
# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
# unlucky boot from losing 83% of its RAM is not passing a health check.
# efi_memattr_init() tbl_size = sizeof(*tbl) + num_entries * desc_size, then
# memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
# is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
# NEGATIVE: a garbage product with bit 31 set sign-extends
# through phys_addr_t and memblock_cap_size() clamps it to
# PHYS_ADDR_MAX — everything above the table is reserved.
# A garbage-but-POSITIVE tbl_size reserves a modest, wrong
# amount and costs nothing measurable.
# efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
# THIS is what prints "Failed to map". It fails for ANY
# sufficiently wrong tbl_size — negative OR merely large.
#
# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
# is the counter-example that falsified the old inference.
#
# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
#
# "Failed to map ... @ 0x..." the bug FIRED: corrupt table, trusted by the kernel.
# FAIL when the shared measurement (node_health_mem_shortfall) says memory is
# catastrophically missing — memory is lost NOW.
# WARN otherwise — this boot drew a survivable garbage value; the next may not.
# "Unexpected ... version <N>" version read > 2, so the kernel bailed out before
# reserving anything. WARN: exposed, and this boot took the safe path.
#
# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
# measurement separates the last two. Never PASS on either line: the firmware is publishing
# a corrupt table and the draw is repeated on every boot.
#
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
# so it cannot be diagnosed once. It has to be measured on every health run.
#
# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
# node_health_mem_shortfall, computed once further up this file; this check does not measure
# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
# one verdict on any host and both checks quote it.
#
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
@@ -793,11 +853,17 @@
ansible_become: false
k: "{{ node_health_facts.klog }}"
z: "{{ node_health_facts.memory.zones }}"
# THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
# near memory.physical_total. This check contributes the kernel-log facts only.
ms: "{{ node_health_mem_shortfall }}"
ma_state: "{{ k.memattr_state | default(none) }}"
# `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
# COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
# this check and memory.physical_total can no longer disagree about lost memory.
memattr_status: >-
{{ 'SKIP' if not (k.available | bool)
else ('FAIL' if ma_state == 'failed_to_map'
else ('WARN' if ma_state == 'bad_version'
else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
else ('WARN' if ma_state in ['failed_to_map', 'bad_version']
else ('PASS' if ma_state == 'corrupt_rejected'
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
@@ -812,14 +878,32 @@
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
unplugging USB only change where the table lands and may APPEAR to fix it — they are
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
# The symptom, measured independently of this log (see memory.physical_total).
memattr_corroboration: >-
{{ (' Measured independently of this log: MemTotal '
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
# The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
# to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
# kernel log has no part in; the verdict word comes from ms.status, so it always agrees
# with memory.physical_total. When the measurement is missing it says so, and says why.
memattr_figures: >-
{{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
from the allocator (/proc/zoneinfo) — check memory.physical_total.')
if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
allocator (/proc/zoneinfo)')
if ms.zones_pct is number
else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
reports as installed (/proc/zoneinfo NOT read — '
~ (z.reason | default('no reason recorded', true)) ~ ')')
if ms.dimm_pct is number else '') }}
memattr_corroboration: >-
{{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
~ ms.status ~ '): ' ~ memattr_figures
~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
else ' — the normal few percent (kernel image, memmap, firmware
reservations, graphics stolen memory). NO measurable memory was lost
on this boot.')))
if (memattr_figures | trim | length) > 0
else (' Whether memory was lost on this boot could NOT be measured here: neither
present-vs-managed pages nor installed DIMM capacity was read — see check
memory.physical_total for the reasons.') }}
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
# against the evidence is not much better than a wrong one.
memattr_evidence: >-
@@ -837,23 +921,43 @@
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
lives only in dmesg.{{ memattr_corroboration }}
{%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
line. The kernel failed to map the EFI Memory Attributes table at
{{ k.memattr_table_address | default('an unreported address') }}, which proves the
table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
from its contents; the measurement below shows that reservation was the catastrophic
kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
{%- elif ma_state == 'failed_to_map' -%}
THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
and reserved memory from the corrupt table anyway — everything from that address
upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
{%- elif ma_state == 'bad_version' -%}
EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
here too; this boot survived only because its garbage version field read
{{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
before reserving anything. The outcome is re-decided on EVERY boot: if that field
reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
loss either. The kernel failed to map the EFI Memory Attributes table at
{{ k.memattr_table_address | default('an unreported address') }}: the table was
corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
measurable and still fails the later memremap() that printed this line. This boot drew a survivable
value; the draw is repeated on EVERY boot, at the same address — this platform has
been seen to lose 13.3 GiB and to keep its full memory at one identical table address
(infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
{{ memattr_fix }}
{%- elif ma_state == 'bad_version' -%}
EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
tbl_size the garbage produces — nothing measurable if that value is positive,
everything above the table if it is negative.{{ memattr_evidence }}
{{ memattr_corroboration }} {{ memattr_fix }}
{%- elif ma_state == 'corrupt_rejected' -%}
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
intact and stays intact across reboots; no action.{{ memattr_evidence }}
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
cannot cost this node memory on this or any future boot; no
action.{{ memattr_evidence }}{{ memattr_corroboration }}
{%- elif not (k.boot_covered | bool) -%}
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in