jakman85andClaude Opus 5
8e913fdd7b
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
93 lines
5.1 KiB
YAML
93 lines
5.1 KiB
YAML
---
|
|
# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
|
|
# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
|
|
# settled and is NOT re-derived here.
|
|
#
|
|
# THE QUESTION THIS ROLE ANSWERS
|
|
# §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
|
|
# Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
|
|
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
|
|
# observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
|
|
# successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
|
|
# address range, the causal chain is closed with no hardware or firmware touched.
|
|
# 2. Is the fault node-2-specific, or does any node reach the bad path given enough
|
|
# boots? Nodes 1 and 3 have each been observed exactly twice.
|
|
#
|
|
# WHY 20 ROUNDS AND NOT FIVE
|
|
# Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
|
|
# interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
|
|
# boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
|
|
# needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
|
|
# not a round number.
|
|
#
|
|
# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
|
|
# No BIOS writes, no kernel change, no package installs, no hardware moves. The role
|
|
# installs nothing and configures nothing; its only write to a node is its own append-
|
|
# only ledger under /var/lib. Anything that alters the boot chain's allocation
|
|
# footprint — and per §0b that includes the initrd, the module set and the boot device —
|
|
# changes the variable under test and voids the arm.
|
|
|
|
# --- Sample size --------------------------------------------------------------------
|
|
# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
|
|
# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
|
|
memattr_survey_rounds: 20
|
|
|
|
# --- Safety interlock ----------------------------------------------------------------
|
|
# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
|
|
# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
|
|
# That is a genuinely useful dry run and it is also how you prove the pipeline works
|
|
# before committing 45 minutes of wall clock to it.
|
|
memattr_survey_reboots_authorised: false
|
|
|
|
# The role refuses to run against a host that is not in this group. The prod boxes carry
|
|
# no k3s and nothing is running on them, which is the entire reason rebooting them is
|
|
# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
|
|
memattr_survey_required_group: k3s_prod
|
|
|
|
# Unit names whose presence or activity means something IS running on this node. Any hit
|
|
# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
|
|
memattr_survey_forbidden_units:
|
|
- k3s
|
|
- k3s-agent
|
|
- kubelet
|
|
- containerd
|
|
- docker
|
|
# Paths whose existence means k3s has been installed here, even if not currently running.
|
|
memattr_survey_forbidden_paths:
|
|
- /etc/rancher/k3s
|
|
- /var/lib/rancher/k3s
|
|
- /usr/local/bin/k3s
|
|
|
|
# --- Reboot behaviour -----------------------------------------------------------------
|
|
# A node that does not answer within this is declared lost: the role stops rebooting THAT
|
|
# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
|
|
# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
|
|
# slow boot.
|
|
memattr_survey_reboot_timeout: 600
|
|
memattr_survey_connect_timeout: 20
|
|
# Let journald settle before reading the new boot's kernel log. The kernel log itself is
|
|
# complete long before SSH is up; this is purely margin.
|
|
memattr_survey_post_reboot_delay: 15
|
|
|
|
# --- On-node ledger (belt) -------------------------------------------------------------
|
|
# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
|
|
# the controller dies. This is what makes the survey resumable: re-running skips rounds
|
|
# already recorded here rather than starting over.
|
|
memattr_survey_state_dir: /var/lib/memattr-survey
|
|
memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
|
|
|
|
# --- Controller artifacts (braces) ------------------------------------------------------
|
|
# Same convention as roles/node_health and conformance/_artifacts/: generated output under
|
|
# a `_artifacts/` directory that .gitignore excludes, never committed.
|
|
# <host>.rounds.jsonl the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
|
|
# mid-survey still leaves every completed round on the controller
|
|
# <host>.boots.json the journal harvest — every retained boot re-derived in one
|
|
# pass. THIS is the authoritative record; the ledger is the backup.
|
|
# summary.md the analysis (also printed)
|
|
memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
|
|
memattr_survey_write_artifacts: true
|
|
|
|
# Print each round's one-line result as it happens. A 45-minute unattended run with no
|
|
# output is indistinguishable from a hung one.
|
|
memattr_survey_show_progress: true
|