jakman85andClaude Opus 5
8e913fdd7b
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
138 lines
7.1 KiB
YAML
138 lines
7.1 KiB
YAML
---
|
|
# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
|
|
# R-ENV-03, R-TEST-02).
|
|
#
|
|
# WHAT QUESTION THIS ANSWERS
|
|
# infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
|
|
# EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
|
|
# Linux 6.12 reserves everything from the table address to the top of RAM without
|
|
# validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
|
|
# playbook measures them:
|
|
#
|
|
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
|
|
# failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
|
|
# fleet-wide were at other addresses. If bad outcomes map onto a narrow address
|
|
# range, the causal chain closes with nothing touched.
|
|
# 2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
|
|
# boots? Nodes 1 and 3 have each been observed exactly twice.
|
|
#
|
|
# §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
|
|
# is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
|
|
# a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
|
|
# boots per arm; detecting a mere odds shift needs ~50.
|
|
#
|
|
# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
|
|
# package installs, no hardware moves, no node_baseline run. Its job is to establish real
|
|
# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
|
|
# a Fast Boot flip, a 6.13 kernel) compares against it.
|
|
#
|
|
# HOW TO RUN IT
|
|
# Dry run first — every stage except the reboots, so the pipeline is proven before 45
|
|
# minutes of wall clock are committed:
|
|
#
|
|
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
|
# ansible-playbook playbooks/memattr-survey.yml'
|
|
#
|
|
# The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
|
|
#
|
|
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
|
# ansible-playbook playbooks/memattr-survey.yml \
|
|
# -e memattr_survey_reboots_authorised=true'
|
|
#
|
|
# Fewer rounds, or one node:
|
|
# -e memattr_survey_rounds=5 --limit k3s-prod-2
|
|
#
|
|
# RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
|
|
# rounds are never lost or double-counted. Analysis alone, over existing artifacts:
|
|
# ansible-playbook playbooks/memattr-survey.yml --tags analyse
|
|
#
|
|
# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
|
|
# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
|
|
# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
|
|
# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
|
|
# engineering against.
|
|
#
|
|
# HOW TO READ THE OUTPUT
|
|
# infra/ansible/_artifacts/memattr-survey/ (gitignored, regenerated every run)
|
|
# <host>.rounds.jsonl one JSON line per round, mirrored off the node after EVERY round
|
|
# <host>.boots.json every boot the persistent journal holds, re-derived in one pass
|
|
# at the end — the AUTHORITATIVE record
|
|
# summary.md the analysis, also printed at the end of the run
|
|
# See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
|
|
# with NO memattr log line is a healthy third state and not a missing measurement.
|
|
|
|
# =====================================================================================
|
|
# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
|
|
# before a single reboot. That is the right asymmetry: the cost of a false stop is a
|
|
# re-run, the cost of a false start is rebooting a machine that was doing something.
|
|
# =====================================================================================
|
|
- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
|
|
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
|
|
gather_facts: false
|
|
become: true
|
|
any_errors_fatal: true
|
|
tags: [survey]
|
|
tasks:
|
|
- name: Preflight and baseline probe
|
|
ansible.builtin.include_role:
|
|
name: memattr_survey
|
|
tasks_from: preflight
|
|
|
|
# =====================================================================================
|
|
# PLAY 2 — THE SURVEY.
|
|
#
|
|
# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
|
|
# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
|
|
# keep collecting.
|
|
#
|
|
# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
|
|
# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
|
|
# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
|
|
# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
|
|
# blocks belongs to exactly one host, so the other two sit idle through it and the survey
|
|
# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
|
|
# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
|
|
# appeared in the output. `free` lets each host walk its own task list at its own pace,
|
|
# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
|
|
#
|
|
# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
|
|
# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
|
|
# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
|
|
# directory create on every host instead. Concurrency is also capped by `forks`
|
|
# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
|
|
# than letting a bigger fleet silently serialise again.
|
|
# =====================================================================================
|
|
- name: memattr survey — reboot rounds and journal harvest
|
|
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
|
|
gather_facts: false
|
|
become: true
|
|
strategy: free
|
|
tags: [survey]
|
|
tasks:
|
|
- name: Run the reboot rounds
|
|
ansible.builtin.include_role:
|
|
name: memattr_survey
|
|
tasks_from: rounds
|
|
|
|
- name: Harvest every retained boot from the persistent journal
|
|
ansible.builtin.include_role:
|
|
name: memattr_survey
|
|
tasks_from: harvest
|
|
|
|
# =====================================================================================
|
|
# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
|
|
# even if every surveyed node died, because the artifacts are already on disk and the
|
|
# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
|
|
# vanished shows up as a short arm with a wider interval, not as a missing report.
|
|
# =====================================================================================
|
|
- name: memattr survey — analyse
|
|
hosts: localhost
|
|
gather_facts: false
|
|
become: false
|
|
tags: [analyse]
|
|
tasks:
|
|
- name: Produce the analysis
|
|
ansible.builtin.include_role:
|
|
name: memattr_survey
|
|
tasks_from: analyse
|