--- # EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01, # R-ENV-03, R-TEST-02). # # WHAT QUESTION THIS ANSWERS # infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the # EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and # Linux 6.12 reserves everything from the table address to the top of RAM without # validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this # playbook measures them: # # 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed # failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes # fleet-wide were at other addresses. If bad outcomes map onto a narrow address # range, the causal chain closes with nothing touched. # 2. Is the fault node-2-specific, or does ANY node reach the bad path given enough # boots? Nodes 1 and 3 have each been observed exactly twice. # # §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate # is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against # a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20 # boots per arm; detecting a mere odds shift needs ~50. # # THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no # package installs, no hardware moves, no node_baseline run. Its job is to establish real # p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap, # a Fast Boot flip, a 6.13 kernel) compares against it. # # HOW TO RUN IT # Dry run first — every stage except the reboots, so the pipeline is proven before 45 # minutes of wall clock are committed: # # ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ # ansible-playbook playbooks/memattr-survey.yml' # # The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate): # # ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \ # ansible-playbook playbooks/memattr-survey.yml \ # -e memattr_survey_reboots_authorised=true' # # Fewer rounds, or one node: # -e memattr_survey_rounds=5 --limit k3s-prod-2 # # RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier # rounds are never lost or double-counted. Analysis alone, over existing artifacts: # ansible-playbook playbooks/memattr-survey.yml --tags analyse # # TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml. # It defaults to k3s_prod and the role additionally REFUSES to run against a host that is # not a member of memattr_survey_required_group — the dev cluster is live, and a survey # that reboots it 20 times because of a mistyped --limit is exactly the accident worth # engineering against. # # HOW TO READ THE OUTPUT # infra/ansible/_artifacts/memattr-survey/ (gitignored, regenerated every run) # .rounds.jsonl one JSON line per round, mirrored off the node after EVERY round # .boots.json every boot the persistent journal holds, re-derived in one pass # at the end — the AUTHORITATIVE record # summary.md the analysis, also printed at the end of the run # See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot # with NO memattr log line is a healthy third state and not a missing measurement. # ===================================================================================== # PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook # before a single reboot. That is the right asymmetry: the cost of a false stop is a # re-run, the cost of a false start is rebooting a machine that was doing something. # ===================================================================================== - name: memattr survey — preflight (refuse to reboot anything that is not idle prod) hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}" gather_facts: false become: true any_errors_fatal: true tags: [survey] tasks: - name: Preflight and baseline probe ansible.builtin.include_role: name: memattr_survey tasks_from: preflight # ===================================================================================== # PLAY 2 — THE SURVEY. # # any_errors_fatal deliberately OFF: here a node that fails to come back must stop only # ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others # keep collecting. # # `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not # assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands # that into one include PER (host, item), appended per host: all 20 of node 1's rounds, # then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those # blocks belongs to exactly one host, so the other two sit idle through it and the survey # runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are # completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever # appeared in the output. `free` lets each host walk its own task list at its own pace, # which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots. # # Consequences worth knowing before editing: output INTERLEAVES between hosts (every # progress line is host-tagged for exactly this reason), and `run_once` is unreliable # under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent # directory create on every host instead. Concurrency is also capped by `forks` # (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather # than letting a bigger fleet silently serialise again. # ===================================================================================== - name: memattr survey — reboot rounds and journal harvest hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}" gather_facts: false become: true strategy: free tags: [survey] tasks: - name: Run the reboot rounds ansible.builtin.include_role: name: memattr_survey tasks_from: rounds - name: Harvest every retained boot from the persistent journal ansible.builtin.include_role: name: memattr_survey tasks_from: harvest # ===================================================================================== # PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs # even if every surveyed node died, because the artifacts are already on disk and the # analysis counts rounds from THOSE rather than from Ansible's host state. A node that # vanished shows up as a short arm with a wider interval, not as a missing report. # ===================================================================================== - name: memattr survey — analyse hosts: localhost gather_facts: false become: false tags: [analyse] tasks: - name: Produce the analysis ansible.builtin.include_role: name: memattr_survey tasks_from: analyse