jakman85andClaude Opus 5
8e913fdd7b
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
67 lines
2.9 KiB
YAML
67 lines
2.9 KiB
YAML
---
|
|
# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
|
|
# scratch in one pass by the same code that produced the per-round ledger.
|
|
#
|
|
# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
|
|
# the append, that round exists on the machine but not in the ledger. The journal does not
|
|
# care whether the driver was alive — the boot happened, the kernel logged it, journald
|
|
# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
|
|
# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
|
|
# any boot the ledger missed still shows up.
|
|
#
|
|
# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
|
|
|
|
- name: Harvest every retained boot from the persistent journal
|
|
ansible.builtin.script:
|
|
cmd: memattr_probe.py --all-boots
|
|
executable: /usr/bin/python3
|
|
register: memattr_survey_harvest_raw
|
|
changed_when: false # strictly a read of the journal
|
|
|
|
- name: Parse the harvest
|
|
ansible.builtin.set_fact:
|
|
memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
|
|
|
|
- name: Ensure the artifact directory exists (on the controller)
|
|
ansible.builtin.file:
|
|
path: "{{ memattr_survey_artifact_dir }}"
|
|
state: directory
|
|
mode: "0755"
|
|
delegate_to: localhost
|
|
become: false
|
|
# NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
|
|
# run_once is unreliable there. `state: directory` is idempotent, so three hosts each
|
|
# asserting the directory exists is cheaper than a conditional that might not fire.
|
|
changed_when: false
|
|
when: memattr_survey_write_artifacts | bool
|
|
|
|
- name: Write the harvested boot record
|
|
ansible.builtin.copy:
|
|
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
|
|
mode: "0644"
|
|
content: "{{ memattr_survey_harvest | to_nice_json }}"
|
|
delegate_to: localhost
|
|
become: false
|
|
changed_when: false
|
|
when: memattr_survey_write_artifacts | bool
|
|
|
|
- name: Mirror the final ledger to the controller
|
|
ansible.builtin.fetch:
|
|
src: "{{ memattr_survey_state_file }}"
|
|
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
|
|
flat: true
|
|
changed_when: false
|
|
when: memattr_survey_write_artifacts | bool
|
|
|
|
- name: Harvest summary
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
|
|
from the journal
|
|
({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
|
|
{{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
|
|
{{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
|
|
{%- if memattr_survey_aborted | default(false) | bool %}
|
|
— NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
|
|
{%- endif %}
|