jakman85andClaude Opus 5
8e913fdd7b
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
41 lines
1.9 KiB
YAML
41 lines
1.9 KiB
YAML
---
|
|
# The loop itself, kept separate from round.yml so the loop bounds (which encode the
|
|
# RESUME logic) are readable in one place.
|
|
#
|
|
# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
|
|
# re-run after an interruption continues where it stopped and can never lose or duplicate
|
|
# an earlier round. Already at or past the target count => nothing to do, and the run
|
|
# collapses to a harvest.
|
|
#
|
|
# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
|
|
# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
|
|
# analysis) still runs, so a dry run exercises the entire pipeline without touching a
|
|
# single machine.
|
|
|
|
- name: Announce the plan
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ inventory_hostname }}:
|
|
{%- if not (memattr_survey_reboots_authorised | bool) %}
|
|
DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
|
|
{%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
|
|
already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
|
|
{%- else %}
|
|
rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
|
|
to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
|
|
{%- endif %}
|
|
|
|
- name: Survey round
|
|
ansible.builtin.include_tasks: round.yml
|
|
loop: >-
|
|
{{ range(memattr_survey_status.completed_rounds | int + 1,
|
|
memattr_survey_rounds | int + 1) | list }}
|
|
loop_control:
|
|
loop_var: memattr_survey_round
|
|
label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
|
|
# Evaluated per iteration, so the abort flag set by a rescue in one round skips every
|
|
# round after it on THIS host and no others.
|
|
when:
|
|
- memattr_survey_reboots_authorised | bool
|
|
- not (memattr_survey_aborted | bool)
|