fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
b19c5db9ae
commit
8e913fdd7b
16 files changed
+2168
-76
No files matched your search
@@ -387,18 +387,28 @@
|
||||
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
|
||||
~ avail_pct | string ~ '%)') }}
|
||||
|
||||
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
|
||||
# one cause we have actually hit here; this one asks the blunter question — "is this
|
||||
# machine using the RAM it has?" — and answers it from two sources that share NOTHING
|
||||
# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
|
||||
# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
|
||||
- name: Assess — usable memory vs the memory this machine physically has
|
||||
# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
|
||||
# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
|
||||
# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
|
||||
# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
|
||||
# from the presence of a kernel-log line — and they duly contradicted each other on the same
|
||||
# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
|
||||
# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
|
||||
# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
|
||||
# measurement was right; the inference from the log line was wrong — the memattr block below
|
||||
# has the refined mechanism.
|
||||
#
|
||||
# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
|
||||
# place and both checks read this fact: neither derives a shortfall of its own, and the
|
||||
# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
|
||||
# disagree about whether memory was lost — there is only one verdict on that question, and
|
||||
# it is this one. Thresholds and their measured baselines: defaults/main.yml.
|
||||
- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.physical_total',
|
||||
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
||||
'status': phys_status,
|
||||
'detail': phys_detail }] }}"
|
||||
node_health_mem_shortfall: "{{ {
|
||||
'zones_pct': z_pct, 'zones_status': z_status,
|
||||
'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
|
||||
'status': worst_status } }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
th: "{{ node_health_thresholds }}"
|
||||
@@ -430,11 +440,36 @@
|
||||
|
||||
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
|
||||
# SKIPs must never add up to a PASS.
|
||||
phys_status: >-
|
||||
worst_status: >-
|
||||
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
|
||||
| default('SKIP', true) }}
|
||||
# What the kernel log says, IF it was read — quoted here only as attribution for a
|
||||
# shortfall this check measured on its own. The verdict above never depends on it.
|
||||
|
||||
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
|
||||
# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
|
||||
# blunter question — "is this machine using the RAM it has?" — and answers it from two
|
||||
# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
|
||||
# losing memory for some other reason still shows up.
|
||||
- name: Assess — usable memory vs the memory this machine physically has
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.physical_total',
|
||||
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
||||
'status': phys_status,
|
||||
'detail': phys_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
ms: "{{ node_health_mem_shortfall }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
dev: "{{ node_health_facts.memory.devices }}"
|
||||
m: "{{ node_health_facts.memory }}"
|
||||
# Straight from the shared measurement — this check re-derives nothing.
|
||||
z_pct: "{{ ms.zones_pct }}"
|
||||
inst_kb: "{{ ms.installed_kb }}"
|
||||
d_pct: "{{ ms.dimm_pct }}"
|
||||
phys_status: "{{ ms.status }}"
|
||||
# What the kernel log says, IF it was read — quoted here only as attribution for the
|
||||
# measured shortfall. The verdict above is the shared measurement and never depends on
|
||||
# the log; firmware.efi_memattr reads that same verdict, so the two always agree.
|
||||
phys_cause: >-
|
||||
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
|
||||
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
|
||||
@@ -757,24 +792,49 @@
|
||||
# what follows is only what this check needs to be read correctly.
|
||||
#
|
||||
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
|
||||
# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
|
||||
# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
|
||||
# the product — everything from the table address to the top of RAM. The kernel prints
|
||||
# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
|
||||
# happens to read > 2:
|
||||
# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
|
||||
# are involved, and conflating them is what made this check wrong until 2026-09-03:
|
||||
#
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel
|
||||
# TRUSTED the table. The node has lost the memory. FAIL.
|
||||
# "Unexpected EFI Memory Attributes table version <N>" version read > 2, the kernel
|
||||
# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
|
||||
# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
|
||||
# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
|
||||
# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
|
||||
# unlucky boot from losing 83% of its RAM is not passing a health check.
|
||||
# efi_memattr_init() tbl_size = sizeof(*tbl) + num_entries * desc_size, then
|
||||
# memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
|
||||
# is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
|
||||
# NEGATIVE: a garbage product with bit 31 set sign-extends
|
||||
# through phys_addr_t and memblock_cap_size() clamps it to
|
||||
# PHYS_ADDR_MAX — everything above the table is reserved.
|
||||
# A garbage-but-POSITIVE tbl_size reserves a modest, wrong
|
||||
# amount and costs nothing measurable.
|
||||
# efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
|
||||
# THIS is what prints "Failed to map". It fails for ANY
|
||||
# sufficiently wrong tbl_size — negative OR merely large.
|
||||
#
|
||||
# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
|
||||
# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
|
||||
# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
|
||||
# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
|
||||
# is the counter-example that falsified the old inference.
|
||||
#
|
||||
# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
|
||||
#
|
||||
# "Failed to map ... @ 0x..." the bug FIRED: corrupt table, trusted by the kernel.
|
||||
# FAIL when the shared measurement (node_health_mem_shortfall) says memory is
|
||||
# catastrophically missing — memory is lost NOW.
|
||||
# WARN otherwise — this boot drew a survivable garbage value; the next may not.
|
||||
# "Unexpected ... version <N>" version read > 2, so the kernel bailed out before
|
||||
# reserving anything. WARN: exposed, and this boot took the safe path.
|
||||
#
|
||||
# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
|
||||
# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
|
||||
# measurement separates the last two. Never PASS on either line: the firmware is publishing
|
||||
# a corrupt table and the draw is repeated on every boot.
|
||||
#
|
||||
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
|
||||
# so it cannot be diagnosed once. It has to be measured on every health run.
|
||||
#
|
||||
# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
|
||||
# node_health_mem_shortfall, computed once further up this file; this check does not measure
|
||||
# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
|
||||
# one verdict on any host and both checks quote it.
|
||||
#
|
||||
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
|
||||
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
|
||||
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
|
||||
@@ -793,11 +853,17 @@
|
||||
ansible_become: false
|
||||
k: "{{ node_health_facts.klog }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
# THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
|
||||
# near memory.physical_total. This check contributes the kernel-log facts only.
|
||||
ms: "{{ node_health_mem_shortfall }}"
|
||||
ma_state: "{{ k.memattr_state | default(none) }}"
|
||||
# `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
|
||||
# COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
|
||||
# this check and memory.physical_total can no longer disagree about lost memory.
|
||||
memattr_status: >-
|
||||
{{ 'SKIP' if not (k.available | bool)
|
||||
else ('FAIL' if ma_state == 'failed_to_map'
|
||||
else ('WARN' if ma_state == 'bad_version'
|
||||
else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
|
||||
else ('WARN' if ma_state in ['failed_to_map', 'bad_version']
|
||||
else ('PASS' if ma_state == 'corrupt_rejected'
|
||||
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
|
||||
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
|
||||
@@ -812,14 +878,32 @@
|
||||
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
|
||||
unplugging USB only change where the table lands and may APPEAR to fix it — they are
|
||||
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
||||
# The symptom, measured independently of this log (see memory.physical_total).
|
||||
memattr_corroboration: >-
|
||||
{{ (' Measured independently of this log: MemTotal '
|
||||
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
# The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
|
||||
# to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
|
||||
# kernel log has no part in; the verdict word comes from ms.status, so it always agrees
|
||||
# with memory.physical_total. When the measurement is missing it says so, and says why.
|
||||
memattr_figures: >-
|
||||
{{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
|
||||
~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
|
||||
from the allocator (/proc/zoneinfo) — check memory.physical_total.')
|
||||
if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
|
||||
~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
|
||||
allocator (/proc/zoneinfo)')
|
||||
if ms.zones_pct is number
|
||||
else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
|
||||
reports as installed (/proc/zoneinfo NOT read — '
|
||||
~ (z.reason | default('no reason recorded', true)) ~ ')')
|
||||
if ms.dimm_pct is number else '') }}
|
||||
memattr_corroboration: >-
|
||||
{{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
|
||||
~ ms.status ~ '): ' ~ memattr_figures
|
||||
~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
|
||||
else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
|
||||
else ' — the normal few percent (kernel image, memmap, firmware
|
||||
reservations, graphics stolen memory). NO measurable memory was lost
|
||||
on this boot.')))
|
||||
if (memattr_figures | trim | length) > 0
|
||||
else (' Whether memory was lost on this boot could NOT be measured here: neither
|
||||
present-vs-managed pages nor installed DIMM capacity was read — see check
|
||||
memory.physical_total for the reasons.') }}
|
||||
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
|
||||
# against the evidence is not much better than a wrong one.
|
||||
memattr_evidence: >-
|
||||
@@ -837,23 +921,43 @@
|
||||
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
|
||||
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
|
||||
lives only in dmesg.{{ memattr_corroboration }}
|
||||
{%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
|
||||
THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
|
||||
line. The kernel failed to map the EFI Memory Attributes table at
|
||||
{{ k.memattr_table_address | default('an unreported address') }}, which proves the
|
||||
table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
|
||||
from its contents; the measurement below shows that reservation was the catastrophic
|
||||
kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
|
||||
through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
|
||||
table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'failed_to_map' -%}
|
||||
THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
|
||||
Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
|
||||
and reserved memory from the corrupt table anyway — everything from that address
|
||||
upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'bad_version' -%}
|
||||
EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
|
||||
here too; this boot survived only because its garbage version field read
|
||||
{{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
|
||||
before reserving anything. The outcome is re-decided on EVERY boot: if that field
|
||||
reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
|
||||
now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
|
||||
THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
|
||||
loss either. The kernel failed to map the EFI Memory Attributes table at
|
||||
{{ k.memattr_table_address | default('an unreported address') }}: the table was
|
||||
corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
|
||||
WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
|
||||
the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
|
||||
with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
|
||||
tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
|
||||
measurable and still fails the later memremap() that printed this line. This boot drew a survivable
|
||||
value; the draw is repeated on EVERY boot, at the same address — this platform has
|
||||
been seen to lose 13.3 GiB and to keep its full memory at one identical table address
|
||||
(infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
|
||||
{{ memattr_fix }}
|
||||
{%- elif ma_state == 'bad_version' -%}
|
||||
EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
|
||||
CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
|
||||
version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
|
||||
kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
|
||||
EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
|
||||
tbl_size the garbage produces — nothing measurable if that value is positive,
|
||||
everything above the table if it is negative.{{ memattr_evidence }}
|
||||
{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'corrupt_rejected' -%}
|
||||
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
|
||||
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
|
||||
intact and stays intact across reboots; no action.{{ memattr_evidence }}
|
||||
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
|
||||
cannot cost this node memory on this or any future boot; no
|
||||
action.{{ memattr_evidence }}{{ memattr_corroboration }}
|
||||
{%- elif not (k.boot_covered | bool) -%}
|
||||
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
|
||||
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
|
||||
|
||||
Reference in new issue
Block a user