feat(node_health): detect the EFI memattr bug on every run, two independent ways

The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.

firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
  Failed to map EFI Memory Attributes table @ 0x..   -> FAIL (memory lost NOW)
  Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
                                                        intact only by luck)
  [Firmware Bug]: Corrupted EFI ... detected!        -> PASS (the >=6.13 fix working)
  no memattr line, log reaches early boot            -> PASS (no table / accepted / not UEFI)
  no memattr line, log does NOT reach early boot     -> SKIP
  dmesg unreadable                                   -> SKIP

memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.

THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
   collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
   memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.

VERDICTS, confirmed live:
  k3s-prod-1  WARN "EXPOSED, not immune ... survived only because its garbage version field read
                    1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
  k3s-prod-2  FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
  k3s-dev-*   PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
                    Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.

Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.

LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.

INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 22:30:45 -05:00
1 parent 47f02c6e06
commit cf33338448
7 files changed
+664 -6

No files matched your search

@@ -387,6 +387,83 @@
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
~ avail_pct | string ~ '%)') }}
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
# one cause we have actually hit here; this one asks the blunter question — "is this
# machine using the RAM it has?" — and answers it from two sources that share NOTHING
# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
- name: Assess — usable memory vs the memory this machine physically has
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'memory.physical_total',
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
'status': phys_status,
'detail': phys_detail }] }}"
vars:
ansible_become: false
th: "{{ node_health_thresholds }}"
z: "{{ node_health_facts.memory.zones }}"
dev: "{{ node_health_facts.memory.devices }}"
m: "{{ node_health_facts.memory }}"
# 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
# actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
# tools, identical on metal and in a VM — so this half is available almost always.
z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
z_status: >-
{{ 'SKIP' if z_pct is not number
else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
# 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
# VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
# never as a zero DIMM count. Independent of the kernel's memory map entirely.
inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
d_pct: >-
{{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
else none }}
d_status: >-
{{ 'SKIP' if d_pct is not number
else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
# SKIPs must never add up to a PASS.
phys_status: >-
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
| default('SKIP', true) }}
# What the kernel log says, IF it was read — quoted here only as attribution for a
# shortfall this check measured on its own. The verdict above never depends on it.
phys_cause: >-
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
else ('. The kernel log was NOT read here, so this shortfall is unattributed —
see check firmware.efi_memattr')
if not (node_health_facts.klog.available | bool)
else '' }}
phys_detail: >-
{{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
was never handed to the allocator (/proc/zoneinfo)')
if z_pct is number
else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
~ '. '
~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
if d_pct is number
else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
reservations and (on this platform) 64 MiB of graphics stolen memory.
A shortfall of this size is memory the machine HAS and cannot use')
if phys_status in ['WARN', 'FAIL'] else '')
~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
- name: Assess — swap (kubelet requires it off)
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +751,127 @@
~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
~ (' — serial numbers not read (root-only DMI fields)'
if d.product_serial is none else '')) }}
# --- EFI Memory Attributes Table ------------------------------------------------------
# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
# what follows is only what this check needs to be read correctly.
#
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
# the product — everything from the table address to the top of RAM. The kernel prints
# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
# happens to read > 2:
#
# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel
# TRUSTED the table. The node has lost the memory. FAIL.
# "Unexpected EFI Memory Attributes table version <N>" version read > 2, the kernel
# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
# unlucky boot from losing 83% of its RAM is not passing a health check.
#
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
# so it cannot be diagnosed once. It has to be measured on every health run.
#
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
# log at all, so a node whose log is unreadable is still not unmeasured.
- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'firmware.efi_memattr',
'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
'status': memattr_status,
'detail': memattr_detail }] }}"
vars:
ansible_become: false
k: "{{ node_health_facts.klog }}"
z: "{{ node_health_facts.memory.zones }}"
ma_state: "{{ k.memattr_state | default(none) }}"
memattr_status: >-
{{ 'SKIP' if not (k.available | bool)
else ('FAIL' if ma_state == 'failed_to_map'
else ('WARN' if ma_state == 'bad_version'
else ('PASS' if ma_state == 'corrupt_rejected'
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
# kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
# could take effect. Suggesting one would be a fix that quietly does nothing.
memattr_fix: >-
FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
bogus", first released in v6.13 and absent from v6.12 and from the current
linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
unplugging USB only change where the table lands and may APPEAR to fix it — they are
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
# The symptom, measured independently of this log (see memory.physical_total).
memattr_corroboration: >-
{{ (' Measured independently of this log: MemTotal '
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
from the allocator (/proc/zoneinfo) — check memory.physical_total.')
if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
# against the evidence is not much better than a wrong one.
memattr_evidence: >-
{{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
if (k.memattr_lines | default([]) | length) > 0 else '')
~ ((' At boot the kernel reported '
~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
else '') }}
memattr_detail: >-
{%- if not (k.available | bool) -%}
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
lives only in dmesg.{{ memattr_corroboration }}
{%- elif ma_state == 'failed_to_map' -%}
THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
and reserved memory from the corrupt table anyway — everything from that address
upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
{%- elif ma_state == 'bad_version' -%}
EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
here too; this boot survived only because its garbage version field read
{{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
before reserving anything. The outcome is re-decided on EVERY boot: if that field
reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
{{ memattr_fix }}
{%- elif ma_state == 'corrupt_rejected' -%}
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
intact and stays intact across reboots; no action.{{ memattr_evidence }}
{%- elif not (k.boot_covered | bool) -%}
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
the ring buffer, which wraps). Every memattr diagnostic is printed within the first
fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
pass. Read /var/log/kern.log for this boot, or re-run after a
reboot.{{ memattr_corroboration }}
{%- elif k.memattr_published is sameas true -%}
the firmware publishes an EFI Memory Attributes table at
{{ k.memattr_table_address | default('an unreported address') }} and the kernel
accepted it without complaint, in a log that still reaches back to early boot. Not
affected on this boot.
{%- elif k.memattr_published is sameas false -%}
this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
guest.
{%- else -%}
no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
boot, so that absence is measured, not assumed.
{%- endif -%}
@@ -0,0 +1,62 @@
---
# The SECOND pass of the collector, run with escalation, for the two facts that are
# genuinely root-gated:
#
# * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
# unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
# diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
# between a node that has lost 13.3 GiB and one that has not.
# * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
# DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
#
# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
# source of truth for the same fact, and the two would drift. `--privileged` makes the
# collector emit exactly those subtrees and nothing else, so the parsing stays in one
# place and is identical whichever pass produced it.
#
# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
# actually failed to read one of them (main.yml checks both). On a node where
# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
# is that everything readable WITHOUT root is read without root.
#
# If escalation is unavailable the checks that consume these SKIP with the exact reason
# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
# and "could not read the kernel log" are different answers and the role says which.
- name: Re-collect the root-only facts with escalation
ansible.builtin.script:
cmd: node_health_collect.py --privileged
executable: /usr/bin/python3
become: true
vars:
ansible_become: true
register: node_health_privileged_raw
changed_when: false # strictly read-only; it can never change the node
- name: Assert the privileged payload is the schema this role understands
ansible.builtin.assert:
that:
- (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
fail_msg: >-
Privileged collector pass returned an unexpected schema; role and collector are
out of step.
quiet: true
vars:
ansible_become: false
# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
# would REPLACE it and throw away the reasons the first pass recorded.
- name: Merge the privileged facts over the unprivileged ones
ansible.builtin.set_fact:
node_health_facts: >-
{{ node_health_facts | combine({
'klog': priv.klog,
'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
}) }}
vars:
ansible_become: false
priv: "{{ node_health_privileged_raw.stdout | from_json }}"
+15 -3
View File
@@ -4,9 +4,11 @@
# ORDER OF OPERATIONS
# 1. Collect everything readable WITHOUT root (one script, one round trip).
# 2. Probe whether privilege escalation actually works on this host.
# 3. Only if it does AND smartmontools is installed: read SMART.
# 4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
# 5. Render the human summary + write the JSON artifact.
# 3. Only if it does, and only for what the first pass could NOT read: re-run the same
# collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
# 4. Only if it does AND smartmontools is installed: read SMART.
# 5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
# 6. Render the human summary + write the JSON artifact.
#
# BECOME HANDLING — the subtle bit, read before editing.
# `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
ansible.builtin.include_tasks: probe_privilege.yml
when: node_health_try_privileged | bool
# Only when the unprivileged pass actually failed to read one of them. On a node with
# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
ansible.builtin.include_tasks: collect_privileged.yml
when:
- node_health_root_available | bool
- (not (node_health_facts.klog.available | bool))
or (not (node_health_facts.memory.devices.available | bool))
- name: Read SMART data (privileged)
ansible.builtin.include_tasks: collect_smart.yml
when: