feat(node_health): detect the EFI memattr bug on every run, two independent ways
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
47f02c6e06
commit
cf33338448
7 files changed
+664
-6
No files matched your search
@@ -45,6 +45,33 @@ node_health_thresholds:
|
||||
# Memory (percentage of MemTotal still available)
|
||||
mem_available_pct_warn: 15
|
||||
mem_available_pct_fail: 5
|
||||
# --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
|
||||
# Two independent measurements of the same question, both fed by memory.physical_total:
|
||||
#
|
||||
# unmanaged /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
|
||||
# memory map says exists, that never reached the allocator. Needs no root
|
||||
# and no tools, works identically on metal and in a VM.
|
||||
# shortfall MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
|
||||
# installed. Needs root, and only bare metal reports it. Independent of the
|
||||
# kernel's memory map entirely, so it also catches RAM the firmware never
|
||||
# handed the kernel in the first place.
|
||||
#
|
||||
# MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
|
||||
# not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
|
||||
# k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
|
||||
# The healthy figure is the kernel image, memmap and firmware reservations; it is a
|
||||
# near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
|
||||
# keeps clearance for that. There is no plausible tuning of a threshold in this band
|
||||
# that changes any of those five verdicts.
|
||||
#
|
||||
# The DIMM shortfall runs a little higher than the unmanaged figure on the same node
|
||||
# (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
|
||||
# the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
|
||||
# kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
|
||||
mem_unmanaged_pct_warn: 10
|
||||
mem_unmanaged_pct_fail: 25
|
||||
mem_dimm_shortfall_pct_warn: 10
|
||||
mem_dimm_shortfall_pct_fail: 25
|
||||
# Temperatures (deg C, any sensor)
|
||||
temp_warn_c: 70
|
||||
temp_fail_c: 85
|
||||
|
||||
Reference in new issue
Block a user