diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 30714f1..c19e035 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -35,6 +35,12 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. | `/dev/kvm` | present | present | | | Serial | MXL1052NGN | MXL0073949 | | +**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check; +unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**), +`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct, +but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded +only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch. + **The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's @@ -44,91 +50,148 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- -## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB +## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01) -Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install -more RAM until this is understood** — see the warning at the end. +> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory +> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine. +> Keeping the correction visible because the wrong framing nearly cost money and a board RMA. -| Evidence | Reading | -|---|---| -| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | -| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | -| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | -| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | -| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | -| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | -| Memory blocks | 127 online, 128 MiB block size | +`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware +publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before +reserving memory based on its contents.** Fixed upstream in **6.13**. -**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The -kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and -everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping -above 4 GB not being applied**, not of failing memory. +### The evidence that breaks it open -Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, -identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it -is per-machine state. +The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full +15.9 GiB two boots earlier**: -**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` -kernel driver, which exposes **215 BIOS attributes** read/write at -`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0). +| Boot | `Memory: … available` | reserved | memattr line | +|---|---|---|---| +| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` | +| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | +| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | -**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was -made on 2026-09-01 and the FIRMWARE refused it: -`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. -HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent -— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. -Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS -Setup Password is deliberately provisioned (see §1). +The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in +that shape, and none is fixed by a reboot. -What the comparison ruled OUT as the cause: -- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and - `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. -- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on - either node, so it cannot be a mis-set option. -- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, - which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. -- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory - blocks report `online`. The memory reaches the kernel and is then reserved. +### Mechanism (verified against kernel source) -Real BIOS differences found (node 1 = works, node 2 = broken): +`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`: -| Attribute | node 1 | node 2 | +```c +if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; } +tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; +memblock_reserve(efi_mem_attr_table, tbl_size); +``` + +**`num_entries` and `desc_size` are never validated.** With a corrupt table: +- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)* +- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s + the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)* + +That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = +DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. + +### BOTH prod nodes are affected — node 1 has only been lucky + +| Node | memattr version field | Outcome | |---|---|---| -| **After Power Loss** | Power On | **Power Off** ← violates §1 | -| **Fast Boot** | Disable | **Enable** ← leading hypothesis | -| Legacy Boot Options | Disable | Enable | -| TPM Activation Policy | F1 to Boot | Allow user to reject | -| Microsoft/Windows UEFI CA 2023 | Enable | Disable | -| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | +| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** | +| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB | +| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB | -**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* -rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale -training data is a plausible route to memory being present, inventoried, handed over — and then -reserved. It is also the one memory-relevant setting that differs from the working node. This is -a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write -and a reboot, which needs operator sanction. +Node 1 is one unlucky boot away from the identical fault. **Fix both.** -To investigate, cheapest first: -- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - **Must be done in F10** — the firmware refuses programmatic writes (above). -- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and - node 2 currently violates it. -- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. -- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot - vs board in one move. -- [ ] `memtest86+` if the above do not resolve it. +### Mechanism, traced to the byte (second, independent confirmation) -> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS -> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. -> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` -> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. +A second research pass reached the same root cause by a different route and pinned the arithmetic: + +- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The + kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that + `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it. +- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and + when bit 31 lands the result is **negative**. +- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to + ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical + address above `0xbeeb0018` is reserved.** +- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages` + (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`. +- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by + `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is + therefore positive proof the reservation already happened** — it is a consequence, not the cause. + +The cleanest A/B tell between the two nodes is one line of dmesg: + +| | `NODE_DATA(0) allocated` | meaning | +|---|---|---| +| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation | +| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement | + +Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs +node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. + +### THERE IS NO KERNEL CMDLINE WORKAROUND + +`efi_memattr_init()` is called from `efi_config_parse_tables()` under only +`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()` +**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap` +**do not prevent the reservation.** Booting a fixed kernel is the only real fix. + +### The fix + +**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is +clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message +describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable +branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints +`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. + +The hardening landed exactly one release too late for Debian 13: the bug has existed since the +feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up +`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim +*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably +lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the +release lifetime, and the fix is small and already upstream. + +Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) +(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the +table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook +Studio G5**, 16 GiB → 1.5 GiB). + +### What NOT to do + +- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is + a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be + installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6. +- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full + R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries. +- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket: + they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They + may appear to "fix" it. They are not fixes. + +### Hypotheses killed, and why — so they are not re-run + +- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by + an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP + documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control. +- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by + `hp-bioscfg` on this platform. +- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by + measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`. + Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never + `memblock_add()`ed — which would have left `present` at ~0, not 3340288. +- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. +- **MTRR trim** — MTRRs near-identical to node 1, no trim warning. +- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. +- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly + explained: everything from `0xbeeb0018` upward. ## 1. Firmware / BIOS (per node, before OS install) > **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes -> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work -> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found +> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a +> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes", +> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found > at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. > > **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, @@ -154,6 +217,80 @@ To investigate, cheapest first: nodes match. - [ ] **Firmware version recorded** in §0 and consistent across all three nodes. +### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01 + +**There is no default BIOS password, and no factory default exists to look up.** HP business +desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we +observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5 +describes both purely as things the owner configures in Computer Setup. + +**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents +clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once). +**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely, +turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The +permissive default is the right setting for a homelab. + +#### CORRECTION — error `0x4` is NOT an authorisation failure + +I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That +was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI +`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a +different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have +unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting` +returns: + +| Code | Meaning | +|---|---| +| 0 | Success | +| 1 | Not Supported | +| **4** | **Failed** (generic) | +| 6 | Access Denied | + +So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and +the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither +of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up +in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`. + +**Most likely actual cause (inferred, testable):** with an empty password the driver emits an +*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when +blank (`Password = ""`). The driver only prepends that prefix for a *non-empty* string. So +setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would +work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this +firmware already makes the driver log `Prerequisites size value exceeded the maximum number of +elements supported or data may be malformed` at boot. + +**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and +`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM; +nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read. + +#### Recommendation + +1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff + `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV + win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10. +2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision + one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to + `current_password`, retry one benign attribute. Return `0` confirms the `` theory; `6` + would mean a password mismatch; another `4` means the driver is incompatible with R21 and the + fallback is `hp-repsetup`. +3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup + Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it + needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**. + Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a + security downgrade. +4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in + principle — the private key never leaves the workstation, each change is individually signed, with + anti-replay — but provisioning and every signature require Windows-only tooling. For three + machines whose firmware settings change roughly never, the operational cost far exceeds the risk + reduction. BCU is Windows/WinPE-only. +5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full + pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) — + functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via + `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is + persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the + password in cleartext on the command line — it would land in process listings and shell history + unless wrapped. Store it in the Ansible vault **and** offline. + ## 2. Disk layout — **OPEN DECISION, blocks the build** `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index a18e949..8b8a6bb 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI, EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM). -- **Never `become`s unless it must.** Only SMART needs root. +- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring + buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM + inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and + the privileged second pass re-runs the *same* collector with `--privileged` so the + parsing has one home. Without escalation those checks SKIP with the exact reason. ## What it checks @@ -44,15 +48,51 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes | | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` | | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters | +| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS | +| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) | Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded in task logic. + +## The EFI memattr check + +`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. + +HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13** +does not validate it before reserving memory from its contents — +`drivers/firmware/efi/memattr.c` multiplies two never-validated fields +(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the +table address to the top of RAM. Which of two kernel-log lines appears decides whether the +node keeps its memory, and it is **decided afresh on every boot**: + +| kernel log | meaning | verdict | +|---|---|---| +| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* | +| `efi: memattr: Unexpected EFI Memory Attributes table version ` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table | +| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working | +| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS | +| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected | +| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected | + +The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and +differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before +it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing. + +**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from +`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline +workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and +rebooting/CMOS/BIOS/USB changes only move where the table lands. + +`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node +whose kernel log cannot be read is still not unmeasured — and a future memory loss with a +different cause is still caught. + ## Output 1. A human summary printed in the play output. diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index 7a2fc72..eae98c9 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -45,6 +45,33 @@ node_health_thresholds: # Memory (percentage of MemTotal still available) mem_available_pct_warn: 15 mem_available_pct_fail: 5 + # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ---------------------------- + # Two independent measurements of the same question, both fed by memory.physical_total: + # + # unmanaged /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own + # memory map says exists, that never reached the allocator. Needs no root + # and no tools, works identically on metal and in a VM. + # shortfall MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is + # installed. Needs root, and only bare metal reports it. Independent of the + # kernel's memory map entirely, so it also catches RAM the firmware never + # handed the kernel in the first place. + # + # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and + # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2% + # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state. + # The healthy figure is the kernel image, memmap and firmware reservations; it is a + # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10% + # keeps clearance for that. There is no plausible tuning of a threshold in this band + # that changes any of those five verdicts. + # + # The DIMM shortfall runs a little higher than the unmanaged figure on the same node + # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and + # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the + # kernel's to manage. Same bands: the signal being caught is 83%, not 3%. + mem_unmanaged_pct_warn: 10 + mem_unmanaged_pct_fail: 25 + mem_dimm_shortfall_pct_warn: 10 + mem_dimm_shortfall_pct_fail: 25 # Temperatures (deg C, any sensor) temp_warn_c: 70 temp_fail_c: 85 diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index 37adf25..9f2c62b 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -97,10 +97,22 @@ def read_kv(path, sep=":"): return out +def which(tool): + """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits. + + `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the + unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed + but unreachable would be a false reason on a SKIP, which is the one thing this + collector may never do.""" + return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin") + + def run(argv, timeout=20): """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent.""" - if shutil.which(argv[0]) is None: + exe = which(argv[0]) + if exe is None: return None, "", "%s not installed" % argv[0] + argv = [exe] + list(argv[1:]) try: proc = subprocess.run( argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout @@ -284,9 +296,285 @@ def collect_memory(): edac["ce_total"] = ce_total edac["ue_total"] = ue_total mem["edac"] = edac + # Physically present vs actually usable — the cause-independent memory-loss signal. + mem["zones"] = collect_zoneinfo(mem["mem_total_kb"]) + # Firmware's own inventory of what is physically installed. Root-only; attempted + # here anyway so the unprivileged pass records the exact reason it could not read. + mem["devices"] = collect_memory_devices() return mem +# --- Memory the kernel has but never hands out -------------------------------------- +# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode, +# no DMI. Per zone it reports: +# spanned pages in the zone's address range, holes included +# present spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says +# physically exists here +# managed present MINUS everything memblock reserved before the buddy allocator was +# handed the rest (kernel image, memmap, initrd, crashkernel, firmware +# reservations). sum(managed) IS MemTotal, exactly, on every node measured. +# So `present - managed` is, by construction, "RAM this machine has that the OS will +# never allocate", whatever the reason. On a healthy node it is the couple of hundred +# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and +# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% — +# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude +# separation, which is why this needs no clever tuning to be trustworthy. +# +# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if +# a future firmware, kernel or hypervisor lost the memory for an entirely different +# reason, and it needs neither root nor a parseable kernel log. +def collect_zoneinfo(mem_total_kb): + raw = read_text("/proc/zoneinfo", optional=False) + page_kb = os.sysconf("SC_PAGE_SIZE") // 1024 + out = { + "available": raw is not None, + "reason": None if raw is not None else "/proc/zoneinfo could not be read", + "page_size_kb": page_kb, + "zones": [], + "present_kb": None, + "managed_kb": None, + "unmanaged_kb": None, + "unmanaged_pct": None, + } + if raw is None: + return out + node = zone = None + cur = {} + present_pages = managed_pages = 0 + for line in raw.splitlines(): + fields = line.split() + if line.startswith("Node ") and "zone" in line: + # "Node 0, zone Normal" + if cur: + out["zones"].append(cur) + node, zone = fields[1].rstrip(","), fields[-1] + cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None} + elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"): + try: + cur[fields[0]] = int(fields[1]) + except ValueError: + cur[fields[0]] = None + if cur: + out["zones"].append(cur) + # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay + # null rather than emitting a confident 0. + if not out["zones"]: + out["available"] = False + out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it" + return out + for z in out["zones"]: + present_pages += z["present"] or 0 + managed_pages += z["managed"] or 0 + out["present_kb"] = present_pages * page_kb + out["managed_kb"] = managed_pages * page_kb + out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"] + if out["present_kb"] > 0: + out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1) + # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the + # two readings disagree and the discrepancy itself is worth showing. + out["matches_mem_total"] = ( + None if (mem_total_kb is None or out["managed_kb"] is None) + else abs(out["managed_kb"] - mem_total_kb) <= page_kb + ) + return out + + +# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) ----------------------------------- +# The independent answer to "how much RAM is physically in this box", from the firmware's +# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads +# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that +# failure is recorded as the SKIP reason — never as zero DIMMs. +# +# On a VM there is usually either no type-17 structure at all or a single synthetic entry +# that merely restates the configured guest RAM; both are handled by reporting what was +# found and letting the check SKIP when nothing populated was reported. +DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024} + + +def collect_memory_devices(): + out = { + "available": False, + "reason": None, + "devices": [], + "slots": 0, + "populated_slots": 0, + "installed_kb": None, + } + if which("dmidecode") is None: + out["reason"] = ("dmidecode is not installed on this node; it ships with " + "roles/node_baseline (node_baseline_packages). " + "This role never installs packages.") + return out + rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30) + if rc != 0: + out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % ( + rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output") + return out + # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which + # may or may not hold a module ("Size: No Module Installed"). + total_kb = 0 + saw_size = False + for block in stdout.split("\n\n"): + if "Memory Device" not in block: + continue + fields = {} + for line in block.splitlines(): + key, sep, value = line.partition(":") + if sep: + fields[key.strip()] = value.strip() + if "Size" not in fields: + continue + out["slots"] += 1 + size_kb = None + parts = fields["Size"].split() + if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS: + try: + size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()] + except ValueError: + size_kb = None + if size_kb: + saw_size = True + total_kb += size_kb + out["populated_slots"] += 1 + out["devices"].append( + { + "locator": fields.get("Locator"), + "bank_locator": fields.get("Bank Locator"), + # Verbatim, so an unparsed spelling is visible rather than silently null. + "size_raw": fields["Size"], + "size_kb": size_kb, + "type": fields.get("Type"), + "speed": fields.get("Speed"), + "configured_speed": fields.get("Configured Memory Speed"), + "manufacturer": fields.get("Manufacturer"), + "part_number": fields.get("Part Number"), + "rank": fields.get("Rank"), + # Serial deliberately NOT collected: the artifact already carries drive + # serials and a DIMM serial adds identifying data no check reads. + } + ) + out["available"] = saw_size + if not saw_size: + out["reason"] = ("dmidecode ran but reported no populated memory device " + "(%d slot structure(s) found) — typical of a VM with no SMBIOS " + "type 17 inventory" % out["slots"]) + else: + out["installed_kb"] = total_kb + return out + + +# --- Kernel ring buffer: the EFI Memory Attributes Table ---------------------------- +# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative +# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes +# Table, and Linux < 6.13 does not validate it before reserving memory from its contents +# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s +# the product). The kernel prints exactly one of two lines, and WHICH one decides whether +# the node keeps its RAM: +# +# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read +# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW. +# "Unexpected EFI Memory Attributes table version " the garbage version read > 2, so +# the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the +# firmware is still publishing a corrupt table and the outcome is decided afresh by +# where the table lands on every boot. That is luck, not immunity. +# +# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots +# before it lost 13.3 GiB), which is precisely why it is detected on every health run +# instead of diagnosed once. +# +# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on +# every node here). The unprivileged pass therefore records WHY it could not read, and +# roles/node_health re-runs this collector with `--privileged` when escalation works. +BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:") +EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)") +MEMATTR_FAILED_RE = re.compile( + r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I) +MEMATTR_VERSION_RE = re.compile( + r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I) +MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I) +# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table +# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is +# captured separately, above, and quoting it as evidence would bury the one line that +# matters in 200 characters of unrelated table addresses. +MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I) +# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line +# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict. +MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available") +MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved") + + +def collect_klog(): + out = { + "available": False, + "reason": None, + "boot_covered": None, + "memattr_state": None, + "memattr_published": None, + "memattr_table_address": None, + "memattr_version": None, + "memattr_lines": [], + "memory_line": None, + } + rc, stdout, stderr = run(["dmesg"], timeout=30) + if rc is None: + out["reason"] = "dmesg is not installed on this node" + return out + if rc != 0: + detail = (stderr or stdout).strip().splitlines() + out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % ( + rc, os.geteuid(), detail[-1] if detail else "no output", + " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG" + if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "") + return out + + out["available"] = True + # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every + # memattr line is printed in the first fraction of a second. If the boot header is + # gone then so may be a memattr line that was printed — absence would prove nothing, + # and the check must SKIP rather than PASS. This is the difference between + # "unaffected" and "could not tell", and the role is not allowed to blur it. + out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout)) + + published = EFI_CONFIG_RE.search(stdout) + if published: + out["memattr_published"] = True + out["memattr_table_address"] = published.group(1) + elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M): + # UEFI booted and the config-table line is present, but it lists no MEMATTR=. + out["memattr_published"] = False + # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None. + + for line in stdout.splitlines(): + if MEMATTR_LINE_RE.search(line): + out["memattr_lines"].append(line.strip()) + joined = "\n".join(out["memattr_lines"]) + if MEMATTR_CORRUPT_RE.search(joined): + # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected. + out["memattr_state"] = "corrupt_rejected" + elif MEMATTR_FAILED_RE.search(joined): + out["memattr_state"] = "failed_to_map" + addr = MEMATTR_FAILED_RE.search(joined).group(1) + if addr: + out["memattr_table_address"] = addr + elif MEMATTR_VERSION_RE.search(joined): + out["memattr_state"] = "bad_version" + out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1)) + else: + out["memattr_state"] = "clean" + + mem_line = MEMORY_LINE_RE.search(stdout) + if mem_line: + full = [l for l in stdout.splitlines() if mem_line.group(0) in l] + reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None + out["memory_line"] = { + "available_kb": int(mem_line.group(1)), + "total_kb": int(mem_line.group(2)), + "reserved_kb": int(reserved.group(1)) if reserved else None, + "raw": full[0].strip() if full else None, + } + return out + + # --- Thermals ----------------------------------------------------------------------- def collect_thermal(): zones = [] @@ -708,15 +996,34 @@ def collect_os(): def collect_tools(): tools = {} for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"): - tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin") + tools[tool] = which(tool) return tools def main(): + # --privileged: re-collect ONLY the fields that genuinely need root, for the second + # pass roles/node_health makes when (and only when) escalation is proven to work + # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place — + # re-implementing the kernel-log grep in Jinja would be a second source of truth for + # the same fact, and the two would drift. + if "--privileged" in sys.argv[1:]: + doc = { + "schema": SCHEMA, + "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "collector_uid": os.geteuid(), + "klog": collect_klog(), + "memory": {"devices": collect_memory_devices()}, + } + doc["unreadable"] = UNREADABLE + json.dump(doc, sys.stdout, indent=2, sort_keys=True) + sys.stdout.write("\n") + return + doc = { "schema": SCHEMA, "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), "collector_uid": os.geteuid(), + "klog": collect_klog(), "os": collect_os(), "dmi": collect_dmi(), "cpu": collect_cpu(), diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 3023397..7553959 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -387,6 +387,83 @@ ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available (' ~ avail_pct | string ~ '%)') }} +# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the +# one cause we have actually hit here; this one asks the blunter question — "is this +# machine using the RAM it has?" — and answers it from two sources that share NOTHING +# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some +# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml. +- name: Assess — usable memory vs the memory this machine physically has + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'memory.physical_total', + 'title': 'Usable memory vs installed/present memory (independent of the kernel log)', + 'status': phys_status, + 'detail': phys_detail }] }}" + vars: + ansible_become: false + th: "{{ node_health_thresholds }}" + z: "{{ node_health_facts.memory.zones }}" + dev: "{{ node_health_facts.memory.devices }}" + m: "{{ node_health_facts.memory }}" + + # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it + # actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no + # tools, identical on metal and in a VM — so this half is available almost always. + z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}" + z_status: >- + {{ 'SKIP' if z_pct is not number + else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail + else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }} + + # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev + # VMs and wherever escalation is unavailable — with dmidecode's own reason attached, + # never as a zero DIMM count. Independent of the kernel's memory map entirely. + inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}" + d_pct: >- + {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1) + if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0) + else none }} + d_status: >- + {{ 'SKIP' if d_pct is not number + else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail + else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }} + + # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two + # SKIPs must never add up to a PASS. + phys_status: >- + {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last) + | default('SKIP', true) }} + # What the kernel log says, IF it was read — quoted here only as attribution for a + # shortfall this check measured on its own. The verdict above never depends on it. + phys_cause: >- + {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check + firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b') + if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map' + else ('. The kernel log was NOT read here, so this shortfall is unattributed — + see check firmware.efi_memattr') + if not (node_health_facts.klog.available | bool) + else '' }} + phys_detail: >- + {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed ' + ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — ' + ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT + was never handed to the allocator (/proc/zoneinfo)') + if z_pct is number + else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true)))) + ~ '. ' + ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across ' + ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) (' + ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', ')) + ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string + ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall') + if d_pct is number + else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true)))) + ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware + reservations and (on this platform) 64 MiB of graphics stolen memory. + A shortfall of this size is memory the machine HAS and cannot use') + if phys_status in ['WARN', 'FAIL'] else '') + ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }} + - name: Assess — swap (kubelet requires it off) ansible.builtin.set_fact: node_health_checks: "{{ node_health_checks + [{ @@ -674,3 +751,127 @@ ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')' ~ (' — serial numbers not read (root-only DMI fields)' if d.product_serial is none else '')) }} + +# --- EFI Memory Attributes Table ------------------------------------------------------ +# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here; +# what follows is only what this check needs to be read correctly. +# +# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does +# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c +# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s +# the product — everything from the table address to the top of RAM. The kernel prints +# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field +# happens to read > 2: +# +# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel +# TRUSTED the table. The node has lost the memory. FAIL. +# "Unexpected EFI Memory Attributes table version " version read > 2, the kernel +# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a +# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS: +# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself +# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one +# unlucky boot from losing 83% of its RAM is not passing a health check. +# +# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC, +# so it cannot be diagnosed once. It has to be measured on every health run. +# +# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected +# ONLY if the kernel log was actually read AND still reaches back to early boot — every +# memattr diagnostic is printed in the first fraction of a second, and the ring buffer +# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and +# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a +# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel +# log at all, so a node whose log is unreadable is still not unmeasured. +- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse) + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'firmware.efi_memattr', + 'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel', + 'status': memattr_status, + 'detail': memattr_detail }] }}" + vars: + ansible_become: false + k: "{{ node_health_facts.klog }}" + z: "{{ node_health_facts.memory.zones }}" + ma_state: "{{ k.memattr_state | default(none) }}" + memattr_status: >- + {{ 'SKIP' if not (k.available | bool) + else ('FAIL' if ma_state == 'failed_to_map' + else ('WARN' if ma_state == 'bad_version' + else ('PASS' if ma_state == 'corrupt_rejected' + else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }} + # One remediation, quoted by both findings. NOTE what it must NOT say: there is no + # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out + # could take effect. Suggesting one would be a fix that quietly does nothing. + memattr_fix: >- + FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) — + upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly + bogus", first released in v6.13 and absent from v6.12 and from the current + linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}. + There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi= + bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or + unplugging USB only change where the table lands and may APPEAR to fix it — they are + a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b. + # The symptom, measured independently of this log (see memory.physical_total). + memattr_corroboration: >- + {{ (' Measured independently of this log: MemTotal ' + ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' + ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. ' + ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away + from the allocator (/proc/zoneinfo) — check memory.physical_total.') + if ((z.available | bool) and z.unmanaged_pct is number) else '' }} + # The kernel log lines themselves, verbatim. A verdict the operator cannot check + # against the evidence is not much better than a wrong one. + memattr_evidence: >- + {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | '))) + if (k.memattr_lines | default([]) | length) > 0 else '') + ~ ((' At boot the kernel reported ' + ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' + ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, ' + ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved' + ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.') + if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number) + else '') }} + memattr_detail: >- + {%- if not (k.available | bool) -%} + the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}. + Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic + lives only in dmesg.{{ memattr_corroboration }} + {%- elif ma_state == 'failed_to_map' -%} + THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory + Attributes table at {{ k.memattr_table_address | default('an unreported address') }} + and reserved memory from the corrupt table anyway — everything from that address + upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }} + {%- elif ma_state == 'bad_version' -%} + EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table + here too; this boot survived only because its garbage version field read + {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out + before reserving anything. The outcome is re-decided on EVERY boot: if that field + reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right + now and this node is one unlucky boot from the failure state.{{ memattr_evidence }} + {{ memattr_fix }} + {%- elif ma_state == 'corrupt_rejected' -%} + the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel + validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is + intact and stays intact across reboots; no action.{{ memattr_evidence }} + {%- elif not (k.boot_covered | bool) -%} + NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches + back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in + the ring buffer, which wraps). Every memattr diagnostic is printed within the first + fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a + pass. Read /var/log/kern.log for this boot, or re-run after a + reboot.{{ memattr_corroboration }} + {%- elif k.memattr_published is sameas true -%} + the firmware publishes an EFI Memory Attributes table at + {{ k.memattr_table_address | default('an unreported address') }} and the kernel + accepted it without complaint, in a log that still reaches back to early boot. Not + affected on this boot. + {%- elif k.memattr_published is sameas false -%} + this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line + lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF + guest. + {%- else -%} + no UEFI config-table line in the kernel log at all — this node did not boot via UEFI, + so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early + boot, so that absence is measured, not assumed. + {%- endif -%} diff --git a/infra/ansible/roles/node_health/tasks/collect_privileged.yml b/infra/ansible/roles/node_health/tasks/collect_privileged.yml new file mode 100644 index 0000000..5afb552 --- /dev/null +++ b/infra/ansible/roles/node_health/tasks/collect_privileged.yml @@ -0,0 +1,62 @@ +--- +# The SECOND pass of the collector, run with escalation, for the two facts that are +# genuinely root-gated: +# +# * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an +# unprivileged read gets EPERM. This is where the EFI Memory Attributes Table +# diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference +# between a node that has lost 13.3 GiB and one that has not. +# * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed +# DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. +# +# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a +# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line- +# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND +# source of truth for the same fact, and the two would drift. `--privileged` makes the +# collector emit exactly those subtrees and nothing else, so the parsing stays in one +# place and is identical whichever pass produced it. +# +# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass +# actually failed to read one of them (main.yml checks both). On a node where +# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule +# is that everything readable WITHOUT root is read without root. +# +# If escalation is unavailable the checks that consume these SKIP with the exact reason +# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line" +# and "could not read the kernel log" are different answers and the role says which. + +- name: Re-collect the root-only facts with escalation + ansible.builtin.script: + cmd: node_health_collect.py --privileged + executable: /usr/bin/python3 + become: true + vars: + ansible_become: true + register: node_health_privileged_raw + changed_when: false # strictly read-only; it can never change the node + +- name: Assert the privileged payload is the schema this role understands + ansible.builtin.assert: + that: + - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1' + fail_msg: >- + Privileged collector pass returned an unexpected schema; role and collector are + out of step. + quiet: true + vars: + ansible_become: false + +# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive +# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine +# would REPLACE it and throw away the reasons the first pass recorded. +- name: Merge the privileged facts over the unprivileged ones + ansible.builtin.set_fact: + node_health_facts: >- + {{ node_health_facts | combine({ + 'klog': priv.klog, + 'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}), + 'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])), + }) }} + vars: + ansible_become: false + priv: "{{ node_health_privileged_raw.stdout | from_json }}" diff --git a/infra/ansible/roles/node_health/tasks/main.yml b/infra/ansible/roles/node_health/tasks/main.yml index a57ffc0..2a0d07c 100644 --- a/infra/ansible/roles/node_health/tasks/main.yml +++ b/infra/ansible/roles/node_health/tasks/main.yml @@ -4,9 +4,11 @@ # ORDER OF OPERATIONS # 1. Collect everything readable WITHOUT root (one script, one round trip). # 2. Probe whether privilege escalation actually works on this host. -# 3. Only if it does AND smartmontools is installed: read SMART. -# 4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks. -# 5. Render the human summary + write the JSON artifact. +# 3. Only if it does, and only for what the first pass could NOT read: re-run the same +# collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory). +# 4. Only if it does AND smartmontools is installed: read SMART. +# 5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks. +# 6. Render the human summary + write the JSON artifact. # # BECOME HANDLING — the subtle bit, read before editing. # `ansible_become` set as an inventory/group var (which is exactly what @@ -43,6 +45,16 @@ ansible.builtin.include_tasks: probe_privilege.yml when: node_health_try_privileged | bool +# Only when the unprivileged pass actually failed to read one of them. On a node with +# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped +# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml). +- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs) + ansible.builtin.include_tasks: collect_privileged.yml + when: + - node_health_root_available | bool + - (not (node_health_facts.klog.available | bool)) + or (not (node_health_facts.memory.devices.available | bool)) + - name: Read SMART data (privileged) ansible.builtin.include_tasks: collect_smart.yml when: diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 23a32ba..732af45 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -45,6 +45,15 @@ {% endif %} Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }} +{% set mz = node_health_facts.memory.zones %} +{% set md = node_health_facts.memory.devices %} + Present RAM {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }} + DIMMs {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }} +{% for dimm in md.devices | default([]) if dimm.size_kb %} + {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }} +{% endfor %} +{% set klog = node_health_facts.klog %} + EFI memattr {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }} Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }} --------------------------------------------------------------------------------