From 49d82491ddcf327a4012901a6a3e04d16d03e89f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:14:14 -0500 Subject: [PATCH] =?UTF-8?q?docs(prod):=20node=202=20memory=20SOLVED=20?= =?UTF-8?q?=E2=80=94=20Linux=206.12=20EFI=20memattr=20bug,=20NOT=20a=20har?= =?UTF-8?q?dware=20fault?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping, and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine. ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c never validates num_entries or desc_size: if (tbl->version > 2) { pr_warn(...); goto unmap; } tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; memblock_reserve(efi_mem_attr_table, tbl_size); Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two garbage fields and reserves GIGABYTES from the table address to the top of RAM. THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline — booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2: 15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version 1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481) pages = 2,753,284 kB exactly, Normal contributing zero. BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version 1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY. It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all (Proxmox OVMF) and are unaffected. FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports. DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting, disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where the table lands and may LOOK like fixes. Records every killed hypothesis with its disproof so none of them get re-run. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 155 ++++++++++++++++++------------------- 1 file changed, 75 insertions(+), 80 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f79ae2f..744c8bc 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -44,100 +44,95 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- -## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB +## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01) -Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install -more RAM until this is understood** — see the warning at the end. +> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory +> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine. +> Keeping the correction visible because the wrong framing nearly cost money and a board RMA. -| Evidence | Reading | -|---|---| -| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | -| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | -| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | -| `last_pfn` | two values, `0x42f800` then `0xc3c10` — **NOT a cap**, see the correction below | -| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | -| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | -| Memory blocks | 127 online, 128 MiB block size | +`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware +publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before +reserving memory based on its contents.** Fixed upstream in **6.13**. -**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The -kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and -everything above 4 GiB is present-but-unmanaged. +### The evidence that breaks it open -> **CORRECTION (2026-09-01).** An earlier version of this section cited the `last_pfn` line as -> evidence of a memory cap, and called the pattern the signature of "memory remapping above 4 GB -> not being applied". **The `last_pfn` reading was wrong.** x86 boot prints `last_pfn` twice: first -> `max_pfn` from the full e820, then `max_low_pfn` — the end of RAM *below 4 GiB*. The e820 low -> usable region ends at `0xc3c0ffff`, so the next PFN is exactly `0xc3c10`, which is the second -> value observed. It matches to the byte and is entirely normal on any machine with >4 GiB. A -> routine boot message was presented as a smoking gun. -> -> The real anomaly is narrower than first stated: the `Normal` zone is `present` but `managed 0`, -> 13.3 GiB is reserved, and **nothing in the boot log explains by what**. +The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full +15.9 GiB two boots earlier**: -Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, -identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it -is per-machine state. +| Boot | `Memory: … available` | reserved | memattr line | +|---|---|---|---| +| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` | +| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | +| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | -**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` -kernel driver, which exposes **215 BIOS attributes** read/write at -`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0). +The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in +that shape, and none is fixed by a reboot. -**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was -made on 2026-09-01 and the FIRMWARE refused it: -`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. -HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent -— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. -Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS -Setup Password is deliberately provisioned (see §1). +### Mechanism (verified against kernel source) -What the comparison ruled OUT as the cause: -- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and - `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. -- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on - either node, so it cannot be a mis-set option. -- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, - which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. -- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory - blocks report `online`. The memory reaches the kernel and is then reserved. -- **Integrated graphics is NOT the cause** (operator hypothesis, tested 2026-09-01). `/proc/iomem` - shows `c8800000-cc7fffff : Graphics Stolen Memory` = **exactly 64 MiB**, matching the BIOS - `Video Memory Size=64 MB` setting, and it sits **below** the 4 GiB line. The UHD 630 with `i915` - bound takes precisely what it is configured to take. It cannot account for 13.3 GiB, and it is - not even in the affected address range. +`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`: -Real BIOS differences found (node 1 = works, node 2 = broken): +```c +if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; } +tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; +memblock_reserve(efi_mem_attr_table, tbl_size); +``` -| Attribute | node 1 | node 2 | +**`num_entries` and `desc_size` are never validated.** With a corrupt table: +- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)* +- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s + the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)* + +That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = +DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. + +### BOTH prod nodes are affected — node 1 has only been lucky + +| Node | memattr version field | Outcome | |---|---|---| -| **After Power Loss** | Power On | **Power Off** ← violates §1 | -| **Fast Boot** | Disable | **Enable** ← leading hypothesis | -| Legacy Boot Options | Disable | Enable | -| TPM Activation Policy | F1 to Boot | Allow user to reject | -| Microsoft/Windows UEFI CA 2023 | Enable | Disable | -| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | +| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** | +| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB | +| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB | -**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* -rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale -training data is a plausible route to memory being present, inventoried, handed over — and then -reserved. It is also the one memory-relevant setting that differs from the working node. This is -a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write -and a reboot, which needs operator sanction. +Node 1 is one unlucky boot away from the identical fault. **Fix both.** -To investigate, cheapest first: -- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - **Must be done in F10** — the firmware refuses programmatic writes (above). -- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and - node 2 currently violates it. -- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. -- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot - vs board in one move. -- [ ] `memtest86+` if the above do not resolve it. +### The fix -> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS -> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. -> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` -> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. +**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is +clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message +describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable +branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints +`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. + +Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) +(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the +table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook +Studio G5**, 16 GiB → 1.5 GiB). + +### What NOT to do + +- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is + a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be + installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6. +- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full + R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries. +- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket: + they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They + may appear to "fix" it. They are not fixes. + +### Hypotheses killed, and why — so they are not re-run + +- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by + an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP + documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control. +- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by + `hp-bioscfg` on this platform. +- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. +- **MTRR trim** — MTRRs near-identical to node 1, no trim warning. +- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. +- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly + explained: everything from `0xbeeb0018` upward. ## 1. Firmware / BIOS (per node, before OS install)