diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f79ae2f..744c8bc 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -44,100 +44,95 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- -## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB +## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01) -Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install -more RAM until this is understood** — see the warning at the end. +> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory +> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine. +> Keeping the correction visible because the wrong framing nearly cost money and a board RMA. -| Evidence | Reading | -|---|---| -| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | -| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | -| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | -| `last_pfn` | two values, `0x42f800` then `0xc3c10` — **NOT a cap**, see the correction below | -| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | -| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | -| Memory blocks | 127 online, 128 MiB block size | +`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware +publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before +reserving memory based on its contents.** Fixed upstream in **6.13**. -**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The -kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and -everything above 4 GiB is present-but-unmanaged. +### The evidence that breaks it open -> **CORRECTION (2026-09-01).** An earlier version of this section cited the `last_pfn` line as -> evidence of a memory cap, and called the pattern the signature of "memory remapping above 4 GB -> not being applied". **The `last_pfn` reading was wrong.** x86 boot prints `last_pfn` twice: first -> `max_pfn` from the full e820, then `max_low_pfn` — the end of RAM *below 4 GiB*. The e820 low -> usable region ends at `0xc3c0ffff`, so the next PFN is exactly `0xc3c10`, which is the second -> value observed. It matches to the byte and is entirely normal on any machine with >4 GiB. A -> routine boot message was presented as a smoking gun. -> -> The real anomaly is narrower than first stated: the `Normal` zone is `present` but `managed 0`, -> 13.3 GiB is reserved, and **nothing in the boot log explains by what**. +The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full +15.9 GiB two boots earlier**: -Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, -identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it -is per-machine state. +| Boot | `Memory: … available` | reserved | memattr line | +|---|---|---|---| +| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` | +| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | +| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | -**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` -kernel driver, which exposes **215 BIOS attributes** read/write at -`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0). +The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in +that shape, and none is fixed by a reboot. -**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was -made on 2026-09-01 and the FIRMWARE refused it: -`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. -HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent -— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. -Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS -Setup Password is deliberately provisioned (see §1). +### Mechanism (verified against kernel source) -What the comparison ruled OUT as the cause: -- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and - `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. -- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on - either node, so it cannot be a mis-set option. -- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, - which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. -- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory - blocks report `online`. The memory reaches the kernel and is then reserved. -- **Integrated graphics is NOT the cause** (operator hypothesis, tested 2026-09-01). `/proc/iomem` - shows `c8800000-cc7fffff : Graphics Stolen Memory` = **exactly 64 MiB**, matching the BIOS - `Video Memory Size=64 MB` setting, and it sits **below** the 4 GiB line. The UHD 630 with `i915` - bound takes precisely what it is configured to take. It cannot account for 13.3 GiB, and it is - not even in the affected address range. +`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`: -Real BIOS differences found (node 1 = works, node 2 = broken): +```c +if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; } +tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; +memblock_reserve(efi_mem_attr_table, tbl_size); +``` -| Attribute | node 1 | node 2 | +**`num_entries` and `desc_size` are never validated.** With a corrupt table: +- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)* +- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s + the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)* + +That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = +DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. + +### BOTH prod nodes are affected — node 1 has only been lucky + +| Node | memattr version field | Outcome | |---|---|---| -| **After Power Loss** | Power On | **Power Off** ← violates §1 | -| **Fast Boot** | Disable | **Enable** ← leading hypothesis | -| Legacy Boot Options | Disable | Enable | -| TPM Activation Policy | F1 to Boot | Allow user to reject | -| Microsoft/Windows UEFI CA 2023 | Enable | Disable | -| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | +| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** | +| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB | +| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB | -**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* -rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale -training data is a plausible route to memory being present, inventoried, handed over — and then -reserved. It is also the one memory-relevant setting that differs from the working node. This is -a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write -and a reboot, which needs operator sanction. +Node 1 is one unlucky boot away from the identical fault. **Fix both.** -To investigate, cheapest first: -- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - **Must be done in F10** — the firmware refuses programmatic writes (above). -- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and - node 2 currently violates it. -- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. -- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot - vs board in one move. -- [ ] `memtest86+` if the above do not resolve it. +### The fix -> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS -> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. -> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` -> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. +**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is +clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message +describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable +branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints +`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. + +Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) +(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the +table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook +Studio G5**, 16 GiB → 1.5 GiB). + +### What NOT to do + +- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is + a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be + installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6. +- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full + R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries. +- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket: + they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They + may appear to "fix" it. They are not fixes. + +### Hypotheses killed, and why — so they are not re-run + +- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by + an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP + documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control. +- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by + `hp-bioscfg` on this platform. +- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. +- **MTRR trim** — MTRRs near-identical to node 1, no trim warning. +- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. +- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly + explained: everything from `0xbeeb0018` upward. ## 1. Firmware / BIOS (per node, before OS install)