diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 744c8bc..6a09c36 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -96,6 +96,41 @@ DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contri Node 1 is one unlucky boot away from the identical fault. **Fix both.** +### Mechanism, traced to the byte (second, independent confirmation) + +A second research pass reached the same root cause by a different route and pinned the arithmetic: + +- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The + kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that + `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it. +- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and + when bit 31 lands the result is **negative**. +- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to + ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical + address above `0xbeeb0018` is reserved.** +- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages` + (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`. +- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by + `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is + therefore positive proof the reservation already happened** — it is a consequence, not the cause. + +The cleanest A/B tell between the two nodes is one line of dmesg: + +| | `NODE_DATA(0) allocated` | meaning | +|---|---|---| +| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation | +| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement | + +Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs +node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. + +### THERE IS NO KERNEL CMDLINE WORKAROUND + +`efi_memattr_init()` is called from `efi_config_parse_tables()` under only +`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()` +**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap` +**do not prevent the reservation.** Booting a fixed kernel is the only real fix. + ### The fix **Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is @@ -104,6 +139,13 @@ describes this bug verbatim. Verified absent from v6.12 and from the current `li branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. +The hardening landed exactly one release too late for Debian 13: the bug has existed since the +feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up +`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim +*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably +lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the +release lifetime, and the fix is small and already upstream. + Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) (Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook @@ -128,6 +170,10 @@ Studio G5**, 16 GiB → 1.5 GiB). - **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by `hp-bioscfg` on this platform. - **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by + measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`. + Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never + `memblock_add()`ed — which would have left `present` at ~0, not 3340288. - **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. - **MTRR trim** — MTRRs near-identical to node 1, no trim warning. - **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.