From 47f02c6e06f4807de9a60be659a15e4bedc69850 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:16:22 -0500 Subject: [PATCH] docs(prod): second independent confirmation + no cmdline workaround exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A second research pass reached the same root cause by a different route and pinned the arithmetic. Recording it because convergence from two independent lines is what turns this from a good hypothesis into a settled diagnosis. MECHANISM TO THE BYTE: - the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls plain System RAM, so the pre-Linux boot chain overwrites it. - tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31 lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base, so EVERYTHING above 0xbeeb0018 is reserved. - memblock_free_all then hands the allocator nothing above that line while present_pages (from memblock.memory) is untouched -> present 3340288 / managed 0. - 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions(); it is positive PROOF the reservation already happened, a consequence rather than the cause. CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM. Identical 0x2a980 allocation, opposite placement. THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the reservation. A fixed kernel is the only real fix. EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT: /sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and is never memblock_add()ed, which would have left present at ~0 rather than 3340288. Also records that the hardening landed exactly one release late for Debian 13 (bug present since 2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 46 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 744c8bc..6a09c36 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -96,6 +96,41 @@ DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contri Node 1 is one unlucky boot away from the identical fault. **Fix both.** +### Mechanism, traced to the byte (second, independent confirmation) + +A second research pass reached the same root cause by a different route and pinned the arithmetic: + +- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The + kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that + `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it. +- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and + when bit 31 lands the result is **negative**. +- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to + ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical + address above `0xbeeb0018` is reserved.** +- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages` + (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`. +- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by + `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is + therefore positive proof the reservation already happened** — it is a consequence, not the cause. + +The cleanest A/B tell between the two nodes is one line of dmesg: + +| | `NODE_DATA(0) allocated` | meaning | +|---|---|---| +| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation | +| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement | + +Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs +node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. + +### THERE IS NO KERNEL CMDLINE WORKAROUND + +`efi_memattr_init()` is called from `efi_config_parse_tables()` under only +`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()` +**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap` +**do not prevent the reservation.** Booting a fixed kernel is the only real fix. + ### The fix **Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is @@ -104,6 +139,13 @@ describes this bug verbatim. Verified absent from v6.12 and from the current `li branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. +The hardening landed exactly one release too late for Debian 13: the bug has existed since the +feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up +`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim +*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably +lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the +release lifetime, and the fix is small and already upstream. + Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) (Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook @@ -128,6 +170,10 @@ Studio G5**, 16 GiB → 1.5 GiB). - **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by `hp-bioscfg` on this platform. - **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by + measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`. + Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never + `memblock_add()`ed — which would have left `present` at ~0, not 3340288. - **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. - **MTRR trim** — MTRRs near-identical to node 1, no trim warning. - **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.