From 3a88abb9589d4a3ca1869b185e3e1d03d994af2c Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 21:54:12 -0500 Subject: [PATCH 1/6] docs(prod): iGPU ruled out as the cause; CORRECT my last_pfn misreading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator asked whether node 2's missing memory could be reserved for integrated graphics. TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB, matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630 (i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is not even in the affected address range. CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the 'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn, the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the 'memory remapping above 4 GB not being applied' signature claim rested partly on it. The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0', 13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health, firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training data cached by Fast Boot — testable with one F10 toggle. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 30714f1..53c3b58 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -54,15 +54,25 @@ more RAM until this is understood** — see the warning at the end. | DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | | BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | | Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | -| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | +| `last_pfn` | two values, `0x42f800` then `0xc3c10` — **NOT a cap**, see the correction below | | Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | | `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | | Memory blocks | 127 online, 128 MiB block size | **The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and -everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping -above 4 GB not being applied**, not of failing memory. +everything above 4 GiB is present-but-unmanaged. + +> **CORRECTION (2026-09-01).** An earlier version of this section cited the `last_pfn` line as +> evidence of a memory cap, and called the pattern the signature of "memory remapping above 4 GB +> not being applied". **The `last_pfn` reading was wrong.** x86 boot prints `last_pfn` twice: first +> `max_pfn` from the full e820, then `max_low_pfn` — the end of RAM *below 4 GiB*. The e820 low +> usable region ends at `0xc3c0ffff`, so the next PFN is exactly `0xc3c10`, which is the second +> value observed. It matches to the byte and is entirely normal on any machine with >4 GiB. A +> routine boot message was presented as a smoking gun. +> +> The real anomaly is narrower than first stated: the `Normal` zone is `present` but `managed 0`, +> 13.3 GiB is reserved, and **nothing in the boot log explains by what**. Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it @@ -90,6 +100,11 @@ What the comparison ruled OUT as the cause: which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. - **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory blocks report `online`. The memory reaches the kernel and is then reserved. +- **Integrated graphics is NOT the cause** (operator hypothesis, tested 2026-09-01). `/proc/iomem` + shows `c8800000-cc7fffff : Graphics Stolen Memory` = **exactly 64 MiB**, matching the BIOS + `Video Memory Size=64 MB` setting, and it sits **below** the 4 GiB line. The UHD 630 with `i915` + bound takes precisely what it is configured to take. It cannot account for 13.3 GiB, and it is + not even in the affected address range. Real BIOS differences found (node 1 = works, node 2 = broken): From 5ea9c93f58651f188dc1d8fd0a3bade75e496187 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:10:07 -0500 Subject: [PATCH 2/6] docs(prod): BIOS password answer + CORRECT the 0x4 'auth failure' reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator asked for the default BIOS password. ANSWER: there is none, and no factory default exists to look up. HP business desktops ship with no Setup and no Power-On password; the is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the physical bypass and turns a 5-minute fix into a signed per-serial HP service case. CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the "SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper, SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream bug worth reporting to platform-driver-x86. Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED empty UTF-16 string, while every HP-documented caller passes Password="" even when blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised. Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is write-only into a buffer nothing ever reads. Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted 215 as the total; it was the non-empty subset. Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it already caught real drift); treat a Setup password as a deliberate experiment on node 3 only; hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU at this scale. Serves: R-ENV-03, R-SEC-01. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 79 +++++++++++++++++++++++++++++++++++++- 1 file changed, 77 insertions(+), 2 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 53c3b58..f79ae2f 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -142,8 +142,9 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) > **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes -> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work -> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found +> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a +> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes", +> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found > at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. > > **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, @@ -169,6 +170,80 @@ To investigate, cheapest first: nodes match. - [ ] **Firmware version recorded** in §0 and consistent across all three nodes. +### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01 + +**There is no default BIOS password, and no factory default exists to look up.** HP business +desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we +observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5 +describes both purely as things the owner configures in Computer Setup. + +**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents +clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once). +**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely, +turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The +permissive default is the right setting for a homelab. + +#### CORRECTION — error `0x4` is NOT an authorisation failure + +I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That +was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI +`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a +different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have +unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting` +returns: + +| Code | Meaning | +|---|---| +| 0 | Success | +| 1 | Not Supported | +| **4** | **Failed** (generic) | +| 6 | Access Denied | + +So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and +the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither +of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up +in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`. + +**Most likely actual cause (inferred, testable):** with an empty password the driver emits an +*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when +blank (`Password = ""`). The driver only prepends that prefix for a *non-empty* string. So +setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would +work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this +firmware already makes the driver log `Prerequisites size value exceeded the maximum number of +elements supported or data may be malformed` at boot. + +**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and +`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM; +nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read. + +#### Recommendation + +1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff + `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV + win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10. +2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision + one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to + `current_password`, retry one benign attribute. Return `0` confirms the `` theory; `6` + would mean a password mismatch; another `4` means the driver is incompatible with R21 and the + fallback is `hp-repsetup`. +3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup + Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it + needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**. + Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a + security downgrade. +4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in + principle — the private key never leaves the workstation, each change is individually signed, with + anti-replay — but provisioning and every signature require Windows-only tooling. For three + machines whose firmware settings change roughly never, the operational cost far exceeds the risk + reduction. BCU is Windows/WinPE-only. +5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full + pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) — + functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via + `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is + persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the + password in cleartext on the command line — it would land in process listings and shell history + unless wrapped. Store it in the Ansible vault **and** offline. + ## 2. Disk layout — **OPEN DECISION, blocks the build** `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went From 49d82491ddcf327a4012901a6a3e04d16d03e89f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:14:14 -0500 Subject: [PATCH 3/6] =?UTF-8?q?docs(prod):=20node=202=20memory=20SOLVED=20?= =?UTF-8?q?=E2=80=94=20Linux=206.12=20EFI=20memattr=20bug,=20NOT=20a=20har?= =?UTF-8?q?dware=20fault?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping, and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine. ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c never validates num_entries or desc_size: if (tbl->version > 2) { pr_warn(...); goto unmap; } tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; memblock_reserve(efi_mem_attr_table, tbl_size); Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two garbage fields and reserves GIGABYTES from the table address to the top of RAM. THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline — booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2: 15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version 1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481) pages = 2,753,284 kB exactly, Normal contributing zero. BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version 1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY. It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all (Proxmox OVMF) and are unaffected. FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports. DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting, disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where the table lands and may LOOK like fixes. Records every killed hypothesis with its disproof so none of them get re-run. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 155 ++++++++++++++++++------------------- 1 file changed, 75 insertions(+), 80 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f79ae2f..744c8bc 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -44,100 +44,95 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- -## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB +## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01) -Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install -more RAM until this is understood** — see the warning at the end. +> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory +> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine. +> Keeping the correction visible because the wrong framing nearly cost money and a board RMA. -| Evidence | Reading | -|---|---| -| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | -| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | -| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | -| `last_pfn` | two values, `0x42f800` then `0xc3c10` — **NOT a cap**, see the correction below | -| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | -| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | -| Memory blocks | 127 online, 128 MiB block size | +`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware +publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before +reserving memory based on its contents.** Fixed upstream in **6.13**. -**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The -kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and -everything above 4 GiB is present-but-unmanaged. +### The evidence that breaks it open -> **CORRECTION (2026-09-01).** An earlier version of this section cited the `last_pfn` line as -> evidence of a memory cap, and called the pattern the signature of "memory remapping above 4 GB -> not being applied". **The `last_pfn` reading was wrong.** x86 boot prints `last_pfn` twice: first -> `max_pfn` from the full e820, then `max_low_pfn` — the end of RAM *below 4 GiB*. The e820 low -> usable region ends at `0xc3c0ffff`, so the next PFN is exactly `0xc3c10`, which is the second -> value observed. It matches to the byte and is entirely normal on any machine with >4 GiB. A -> routine boot message was presented as a smoking gun. -> -> The real anomaly is narrower than first stated: the `Normal` zone is `present` but `managed 0`, -> 13.3 GiB is reserved, and **nothing in the boot log explains by what**. +The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full +15.9 GiB two boots earlier**: -Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, -identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it -is per-machine state. +| Boot | `Memory: … available` | reserved | memattr line | +|---|---|---|---| +| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` | +| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | +| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | -**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` -kernel driver, which exposes **215 BIOS attributes** read/write at -`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0). +The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in +that shape, and none is fixed by a reboot. -**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was -made on 2026-09-01 and the FIRMWARE refused it: -`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. -HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent -— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. -Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS -Setup Password is deliberately provisioned (see §1). +### Mechanism (verified against kernel source) -What the comparison ruled OUT as the cause: -- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and - `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. -- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on - either node, so it cannot be a mis-set option. -- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, - which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. -- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory - blocks report `online`. The memory reaches the kernel and is then reserved. -- **Integrated graphics is NOT the cause** (operator hypothesis, tested 2026-09-01). `/proc/iomem` - shows `c8800000-cc7fffff : Graphics Stolen Memory` = **exactly 64 MiB**, matching the BIOS - `Video Memory Size=64 MB` setting, and it sits **below** the 4 GiB line. The UHD 630 with `i915` - bound takes precisely what it is configured to take. It cannot account for 13.3 GiB, and it is - not even in the affected address range. +`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`: -Real BIOS differences found (node 1 = works, node 2 = broken): +```c +if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; } +tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; +memblock_reserve(efi_mem_attr_table, tbl_size); +``` -| Attribute | node 1 | node 2 | +**`num_entries` and `desc_size` are never validated.** With a corrupt table: +- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)* +- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s + the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)* + +That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = +DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. + +### BOTH prod nodes are affected — node 1 has only been lucky + +| Node | memattr version field | Outcome | |---|---|---| -| **After Power Loss** | Power On | **Power Off** ← violates §1 | -| **Fast Boot** | Disable | **Enable** ← leading hypothesis | -| Legacy Boot Options | Disable | Enable | -| TPM Activation Policy | F1 to Boot | Allow user to reject | -| Microsoft/Windows UEFI CA 2023 | Enable | Disable | -| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | +| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** | +| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB | +| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB | -**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* -rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale -training data is a plausible route to memory being present, inventoried, handed over — and then -reserved. It is also the one memory-relevant setting that differs from the working node. This is -a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write -and a reboot, which needs operator sanction. +Node 1 is one unlucky boot away from the identical fault. **Fix both.** -To investigate, cheapest first: -- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - **Must be done in F10** — the firmware refuses programmatic writes (above). -- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and - node 2 currently violates it. -- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. -- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot - vs board in one move. -- [ ] `memtest86+` if the above do not resolve it. +### The fix -> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS -> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. -> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` -> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. +**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is +clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message +describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable +branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints +`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. + +Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) +(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the +table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook +Studio G5**, 16 GiB → 1.5 GiB). + +### What NOT to do + +- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is + a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be + installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6. +- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full + R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries. +- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket: + they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They + may appear to "fix" it. They are not fixes. + +### Hypotheses killed, and why — so they are not re-run + +- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by + an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP + documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control. +- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by + `hp-bioscfg` on this platform. +- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. +- **MTRR trim** — MTRRs near-identical to node 1, no trim warning. +- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. +- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly + explained: everything from `0xbeeb0018` upward. ## 1. Firmware / BIOS (per node, before OS install) From 47f02c6e06f4807de9a60be659a15e4bedc69850 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:16:22 -0500 Subject: [PATCH 4/6] docs(prod): second independent confirmation + no cmdline workaround exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A second research pass reached the same root cause by a different route and pinned the arithmetic. Recording it because convergence from two independent lines is what turns this from a good hypothesis into a settled diagnosis. MECHANISM TO THE BYTE: - the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls plain System RAM, so the pre-Linux boot chain overwrites it. - tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31 lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base, so EVERYTHING above 0xbeeb0018 is reserved. - memblock_free_all then hands the allocator nothing above that line while present_pages (from memblock.memory) is untouched -> present 3340288 / managed 0. - 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions(); it is positive PROOF the reservation already happened, a consequence rather than the cause. CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM. Identical 0x2a980 allocation, opposite placement. THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the reservation. A fixed kernel is the only real fix. EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT: /sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and is never memblock_add()ed, which would have left present at ~0 rather than 3340288. Also records that the hardening landed exactly one release late for Debian 13 (bug present since 2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 46 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 744c8bc..6a09c36 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -96,6 +96,41 @@ DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contri Node 1 is one unlucky boot away from the identical fault. **Fix both.** +### Mechanism, traced to the byte (second, independent confirmation) + +A second research pass reached the same root cause by a different route and pinned the arithmetic: + +- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The + kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that + `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it. +- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and + when bit 31 lands the result is **negative**. +- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to + ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical + address above `0xbeeb0018` is reserved.** +- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages` + (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`. +- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by + `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is + therefore positive proof the reservation already happened** — it is a consequence, not the cause. + +The cleanest A/B tell between the two nodes is one line of dmesg: + +| | `NODE_DATA(0) allocated` | meaning | +|---|---|---| +| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation | +| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement | + +Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs +node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. + +### THERE IS NO KERNEL CMDLINE WORKAROUND + +`efi_memattr_init()` is called from `efi_config_parse_tables()` under only +`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()` +**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap` +**do not prevent the reservation.** Booting a fixed kernel is the only real fix. + ### The fix **Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is @@ -104,6 +139,13 @@ describes this bug verbatim. Verified absent from v6.12 and from the current `li branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. +The hardening landed exactly one release too late for Debian 13: the bug has existed since the +feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up +`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim +*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably +lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the +release lifetime, and the fix is small and already upstream. + Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) (Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook @@ -128,6 +170,10 @@ Studio G5**, 16 GiB → 1.5 GiB). - **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by `hp-bioscfg` on this platform. - **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. +- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by + measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`. + Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never + `memblock_add()`ed — which would have left `present` at ~0, not 3340288. - **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. - **MTRR trim** — MTRRs near-identical to node 1, no trim warning. - **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. From cf33338448c94fb7ac5750830b5dc62408ca4381 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:30:45 -0500 Subject: [PATCH 5/6] feat(node_health): detect the EFI memattr bug on every run, two independent ways MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is exposed to the identical lottery on every reboot while currently looking healthy. firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states: Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW) Unexpected EFI Memory Attributes table version -> WARN (corrupt table still published; intact only by luck) [Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working) no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI) no memattr line, log does NOT reach early boot -> SKIP dmesg unreadable -> SKIP memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts, worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25% FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2. THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine": 1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP 2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no memattr line WITHOUT that anchor is a SKIP, not a PASS 3. memory.physical_total needs no kernel log at all Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed. VERDICTS, confirmed live: k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read 1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0% k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6% k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI. Log reaches back to early boot, so that absence is measured, not assumed." Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline workaround because efi_memattr_init() runs before any efi= bail-out. Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under become, only when the unprivileged pass actually failed to read something. Reimplementing the log grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact. LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in /usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed" skip reason. Now shares the sbin-aware which() collect_tools() already had. INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault; recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- infra/ansible/roles/node_health/README.md | 42 ++- .../roles/node_health/defaults/main.yml | 27 ++ .../node_health/files/node_health_collect.py | 311 +++++++++++++++++- .../roles/node_health/tasks/assess.yml | 201 +++++++++++ .../node_health/tasks/collect_privileged.yml | 62 ++++ .../ansible/roles/node_health/tasks/main.yml | 18 +- .../node_health/templates/summary.txt.j2 | 9 + 7 files changed, 664 insertions(+), 6 deletions(-) create mode 100644 infra/ansible/roles/node_health/tasks/collect_privileged.yml diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index a18e949..8b8a6bb 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI, EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM). -- **Never `become`s unless it must.** Only SMART needs root. +- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring + buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM + inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and + the privileged second pass re-runs the *same* collector with `--privileged` so the + parsing has one home. Without escalation those checks SKIP with the exact reason. ## What it checks @@ -44,15 +48,51 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes | | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` | | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters | +| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) | | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) | | `thermal.max` | hottest sensor across thermal zones and hwmon chips | | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically | | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family | | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS | +| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) | Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded in task logic. + +## The EFI memattr check + +`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs. + +HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13** +does not validate it before reserving memory from its contents — +`drivers/firmware/efi/memattr.c` multiplies two never-validated fields +(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the +table address to the top of RAM. Which of two kernel-log lines appears decides whether the +node keeps its memory, and it is **decided afresh on every boot**: + +| kernel log | meaning | verdict | +|---|---|---| +| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* | +| `efi: memattr: Unexpected EFI Memory Attributes table version ` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table | +| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working | +| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS | +| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected | +| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected | + +The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and +differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before +it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing. + +**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from +`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline +workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and +rebooting/CMOS/BIOS/USB changes only move where the table lands. + +`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node +whose kernel log cannot be read is still not unmeasured — and a future memory loss with a +different cause is still caught. + ## Output 1. A human summary printed in the play output. diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index 7a2fc72..eae98c9 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -45,6 +45,33 @@ node_health_thresholds: # Memory (percentage of MemTotal still available) mem_available_pct_warn: 15 mem_available_pct_fail: 5 + # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ---------------------------- + # Two independent measurements of the same question, both fed by memory.physical_total: + # + # unmanaged /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own + # memory map says exists, that never reached the allocator. Needs no root + # and no tools, works identically on metal and in a VM. + # shortfall MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is + # installed. Needs root, and only bare metal reports it. Independent of the + # kernel's memory map entirely, so it also catches RAM the firmware never + # handed the kernel in the first place. + # + # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and + # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2% + # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state. + # The healthy figure is the kernel image, memmap and firmware reservations; it is a + # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10% + # keeps clearance for that. There is no plausible tuning of a threshold in this band + # that changes any of those five verdicts. + # + # The DIMM shortfall runs a little higher than the unmanaged figure on the same node + # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and + # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the + # kernel's to manage. Same bands: the signal being caught is 83%, not 3%. + mem_unmanaged_pct_warn: 10 + mem_unmanaged_pct_fail: 25 + mem_dimm_shortfall_pct_warn: 10 + mem_dimm_shortfall_pct_fail: 25 # Temperatures (deg C, any sensor) temp_warn_c: 70 temp_fail_c: 85 diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index 37adf25..9f2c62b 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -97,10 +97,22 @@ def read_kv(path, sep=":"): return out +def which(tool): + """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits. + + `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the + unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed + but unreachable would be a false reason on a SKIP, which is the one thing this + collector may never do.""" + return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin") + + def run(argv, timeout=20): """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent.""" - if shutil.which(argv[0]) is None: + exe = which(argv[0]) + if exe is None: return None, "", "%s not installed" % argv[0] + argv = [exe] + list(argv[1:]) try: proc = subprocess.run( argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout @@ -284,9 +296,285 @@ def collect_memory(): edac["ce_total"] = ce_total edac["ue_total"] = ue_total mem["edac"] = edac + # Physically present vs actually usable — the cause-independent memory-loss signal. + mem["zones"] = collect_zoneinfo(mem["mem_total_kb"]) + # Firmware's own inventory of what is physically installed. Root-only; attempted + # here anyway so the unprivileged pass records the exact reason it could not read. + mem["devices"] = collect_memory_devices() return mem +# --- Memory the kernel has but never hands out -------------------------------------- +# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode, +# no DMI. Per zone it reports: +# spanned pages in the zone's address range, holes included +# present spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says +# physically exists here +# managed present MINUS everything memblock reserved before the buddy allocator was +# handed the rest (kernel image, memmap, initrd, crashkernel, firmware +# reservations). sum(managed) IS MemTotal, exactly, on every node measured. +# So `present - managed` is, by construction, "RAM this machine has that the OS will +# never allocate", whatever the reason. On a healthy node it is the couple of hundred +# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and +# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% — +# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude +# separation, which is why this needs no clever tuning to be trustworthy. +# +# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if +# a future firmware, kernel or hypervisor lost the memory for an entirely different +# reason, and it needs neither root nor a parseable kernel log. +def collect_zoneinfo(mem_total_kb): + raw = read_text("/proc/zoneinfo", optional=False) + page_kb = os.sysconf("SC_PAGE_SIZE") // 1024 + out = { + "available": raw is not None, + "reason": None if raw is not None else "/proc/zoneinfo could not be read", + "page_size_kb": page_kb, + "zones": [], + "present_kb": None, + "managed_kb": None, + "unmanaged_kb": None, + "unmanaged_pct": None, + } + if raw is None: + return out + node = zone = None + cur = {} + present_pages = managed_pages = 0 + for line in raw.splitlines(): + fields = line.split() + if line.startswith("Node ") and "zone" in line: + # "Node 0, zone Normal" + if cur: + out["zones"].append(cur) + node, zone = fields[1].rstrip(","), fields[-1] + cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None} + elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"): + try: + cur[fields[0]] = int(fields[1]) + except ValueError: + cur[fields[0]] = None + if cur: + out["zones"].append(cur) + # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay + # null rather than emitting a confident 0. + if not out["zones"]: + out["available"] = False + out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it" + return out + for z in out["zones"]: + present_pages += z["present"] or 0 + managed_pages += z["managed"] or 0 + out["present_kb"] = present_pages * page_kb + out["managed_kb"] = managed_pages * page_kb + out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"] + if out["present_kb"] > 0: + out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1) + # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the + # two readings disagree and the discrepancy itself is worth showing. + out["matches_mem_total"] = ( + None if (mem_total_kb is None or out["managed_kb"] is None) + else abs(out["managed_kb"] - mem_total_kb) <= page_kb + ) + return out + + +# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) ----------------------------------- +# The independent answer to "how much RAM is physically in this box", from the firmware's +# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads +# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that +# failure is recorded as the SKIP reason — never as zero DIMMs. +# +# On a VM there is usually either no type-17 structure at all or a single synthetic entry +# that merely restates the configured guest RAM; both are handled by reporting what was +# found and letting the check SKIP when nothing populated was reported. +DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024} + + +def collect_memory_devices(): + out = { + "available": False, + "reason": None, + "devices": [], + "slots": 0, + "populated_slots": 0, + "installed_kb": None, + } + if which("dmidecode") is None: + out["reason"] = ("dmidecode is not installed on this node; it ships with " + "roles/node_baseline (node_baseline_packages). " + "This role never installs packages.") + return out + rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30) + if rc != 0: + out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % ( + rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output") + return out + # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which + # may or may not hold a module ("Size: No Module Installed"). + total_kb = 0 + saw_size = False + for block in stdout.split("\n\n"): + if "Memory Device" not in block: + continue + fields = {} + for line in block.splitlines(): + key, sep, value = line.partition(":") + if sep: + fields[key.strip()] = value.strip() + if "Size" not in fields: + continue + out["slots"] += 1 + size_kb = None + parts = fields["Size"].split() + if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS: + try: + size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()] + except ValueError: + size_kb = None + if size_kb: + saw_size = True + total_kb += size_kb + out["populated_slots"] += 1 + out["devices"].append( + { + "locator": fields.get("Locator"), + "bank_locator": fields.get("Bank Locator"), + # Verbatim, so an unparsed spelling is visible rather than silently null. + "size_raw": fields["Size"], + "size_kb": size_kb, + "type": fields.get("Type"), + "speed": fields.get("Speed"), + "configured_speed": fields.get("Configured Memory Speed"), + "manufacturer": fields.get("Manufacturer"), + "part_number": fields.get("Part Number"), + "rank": fields.get("Rank"), + # Serial deliberately NOT collected: the artifact already carries drive + # serials and a DIMM serial adds identifying data no check reads. + } + ) + out["available"] = saw_size + if not saw_size: + out["reason"] = ("dmidecode ran but reported no populated memory device " + "(%d slot structure(s) found) — typical of a VM with no SMBIOS " + "type 17 inventory" % out["slots"]) + else: + out["installed_kb"] = total_kb + return out + + +# --- Kernel ring buffer: the EFI Memory Attributes Table ---------------------------- +# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative +# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes +# Table, and Linux < 6.13 does not validate it before reserving memory from its contents +# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s +# the product). The kernel prints exactly one of two lines, and WHICH one decides whether +# the node keeps its RAM: +# +# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read +# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW. +# "Unexpected EFI Memory Attributes table version " the garbage version read > 2, so +# the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the +# firmware is still publishing a corrupt table and the outcome is decided afresh by +# where the table lands on every boot. That is luck, not immunity. +# +# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots +# before it lost 13.3 GiB), which is precisely why it is detected on every health run +# instead of diagnosed once. +# +# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on +# every node here). The unprivileged pass therefore records WHY it could not read, and +# roles/node_health re-runs this collector with `--privileged` when escalation works. +BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:") +EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)") +MEMATTR_FAILED_RE = re.compile( + r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I) +MEMATTR_VERSION_RE = re.compile( + r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I) +MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I) +# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table +# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is +# captured separately, above, and quoting it as evidence would bury the one line that +# matters in 200 characters of unrelated table addresses. +MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I) +# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line +# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict. +MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available") +MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved") + + +def collect_klog(): + out = { + "available": False, + "reason": None, + "boot_covered": None, + "memattr_state": None, + "memattr_published": None, + "memattr_table_address": None, + "memattr_version": None, + "memattr_lines": [], + "memory_line": None, + } + rc, stdout, stderr = run(["dmesg"], timeout=30) + if rc is None: + out["reason"] = "dmesg is not installed on this node" + return out + if rc != 0: + detail = (stderr or stdout).strip().splitlines() + out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % ( + rc, os.geteuid(), detail[-1] if detail else "no output", + " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG" + if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "") + return out + + out["available"] = True + # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every + # memattr line is printed in the first fraction of a second. If the boot header is + # gone then so may be a memattr line that was printed — absence would prove nothing, + # and the check must SKIP rather than PASS. This is the difference between + # "unaffected" and "could not tell", and the role is not allowed to blur it. + out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout)) + + published = EFI_CONFIG_RE.search(stdout) + if published: + out["memattr_published"] = True + out["memattr_table_address"] = published.group(1) + elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M): + # UEFI booted and the config-table line is present, but it lists no MEMATTR=. + out["memattr_published"] = False + # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None. + + for line in stdout.splitlines(): + if MEMATTR_LINE_RE.search(line): + out["memattr_lines"].append(line.strip()) + joined = "\n".join(out["memattr_lines"]) + if MEMATTR_CORRUPT_RE.search(joined): + # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected. + out["memattr_state"] = "corrupt_rejected" + elif MEMATTR_FAILED_RE.search(joined): + out["memattr_state"] = "failed_to_map" + addr = MEMATTR_FAILED_RE.search(joined).group(1) + if addr: + out["memattr_table_address"] = addr + elif MEMATTR_VERSION_RE.search(joined): + out["memattr_state"] = "bad_version" + out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1)) + else: + out["memattr_state"] = "clean" + + mem_line = MEMORY_LINE_RE.search(stdout) + if mem_line: + full = [l for l in stdout.splitlines() if mem_line.group(0) in l] + reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None + out["memory_line"] = { + "available_kb": int(mem_line.group(1)), + "total_kb": int(mem_line.group(2)), + "reserved_kb": int(reserved.group(1)) if reserved else None, + "raw": full[0].strip() if full else None, + } + return out + + # --- Thermals ----------------------------------------------------------------------- def collect_thermal(): zones = [] @@ -708,15 +996,34 @@ def collect_os(): def collect_tools(): tools = {} for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"): - tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin") + tools[tool] = which(tool) return tools def main(): + # --privileged: re-collect ONLY the fields that genuinely need root, for the second + # pass roles/node_health makes when (and only when) escalation is proven to work + # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place — + # re-implementing the kernel-log grep in Jinja would be a second source of truth for + # the same fact, and the two would drift. + if "--privileged" in sys.argv[1:]: + doc = { + "schema": SCHEMA, + "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "collector_uid": os.geteuid(), + "klog": collect_klog(), + "memory": {"devices": collect_memory_devices()}, + } + doc["unreadable"] = UNREADABLE + json.dump(doc, sys.stdout, indent=2, sort_keys=True) + sys.stdout.write("\n") + return + doc = { "schema": SCHEMA, "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"), "collector_uid": os.geteuid(), + "klog": collect_klog(), "os": collect_os(), "dmi": collect_dmi(), "cpu": collect_cpu(), diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 3023397..7553959 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -387,6 +387,83 @@ ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available (' ~ avail_pct | string ~ '%)') }} +# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the +# one cause we have actually hit here; this one asks the blunter question — "is this +# machine using the RAM it has?" — and answers it from two sources that share NOTHING +# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some +# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml. +- name: Assess — usable memory vs the memory this machine physically has + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'memory.physical_total', + 'title': 'Usable memory vs installed/present memory (independent of the kernel log)', + 'status': phys_status, + 'detail': phys_detail }] }}" + vars: + ansible_become: false + th: "{{ node_health_thresholds }}" + z: "{{ node_health_facts.memory.zones }}" + dev: "{{ node_health_facts.memory.devices }}" + m: "{{ node_health_facts.memory }}" + + # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it + # actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no + # tools, identical on metal and in a VM — so this half is available almost always. + z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}" + z_status: >- + {{ 'SKIP' if z_pct is not number + else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail + else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }} + + # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev + # VMs and wherever escalation is unavailable — with dmidecode's own reason attached, + # never as a zero DIMM count. Independent of the kernel's memory map entirely. + inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}" + d_pct: >- + {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1) + if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0) + else none }} + d_status: >- + {{ 'SKIP' if d_pct is not number + else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail + else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }} + + # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two + # SKIPs must never add up to a PASS. + phys_status: >- + {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last) + | default('SKIP', true) }} + # What the kernel log says, IF it was read — quoted here only as attribution for a + # shortfall this check measured on its own. The verdict above never depends on it. + phys_cause: >- + {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check + firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b') + if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map' + else ('. The kernel log was NOT read here, so this shortfall is unattributed — + see check firmware.efi_memattr') + if not (node_health_facts.klog.available | bool) + else '' }} + phys_detail: >- + {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed ' + ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — ' + ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT + was never handed to the allocator (/proc/zoneinfo)') + if z_pct is number + else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true)))) + ~ '. ' + ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across ' + ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) (' + ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', ')) + ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string + ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall') + if d_pct is number + else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true)))) + ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware + reservations and (on this platform) 64 MiB of graphics stolen memory. + A shortfall of this size is memory the machine HAS and cannot use') + if phys_status in ['WARN', 'FAIL'] else '') + ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }} + - name: Assess — swap (kubelet requires it off) ansible.builtin.set_fact: node_health_checks: "{{ node_health_checks + [{ @@ -674,3 +751,127 @@ ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')' ~ (' — serial numbers not read (root-only DMI fields)' if d.product_serial is none else '')) }} + +# --- EFI Memory Attributes Table ------------------------------------------------------ +# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here; +# what follows is only what this check needs to be read correctly. +# +# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does +# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c +# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s +# the product — everything from the table address to the top of RAM. The kernel prints +# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field +# happens to read > 2: +# +# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel +# TRUSTED the table. The node has lost the memory. FAIL. +# "Unexpected EFI Memory Attributes table version " version read > 2, the kernel +# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a +# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS: +# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself +# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one +# unlucky boot from losing 83% of its RAM is not passing a health check. +# +# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC, +# so it cannot be diagnosed once. It has to be measured on every health run. +# +# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected +# ONLY if the kernel log was actually read AND still reaches back to early boot — every +# memattr diagnostic is printed in the first fraction of a second, and the ring buffer +# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and +# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a +# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel +# log at all, so a node whose log is unreadable is still not unmeasured. +- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse) + ansible.builtin.set_fact: + node_health_checks: "{{ node_health_checks + [{ + 'id': 'firmware.efi_memattr', + 'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel', + 'status': memattr_status, + 'detail': memattr_detail }] }}" + vars: + ansible_become: false + k: "{{ node_health_facts.klog }}" + z: "{{ node_health_facts.memory.zones }}" + ma_state: "{{ k.memattr_state | default(none) }}" + memattr_status: >- + {{ 'SKIP' if not (k.available | bool) + else ('FAIL' if ma_state == 'failed_to_map' + else ('WARN' if ma_state == 'bad_version' + else ('PASS' if ma_state == 'corrupt_rejected' + else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }} + # One remediation, quoted by both findings. NOTE what it must NOT say: there is no + # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out + # could take effect. Suggesting one would be a fix that quietly does nothing. + memattr_fix: >- + FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) — + upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly + bogus", first released in v6.13 and absent from v6.12 and from the current + linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}. + There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi= + bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or + unplugging USB only change where the table lands and may APPEAR to fix it — they are + a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b. + # The symptom, measured independently of this log (see memory.physical_total). + memattr_corroboration: >- + {{ (' Measured independently of this log: MemTotal ' + ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' + ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. ' + ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away + from the allocator (/proc/zoneinfo) — check memory.physical_total.') + if ((z.available | bool) and z.unmanaged_pct is number) else '' }} + # The kernel log lines themselves, verbatim. A verdict the operator cannot check + # against the evidence is not much better than a wrong one. + memattr_evidence: >- + {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | '))) + if (k.memattr_lines | default([]) | length) > 0 else '') + ~ ((' At boot the kernel reported ' + ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of ' + ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, ' + ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved' + ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.') + if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number) + else '') }} + memattr_detail: >- + {%- if not (k.available | bool) -%} + the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}. + Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic + lives only in dmesg.{{ memattr_corroboration }} + {%- elif ma_state == 'failed_to_map' -%} + THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory + Attributes table at {{ k.memattr_table_address | default('an unreported address') }} + and reserved memory from the corrupt table anyway — everything from that address + upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }} + {%- elif ma_state == 'bad_version' -%} + EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table + here too; this boot survived only because its garbage version field read + {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out + before reserving anything. The outcome is re-decided on EVERY boot: if that field + reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right + now and this node is one unlucky boot from the failure state.{{ memattr_evidence }} + {{ memattr_fix }} + {%- elif ma_state == 'corrupt_rejected' -%} + the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel + validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is + intact and stays intact across reboots; no action.{{ memattr_evidence }} + {%- elif not (k.boot_covered | bool) -%} + NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches + back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in + the ring buffer, which wraps). Every memattr diagnostic is printed within the first + fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a + pass. Read /var/log/kern.log for this boot, or re-run after a + reboot.{{ memattr_corroboration }} + {%- elif k.memattr_published is sameas true -%} + the firmware publishes an EFI Memory Attributes table at + {{ k.memattr_table_address | default('an unreported address') }} and the kernel + accepted it without complaint, in a log that still reaches back to early boot. Not + affected on this boot. + {%- elif k.memattr_published is sameas false -%} + this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line + lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF + guest. + {%- else -%} + no UEFI config-table line in the kernel log at all — this node did not boot via UEFI, + so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early + boot, so that absence is measured, not assumed. + {%- endif -%} diff --git a/infra/ansible/roles/node_health/tasks/collect_privileged.yml b/infra/ansible/roles/node_health/tasks/collect_privileged.yml new file mode 100644 index 0000000..5afb552 --- /dev/null +++ b/infra/ansible/roles/node_health/tasks/collect_privileged.yml @@ -0,0 +1,62 @@ +--- +# The SECOND pass of the collector, run with escalation, for the two facts that are +# genuinely root-gated: +# +# * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an +# unprivileged read gets EPERM. This is where the EFI Memory Attributes Table +# diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference +# between a node that has lost 13.3 GiB and one that has not. +# * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed +# DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. +# +# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a +# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line- +# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND +# source of truth for the same fact, and the two would drift. `--privileged` makes the +# collector emit exactly those subtrees and nothing else, so the parsing stays in one +# place and is identical whichever pass produced it. +# +# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass +# actually failed to read one of them (main.yml checks both). On a node where +# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule +# is that everything readable WITHOUT root is read without root. +# +# If escalation is unavailable the checks that consume these SKIP with the exact reason +# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line" +# and "could not read the kernel log" are different answers and the role says which. + +- name: Re-collect the root-only facts with escalation + ansible.builtin.script: + cmd: node_health_collect.py --privileged + executable: /usr/bin/python3 + become: true + vars: + ansible_become: true + register: node_health_privileged_raw + changed_when: false # strictly read-only; it can never change the node + +- name: Assert the privileged payload is the schema this role understands + ansible.builtin.assert: + that: + - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1' + fail_msg: >- + Privileged collector pass returned an unexpected schema; role and collector are + out of step. + quiet: true + vars: + ansible_become: false + +# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive +# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine +# would REPLACE it and throw away the reasons the first pass recorded. +- name: Merge the privileged facts over the unprivileged ones + ansible.builtin.set_fact: + node_health_facts: >- + {{ node_health_facts | combine({ + 'klog': priv.klog, + 'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}), + 'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])), + }) }} + vars: + ansible_become: false + priv: "{{ node_health_privileged_raw.stdout | from_json }}" diff --git a/infra/ansible/roles/node_health/tasks/main.yml b/infra/ansible/roles/node_health/tasks/main.yml index a57ffc0..2a0d07c 100644 --- a/infra/ansible/roles/node_health/tasks/main.yml +++ b/infra/ansible/roles/node_health/tasks/main.yml @@ -4,9 +4,11 @@ # ORDER OF OPERATIONS # 1. Collect everything readable WITHOUT root (one script, one round trip). # 2. Probe whether privilege escalation actually works on this host. -# 3. Only if it does AND smartmontools is installed: read SMART. -# 4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks. -# 5. Render the human summary + write the JSON artifact. +# 3. Only if it does, and only for what the first pass could NOT read: re-run the same +# collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory). +# 4. Only if it does AND smartmontools is installed: read SMART. +# 5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks. +# 6. Render the human summary + write the JSON artifact. # # BECOME HANDLING — the subtle bit, read before editing. # `ansible_become` set as an inventory/group var (which is exactly what @@ -43,6 +45,16 @@ ansible.builtin.include_tasks: probe_privilege.yml when: node_health_try_privileged | bool +# Only when the unprivileged pass actually failed to read one of them. On a node with +# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped +# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml). +- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs) + ansible.builtin.include_tasks: collect_privileged.yml + when: + - node_health_root_available | bool + - (not (node_health_facts.klog.available | bool)) + or (not (node_health_facts.memory.devices.available | bool)) + - name: Read SMART data (privileged) ansible.builtin.include_tasks: collect_smart.yml when: diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 23a32ba..732af45 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -45,6 +45,15 @@ {% endif %} Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }} +{% set mz = node_health_facts.memory.zones %} +{% set md = node_health_facts.memory.devices %} + Present RAM {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }} + DIMMs {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }} +{% for dimm in md.devices | default([]) if dimm.size_kb %} + {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }} +{% endfor %} +{% set klog = node_health_facts.klog %} + EFI memattr {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }} Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }} -------------------------------------------------------------------------------- From 0a733294ffaa301c20229a5b8177ac15e5ea5841 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 22:31:26 -0500 Subject: [PATCH 6/6] docs(prod): node 1's DIMMs are not a matched pair MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair runs at the slower 2667. Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 6a09c36..c19e035 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -35,6 +35,12 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. | `/dev/kvm` | present | present | | | Serial | MXL1052NGN | MXL0073949 | | +**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check; +unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**), +`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct, +but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded +only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch. + **The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's