Merge pull request 'Node 2 memory: solved — Linux 6.12 EFI memattr bug, not hardware' (#15) from docs/node2-memory-refine into main
This commit is contained in:
gitea-admin committed
commit
4809199da6
8 files changed
+869
-74
No files matched your search
+205
-68
@@ -35,6 +35,12 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|
||||
| `/dev/kvm` | present | present | |
|
||||
| Serial | MXL1052NGN | MXL0073949 | |
|
||||
|
||||
**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
|
||||
unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
|
||||
`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct,
|
||||
but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
|
||||
only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
|
||||
|
||||
**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
|
||||
healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
|
||||
placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
|
||||
@@ -44,91 +50,148 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di
|
||||
|
||||
---
|
||||
|
||||
## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB
|
||||
## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)
|
||||
|
||||
Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install
|
||||
more RAM until this is understood** — see the warning at the end.
|
||||
> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
|
||||
> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
|
||||
> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.
|
||||
|
||||
| Evidence | Reading |
|
||||
|---|---|
|
||||
| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty |
|
||||
| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` |
|
||||
| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` |
|
||||
| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot |
|
||||
| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** |
|
||||
| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** |
|
||||
| Memory blocks | 127 online, 128 MiB block size |
|
||||
`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
|
||||
publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
|
||||
reserving memory based on its contents.** Fixed upstream in **6.13**.
|
||||
|
||||
**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The
|
||||
kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and
|
||||
everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping
|
||||
above 4 GB not being applied**, not of failing memory.
|
||||
### The evidence that breaks it open
|
||||
|
||||
Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00,
|
||||
identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it
|
||||
is per-machine state.
|
||||
The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
|
||||
15.9 GiB two boots earlier**:
|
||||
|
||||
**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg`
|
||||
kernel driver, which exposes **215 BIOS attributes** read/write at
|
||||
`/sys/class/firmware-attributes/hp-bioscfg/attributes/<name>/current_value`. No BIOS passwords
|
||||
are set (`authentication/*/is_enabled` = 0).
|
||||
| Boot | `Memory: … available` | reserved | memattr line |
|
||||
|---|---|---|---|
|
||||
| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
|
||||
| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
|
||||
| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
|
||||
|
||||
**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was
|
||||
made on 2026-09-01 and the FIRMWARE refused it:
|
||||
`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`.
|
||||
HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent
|
||||
— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`.
|
||||
Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS
|
||||
Setup Password is deliberately provisioned (see §1).
|
||||
The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
|
||||
that shape, and none is fixed by a reboot.
|
||||
|
||||
What the comparison ruled OUT as the cause:
|
||||
- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and
|
||||
`Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting.
|
||||
- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on
|
||||
either node, so it cannot be a mis-set option.
|
||||
- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**,
|
||||
which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path.
|
||||
- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory
|
||||
blocks report `online`. The memory reaches the kernel and is then reserved.
|
||||
### Mechanism (verified against kernel source)
|
||||
|
||||
Real BIOS differences found (node 1 = works, node 2 = broken):
|
||||
`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:
|
||||
|
||||
| Attribute | node 1 | node 2 |
|
||||
```c
|
||||
if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
|
||||
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
|
||||
memblock_reserve(efi_mem_attr_table, tbl_size);
|
||||
```
|
||||
|
||||
**`num_entries` and `desc_size` are never validated.** With a corrupt table:
|
||||
- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
|
||||
- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
|
||||
the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*
|
||||
|
||||
That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
|
||||
DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
|
||||
|
||||
### BOTH prod nodes are affected — node 1 has only been lucky
|
||||
|
||||
| Node | memattr version field | Outcome |
|
||||
|---|---|---|
|
||||
| **After Power Loss** | Power On | **Power Off** ← violates §1 |
|
||||
| **Fast Boot** | Disable | **Enable** ← leading hypothesis |
|
||||
| Legacy Boot Options | Disable | Enable |
|
||||
| TPM Activation Policy | F1 to Boot | Allow user to reject |
|
||||
| Microsoft/Windows UEFI CA 2023 | Enable | Disable |
|
||||
| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) |
|
||||
| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
|
||||
| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
|
||||
| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |
|
||||
|
||||
**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data*
|
||||
rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale
|
||||
training data is a plausible route to memory being present, inventoried, handed over — and then
|
||||
reserved. It is also the one memory-relevant setting that differs from the working node. This is
|
||||
a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write
|
||||
and a reboot, which needs operator sanction.
|
||||
Node 1 is one unlucky boot away from the identical fault. **Fix both.**
|
||||
|
||||
To investigate, cheapest first:
|
||||
- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1).
|
||||
**Must be done in F10** — the firmware refuses programmatic writes (above).
|
||||
- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and
|
||||
node 2 currently violates it.
|
||||
- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check.
|
||||
- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot
|
||||
vs board in one move.
|
||||
- [ ] `memtest86+` if the above do not resolve it.
|
||||
### Mechanism, traced to the byte (second, independent confirmation)
|
||||
|
||||
> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS
|
||||
> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted.
|
||||
> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod`
|
||||
> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share.
|
||||
A second research pass reached the same root cause by a different route and pinned the arithmetic:
|
||||
|
||||
- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
|
||||
kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
|
||||
`/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
|
||||
- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
|
||||
when bit 31 lands the result is **negative**.
|
||||
- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
|
||||
~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
|
||||
address above `0xbeeb0018` is reserved.**
|
||||
- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
|
||||
(which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
|
||||
- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
|
||||
`efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
|
||||
therefore positive proof the reservation already happened** — it is a consequence, not the cause.
|
||||
|
||||
The cleanest A/B tell between the two nodes is one line of dmesg:
|
||||
|
||||
| | `NODE_DATA(0) allocated` | meaning |
|
||||
|---|---|---|
|
||||
| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
|
||||
| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |
|
||||
|
||||
Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
|
||||
node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
|
||||
|
||||
### THERE IS NO KERNEL CMDLINE WORKAROUND
|
||||
|
||||
`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
|
||||
`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
|
||||
**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
|
||||
**do not prevent the reservation.** Booting a fixed kernel is the only real fix.
|
||||
|
||||
### The fix
|
||||
|
||||
**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
|
||||
clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
|
||||
describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
|
||||
branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
|
||||
`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.
|
||||
|
||||
The hardening landed exactly one release too late for Debian 13: the bug has existed since the
|
||||
feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
|
||||
`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
|
||||
*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
|
||||
lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
|
||||
release lifetime, and the fix is small and already upstream.
|
||||
|
||||
Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
|
||||
(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
|
||||
table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
|
||||
Studio G5**, 16 GiB → 1.5 GiB).
|
||||
|
||||
### What NOT to do
|
||||
|
||||
- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
|
||||
a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
|
||||
installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
|
||||
- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
|
||||
R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
|
||||
- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
|
||||
they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
|
||||
may appear to "fix" it. They are not fixes.
|
||||
|
||||
### Hypotheses killed, and why — so they are not re-run
|
||||
|
||||
- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
|
||||
an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
|
||||
documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
|
||||
- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
|
||||
`hp-bioscfg` on this platform.
|
||||
- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
|
||||
- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
|
||||
measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
|
||||
Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
|
||||
`memblock_add()`ed — which would have left `present` at ~0, not 3340288.
|
||||
- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
|
||||
- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
|
||||
- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
|
||||
- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
|
||||
explained: everything from `0xbeeb0018` upward.
|
||||
|
||||
## 1. Firmware / BIOS (per node, before OS install)
|
||||
|
||||
> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
|
||||
> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work
|
||||
> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
|
||||
> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
|
||||
> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
|
||||
> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
|
||||
> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
|
||||
>
|
||||
> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
|
||||
@@ -154,6 +217,80 @@ To investigate, cheapest first:
|
||||
nodes match.
|
||||
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
|
||||
|
||||
### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01
|
||||
|
||||
**There is no default BIOS password, and no factory default exists to look up.** HP business
|
||||
desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
|
||||
observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
|
||||
describes both purely as things the owner configures in Computer Setup.
|
||||
|
||||
**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
|
||||
clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
|
||||
**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
|
||||
turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
|
||||
permissive default is the right setting for a homelab.
|
||||
|
||||
#### CORRECTION — error `0x4` is NOT an authorisation failure
|
||||
|
||||
I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
|
||||
was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
|
||||
`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
|
||||
different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
|
||||
unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
|
||||
returns:
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Success |
|
||||
| 1 | Not Supported |
|
||||
| **4** | **Failed** (generic) |
|
||||
| 6 | Access Denied |
|
||||
|
||||
So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
|
||||
the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
|
||||
of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
|
||||
in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.
|
||||
|
||||
**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
|
||||
*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
|
||||
blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
|
||||
setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
|
||||
work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
|
||||
firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
|
||||
elements supported or data may be malformed` at boot.
|
||||
|
||||
**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
|
||||
`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
|
||||
nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.
|
||||
|
||||
#### Recommendation
|
||||
|
||||
1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
|
||||
`current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
|
||||
win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
|
||||
2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
|
||||
one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
|
||||
`current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
|
||||
would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
|
||||
fallback is `hp-repsetup`.
|
||||
3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
|
||||
Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
|
||||
needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
|
||||
Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
|
||||
security downgrade.
|
||||
4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
|
||||
principle — the private key never leaves the workstation, each change is individually signed, with
|
||||
anti-replay — but provisioning and every signature require Windows-only tooling. For three
|
||||
machines whose firmware settings change roughly never, the operational cost far exceeds the risk
|
||||
reduction. BCU is Windows/WinPE-only.
|
||||
5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
|
||||
pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
|
||||
functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
|
||||
`hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
|
||||
persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
|
||||
password in cleartext on the command line — it would land in process listings and shell history
|
||||
unless wrapped. Store it in the Ansible vault **and** offline.
|
||||
|
||||
## 2. Disk layout — **OPEN DECISION, blocks the build**
|
||||
|
||||
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
|
||||
|
||||
@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
- **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
|
||||
EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
|
||||
against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
|
||||
- **Never `become`s unless it must.** Only SMART needs root.
|
||||
- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
|
||||
buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
|
||||
inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
|
||||
the privileged second pass re-runs the *same* collector with `--privileged` so the
|
||||
parsing has one home. Without escalation those checks SKIP with the exact reason.
|
||||
|
||||
## What it checks
|
||||
|
||||
@@ -44,15 +48,51 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
|
||||
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
|
||||
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
|
||||
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
|
||||
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
|
||||
| `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
|
||||
| `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
|
||||
| `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
|
||||
| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
|
||||
|
||||
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
|
||||
in task logic.
|
||||
|
||||
|
||||
## The EFI memattr check
|
||||
|
||||
`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
|
||||
|
||||
HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
|
||||
does not validate it before reserving memory from its contents —
|
||||
`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
|
||||
(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
|
||||
table address to the top of RAM. Which of two kernel-log lines appears decides whether the
|
||||
node keeps its memory, and it is **decided afresh on every boot**:
|
||||
|
||||
| kernel log | meaning | verdict |
|
||||
|---|---|---|
|
||||
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
|
||||
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
|
||||
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
|
||||
| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
|
||||
| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
|
||||
| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
|
||||
|
||||
The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
|
||||
differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
|
||||
it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
|
||||
|
||||
**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
|
||||
`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
|
||||
workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
|
||||
rebooting/CMOS/BIOS/USB changes only move where the table lands.
|
||||
|
||||
`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
|
||||
whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
|
||||
different cause is still caught.
|
||||
|
||||
## Output
|
||||
|
||||
1. A human summary printed in the play output.
|
||||
|
||||
@@ -45,6 +45,33 @@ node_health_thresholds:
|
||||
# Memory (percentage of MemTotal still available)
|
||||
mem_available_pct_warn: 15
|
||||
mem_available_pct_fail: 5
|
||||
# --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
|
||||
# Two independent measurements of the same question, both fed by memory.physical_total:
|
||||
#
|
||||
# unmanaged /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
|
||||
# memory map says exists, that never reached the allocator. Needs no root
|
||||
# and no tools, works identically on metal and in a VM.
|
||||
# shortfall MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
|
||||
# installed. Needs root, and only bare metal reports it. Independent of the
|
||||
# kernel's memory map entirely, so it also catches RAM the firmware never
|
||||
# handed the kernel in the first place.
|
||||
#
|
||||
# MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
|
||||
# not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
|
||||
# k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
|
||||
# The healthy figure is the kernel image, memmap and firmware reservations; it is a
|
||||
# near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
|
||||
# keeps clearance for that. There is no plausible tuning of a threshold in this band
|
||||
# that changes any of those five verdicts.
|
||||
#
|
||||
# The DIMM shortfall runs a little higher than the unmanaged figure on the same node
|
||||
# (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
|
||||
# the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
|
||||
# kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
|
||||
mem_unmanaged_pct_warn: 10
|
||||
mem_unmanaged_pct_fail: 25
|
||||
mem_dimm_shortfall_pct_warn: 10
|
||||
mem_dimm_shortfall_pct_fail: 25
|
||||
# Temperatures (deg C, any sensor)
|
||||
temp_warn_c: 70
|
||||
temp_fail_c: 85
|
||||
|
||||
@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
|
||||
return out
|
||||
|
||||
|
||||
def which(tool):
|
||||
"""Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
|
||||
|
||||
`dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
|
||||
unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
|
||||
but unreachable would be a false reason on a SKIP, which is the one thing this
|
||||
collector may never do."""
|
||||
return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
|
||||
|
||||
|
||||
def run(argv, timeout=20):
|
||||
"""Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
|
||||
if shutil.which(argv[0]) is None:
|
||||
exe = which(argv[0])
|
||||
if exe is None:
|
||||
return None, "", "%s not installed" % argv[0]
|
||||
argv = [exe] + list(argv[1:])
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
|
||||
@@ -284,9 +296,285 @@ def collect_memory():
|
||||
edac["ce_total"] = ce_total
|
||||
edac["ue_total"] = ue_total
|
||||
mem["edac"] = edac
|
||||
# Physically present vs actually usable — the cause-independent memory-loss signal.
|
||||
mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
|
||||
# Firmware's own inventory of what is physically installed. Root-only; attempted
|
||||
# here anyway so the unprivileged pass records the exact reason it could not read.
|
||||
mem["devices"] = collect_memory_devices()
|
||||
return mem
|
||||
|
||||
|
||||
# --- Memory the kernel has but never hands out --------------------------------------
|
||||
# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
|
||||
# no DMI. Per zone it reports:
|
||||
# spanned pages in the zone's address range, holes included
|
||||
# present spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
|
||||
# physically exists here
|
||||
# managed present MINUS everything memblock reserved before the buddy allocator was
|
||||
# handed the rest (kernel image, memmap, initrd, crashkernel, firmware
|
||||
# reservations). sum(managed) IS MemTotal, exactly, on every node measured.
|
||||
# So `present - managed` is, by construction, "RAM this machine has that the OS will
|
||||
# never allocate", whatever the reason. On a healthy node it is the couple of hundred
|
||||
# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
|
||||
# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
|
||||
# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
|
||||
# separation, which is why this needs no clever tuning to be trustworthy.
|
||||
#
|
||||
# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
|
||||
# a future firmware, kernel or hypervisor lost the memory for an entirely different
|
||||
# reason, and it needs neither root nor a parseable kernel log.
|
||||
def collect_zoneinfo(mem_total_kb):
|
||||
raw = read_text("/proc/zoneinfo", optional=False)
|
||||
page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
|
||||
out = {
|
||||
"available": raw is not None,
|
||||
"reason": None if raw is not None else "/proc/zoneinfo could not be read",
|
||||
"page_size_kb": page_kb,
|
||||
"zones": [],
|
||||
"present_kb": None,
|
||||
"managed_kb": None,
|
||||
"unmanaged_kb": None,
|
||||
"unmanaged_pct": None,
|
||||
}
|
||||
if raw is None:
|
||||
return out
|
||||
node = zone = None
|
||||
cur = {}
|
||||
present_pages = managed_pages = 0
|
||||
for line in raw.splitlines():
|
||||
fields = line.split()
|
||||
if line.startswith("Node ") and "zone" in line:
|
||||
# "Node 0, zone Normal"
|
||||
if cur:
|
||||
out["zones"].append(cur)
|
||||
node, zone = fields[1].rstrip(","), fields[-1]
|
||||
cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
|
||||
elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
|
||||
try:
|
||||
cur[fields[0]] = int(fields[1])
|
||||
except ValueError:
|
||||
cur[fields[0]] = None
|
||||
if cur:
|
||||
out["zones"].append(cur)
|
||||
# An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
|
||||
# null rather than emitting a confident 0.
|
||||
if not out["zones"]:
|
||||
out["available"] = False
|
||||
out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
|
||||
return out
|
||||
for z in out["zones"]:
|
||||
present_pages += z["present"] or 0
|
||||
managed_pages += z["managed"] or 0
|
||||
out["present_kb"] = present_pages * page_kb
|
||||
out["managed_kb"] = managed_pages * page_kb
|
||||
out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
|
||||
if out["present_kb"] > 0:
|
||||
out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
|
||||
# Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
|
||||
# two readings disagree and the discrepancy itself is worth showing.
|
||||
out["matches_mem_total"] = (
|
||||
None if (mem_total_kb is None or out["managed_kb"] is None)
|
||||
else abs(out["managed_kb"] - mem_total_kb) <= page_kb
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
|
||||
# The independent answer to "how much RAM is physically in this box", from the firmware's
|
||||
# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
|
||||
# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
|
||||
# failure is recorded as the SKIP reason — never as zero DIMMs.
|
||||
#
|
||||
# On a VM there is usually either no type-17 structure at all or a single synthetic entry
|
||||
# that merely restates the configured guest RAM; both are handled by reporting what was
|
||||
# found and letting the check SKIP when nothing populated was reported.
|
||||
DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
|
||||
|
||||
|
||||
def collect_memory_devices():
|
||||
out = {
|
||||
"available": False,
|
||||
"reason": None,
|
||||
"devices": [],
|
||||
"slots": 0,
|
||||
"populated_slots": 0,
|
||||
"installed_kb": None,
|
||||
}
|
||||
if which("dmidecode") is None:
|
||||
out["reason"] = ("dmidecode is not installed on this node; it ships with "
|
||||
"roles/node_baseline (node_baseline_packages). "
|
||||
"This role never installs packages.")
|
||||
return out
|
||||
rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
|
||||
if rc != 0:
|
||||
out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
|
||||
rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
|
||||
return out
|
||||
# Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
|
||||
# may or may not hold a module ("Size: No Module Installed").
|
||||
total_kb = 0
|
||||
saw_size = False
|
||||
for block in stdout.split("\n\n"):
|
||||
if "Memory Device" not in block:
|
||||
continue
|
||||
fields = {}
|
||||
for line in block.splitlines():
|
||||
key, sep, value = line.partition(":")
|
||||
if sep:
|
||||
fields[key.strip()] = value.strip()
|
||||
if "Size" not in fields:
|
||||
continue
|
||||
out["slots"] += 1
|
||||
size_kb = None
|
||||
parts = fields["Size"].split()
|
||||
if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
|
||||
try:
|
||||
size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
|
||||
except ValueError:
|
||||
size_kb = None
|
||||
if size_kb:
|
||||
saw_size = True
|
||||
total_kb += size_kb
|
||||
out["populated_slots"] += 1
|
||||
out["devices"].append(
|
||||
{
|
||||
"locator": fields.get("Locator"),
|
||||
"bank_locator": fields.get("Bank Locator"),
|
||||
# Verbatim, so an unparsed spelling is visible rather than silently null.
|
||||
"size_raw": fields["Size"],
|
||||
"size_kb": size_kb,
|
||||
"type": fields.get("Type"),
|
||||
"speed": fields.get("Speed"),
|
||||
"configured_speed": fields.get("Configured Memory Speed"),
|
||||
"manufacturer": fields.get("Manufacturer"),
|
||||
"part_number": fields.get("Part Number"),
|
||||
"rank": fields.get("Rank"),
|
||||
# Serial deliberately NOT collected: the artifact already carries drive
|
||||
# serials and a DIMM serial adds identifying data no check reads.
|
||||
}
|
||||
)
|
||||
out["available"] = saw_size
|
||||
if not saw_size:
|
||||
out["reason"] = ("dmidecode ran but reported no populated memory device "
|
||||
"(%d slot structure(s) found) — typical of a VM with no SMBIOS "
|
||||
"type 17 inventory" % out["slots"])
|
||||
else:
|
||||
out["installed_kb"] = total_kb
|
||||
return out
|
||||
|
||||
|
||||
# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
|
||||
# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
|
||||
# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
|
||||
# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
|
||||
# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
|
||||
# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
|
||||
# the node keeps its RAM:
|
||||
#
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read
|
||||
# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
|
||||
# "Unexpected EFI Memory Attributes table version <N>" the garbage version read > 2, so
|
||||
# the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
|
||||
# firmware is still publishing a corrupt table and the outcome is decided afresh by
|
||||
# where the table lands on every boot. That is luck, not immunity.
|
||||
#
|
||||
# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
|
||||
# before it lost 13.3 GiB), which is precisely why it is detected on every health run
|
||||
# instead of diagnosed once.
|
||||
#
|
||||
# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
|
||||
# every node here). The unprivileged pass therefore records WHY it could not read, and
|
||||
# roles/node_health re-runs this collector with `--privileged` when escalation works.
|
||||
BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
|
||||
EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
|
||||
MEMATTR_FAILED_RE = re.compile(
|
||||
r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
|
||||
MEMATTR_VERSION_RE = re.compile(
|
||||
r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
|
||||
MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
|
||||
# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
|
||||
# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
|
||||
# captured separately, above, and quoting it as evidence would bury the one line that
|
||||
# matters in 200 characters of unrelated table addresses.
|
||||
MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
|
||||
# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
|
||||
# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
|
||||
MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
|
||||
MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
|
||||
|
||||
|
||||
def collect_klog():
|
||||
out = {
|
||||
"available": False,
|
||||
"reason": None,
|
||||
"boot_covered": None,
|
||||
"memattr_state": None,
|
||||
"memattr_published": None,
|
||||
"memattr_table_address": None,
|
||||
"memattr_version": None,
|
||||
"memattr_lines": [],
|
||||
"memory_line": None,
|
||||
}
|
||||
rc, stdout, stderr = run(["dmesg"], timeout=30)
|
||||
if rc is None:
|
||||
out["reason"] = "dmesg is not installed on this node"
|
||||
return out
|
||||
if rc != 0:
|
||||
detail = (stderr or stdout).strip().splitlines()
|
||||
out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
|
||||
rc, os.geteuid(), detail[-1] if detail else "no output",
|
||||
" — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
|
||||
if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
|
||||
return out
|
||||
|
||||
out["available"] = True
|
||||
# Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
|
||||
# memattr line is printed in the first fraction of a second. If the boot header is
|
||||
# gone then so may be a memattr line that was printed — absence would prove nothing,
|
||||
# and the check must SKIP rather than PASS. This is the difference between
|
||||
# "unaffected" and "could not tell", and the role is not allowed to blur it.
|
||||
out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
|
||||
|
||||
published = EFI_CONFIG_RE.search(stdout)
|
||||
if published:
|
||||
out["memattr_published"] = True
|
||||
out["memattr_table_address"] = published.group(1)
|
||||
elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
|
||||
# UEFI booted and the config-table line is present, but it lists no MEMATTR=.
|
||||
out["memattr_published"] = False
|
||||
# else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
|
||||
|
||||
for line in stdout.splitlines():
|
||||
if MEMATTR_LINE_RE.search(line):
|
||||
out["memattr_lines"].append(line.strip())
|
||||
joined = "\n".join(out["memattr_lines"])
|
||||
if MEMATTR_CORRUPT_RE.search(joined):
|
||||
# Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
|
||||
out["memattr_state"] = "corrupt_rejected"
|
||||
elif MEMATTR_FAILED_RE.search(joined):
|
||||
out["memattr_state"] = "failed_to_map"
|
||||
addr = MEMATTR_FAILED_RE.search(joined).group(1)
|
||||
if addr:
|
||||
out["memattr_table_address"] = addr
|
||||
elif MEMATTR_VERSION_RE.search(joined):
|
||||
out["memattr_state"] = "bad_version"
|
||||
out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
|
||||
else:
|
||||
out["memattr_state"] = "clean"
|
||||
|
||||
mem_line = MEMORY_LINE_RE.search(stdout)
|
||||
if mem_line:
|
||||
full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
|
||||
reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
|
||||
out["memory_line"] = {
|
||||
"available_kb": int(mem_line.group(1)),
|
||||
"total_kb": int(mem_line.group(2)),
|
||||
"reserved_kb": int(reserved.group(1)) if reserved else None,
|
||||
"raw": full[0].strip() if full else None,
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
# --- Thermals -----------------------------------------------------------------------
|
||||
def collect_thermal():
|
||||
zones = []
|
||||
@@ -708,15 +996,34 @@ def collect_os():
|
||||
def collect_tools():
|
||||
tools = {}
|
||||
for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
|
||||
tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
|
||||
tools[tool] = which(tool)
|
||||
return tools
|
||||
|
||||
|
||||
def main():
|
||||
# --privileged: re-collect ONLY the fields that genuinely need root, for the second
|
||||
# pass roles/node_health makes when (and only when) escalation is proven to work
|
||||
# (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
|
||||
# re-implementing the kernel-log grep in Jinja would be a second source of truth for
|
||||
# the same fact, and the two would drift.
|
||||
if "--privileged" in sys.argv[1:]:
|
||||
doc = {
|
||||
"schema": SCHEMA,
|
||||
"collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"collector_uid": os.geteuid(),
|
||||
"klog": collect_klog(),
|
||||
"memory": {"devices": collect_memory_devices()},
|
||||
}
|
||||
doc["unreadable"] = UNREADABLE
|
||||
json.dump(doc, sys.stdout, indent=2, sort_keys=True)
|
||||
sys.stdout.write("\n")
|
||||
return
|
||||
|
||||
doc = {
|
||||
"schema": SCHEMA,
|
||||
"collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"collector_uid": os.geteuid(),
|
||||
"klog": collect_klog(),
|
||||
"os": collect_os(),
|
||||
"dmi": collect_dmi(),
|
||||
"cpu": collect_cpu(),
|
||||
|
||||
@@ -387,6 +387,83 @@
|
||||
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
|
||||
~ avail_pct | string ~ '%)') }}
|
||||
|
||||
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
|
||||
# one cause we have actually hit here; this one asks the blunter question — "is this
|
||||
# machine using the RAM it has?" — and answers it from two sources that share NOTHING
|
||||
# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
|
||||
# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
|
||||
- name: Assess — usable memory vs the memory this machine physically has
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.physical_total',
|
||||
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
||||
'status': phys_status,
|
||||
'detail': phys_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
th: "{{ node_health_thresholds }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
dev: "{{ node_health_facts.memory.devices }}"
|
||||
m: "{{ node_health_facts.memory }}"
|
||||
|
||||
# 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
|
||||
# actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
|
||||
# tools, identical on metal and in a VM — so this half is available almost always.
|
||||
z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
|
||||
z_status: >-
|
||||
{{ 'SKIP' if z_pct is not number
|
||||
else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
|
||||
else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
|
||||
|
||||
# 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
|
||||
# VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
|
||||
# never as a zero DIMM count. Independent of the kernel's memory map entirely.
|
||||
inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
|
||||
d_pct: >-
|
||||
{{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
|
||||
if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
|
||||
else none }}
|
||||
d_status: >-
|
||||
{{ 'SKIP' if d_pct is not number
|
||||
else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
|
||||
else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
|
||||
|
||||
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
|
||||
# SKIPs must never add up to a PASS.
|
||||
phys_status: >-
|
||||
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
|
||||
| default('SKIP', true) }}
|
||||
# What the kernel log says, IF it was read — quoted here only as attribution for a
|
||||
# shortfall this check measured on its own. The verdict above never depends on it.
|
||||
phys_cause: >-
|
||||
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
|
||||
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
|
||||
if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
|
||||
else ('. The kernel log was NOT read here, so this shortfall is unattributed —
|
||||
see check firmware.efi_memattr')
|
||||
if not (node_health_facts.klog.available | bool)
|
||||
else '' }}
|
||||
phys_detail: >-
|
||||
{{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
|
||||
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
|
||||
~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
|
||||
was never handed to the allocator (/proc/zoneinfo)')
|
||||
if z_pct is number
|
||||
else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
|
||||
~ '. '
|
||||
~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
|
||||
~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
|
||||
~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
|
||||
~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
|
||||
~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
|
||||
if d_pct is number
|
||||
else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
|
||||
~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
|
||||
reservations and (on this platform) 64 MiB of graphics stolen memory.
|
||||
A shortfall of this size is memory the machine HAS and cannot use')
|
||||
if phys_status in ['WARN', 'FAIL'] else '')
|
||||
~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
|
||||
|
||||
- name: Assess — swap (kubelet requires it off)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
@@ -674,3 +751,127 @@
|
||||
~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
|
||||
~ (' — serial numbers not read (root-only DMI fields)'
|
||||
if d.product_serial is none else '')) }}
|
||||
|
||||
# --- EFI Memory Attributes Table ------------------------------------------------------
|
||||
# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
|
||||
# what follows is only what this check needs to be read correctly.
|
||||
#
|
||||
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
|
||||
# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
|
||||
# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
|
||||
# the product — everything from the table address to the top of RAM. The kernel prints
|
||||
# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
|
||||
# happens to read > 2:
|
||||
#
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel
|
||||
# TRUSTED the table. The node has lost the memory. FAIL.
|
||||
# "Unexpected EFI Memory Attributes table version <N>" version read > 2, the kernel
|
||||
# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
|
||||
# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
|
||||
# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
|
||||
# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
|
||||
# unlucky boot from losing 83% of its RAM is not passing a health check.
|
||||
#
|
||||
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
|
||||
# so it cannot be diagnosed once. It has to be measured on every health run.
|
||||
#
|
||||
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
|
||||
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
|
||||
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
|
||||
# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
|
||||
# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
|
||||
# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
|
||||
# log at all, so a node whose log is unreadable is still not unmeasured.
|
||||
- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'firmware.efi_memattr',
|
||||
'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
|
||||
'status': memattr_status,
|
||||
'detail': memattr_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
k: "{{ node_health_facts.klog }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
ma_state: "{{ k.memattr_state | default(none) }}"
|
||||
memattr_status: >-
|
||||
{{ 'SKIP' if not (k.available | bool)
|
||||
else ('FAIL' if ma_state == 'failed_to_map'
|
||||
else ('WARN' if ma_state == 'bad_version'
|
||||
else ('PASS' if ma_state == 'corrupt_rejected'
|
||||
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
|
||||
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
|
||||
# kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
|
||||
# could take effect. Suggesting one would be a fix that quietly does nothing.
|
||||
memattr_fix: >-
|
||||
FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
|
||||
upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
|
||||
bogus", first released in v6.13 and absent from v6.12 and from the current
|
||||
linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
|
||||
There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
|
||||
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
|
||||
unplugging USB only change where the table lands and may APPEAR to fix it — they are
|
||||
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
||||
# The symptom, measured independently of this log (see memory.physical_total).
|
||||
memattr_corroboration: >-
|
||||
{{ (' Measured independently of this log: MemTotal '
|
||||
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
|
||||
~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
|
||||
from the allocator (/proc/zoneinfo) — check memory.physical_total.')
|
||||
if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
|
||||
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
|
||||
# against the evidence is not much better than a wrong one.
|
||||
memattr_evidence: >-
|
||||
{{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
|
||||
if (k.memattr_lines | default([]) | length) > 0 else '')
|
||||
~ ((' At boot the kernel reported '
|
||||
~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
|
||||
~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
|
||||
~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
|
||||
if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
|
||||
else '') }}
|
||||
memattr_detail: >-
|
||||
{%- if not (k.available | bool) -%}
|
||||
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
|
||||
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
|
||||
lives only in dmesg.{{ memattr_corroboration }}
|
||||
{%- elif ma_state == 'failed_to_map' -%}
|
||||
THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
|
||||
Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
|
||||
and reserved memory from the corrupt table anyway — everything from that address
|
||||
upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'bad_version' -%}
|
||||
EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
|
||||
here too; this boot survived only because its garbage version field read
|
||||
{{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
|
||||
before reserving anything. The outcome is re-decided on EVERY boot: if that field
|
||||
reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
|
||||
now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
|
||||
{{ memattr_fix }}
|
||||
{%- elif ma_state == 'corrupt_rejected' -%}
|
||||
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
|
||||
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
|
||||
intact and stays intact across reboots; no action.{{ memattr_evidence }}
|
||||
{%- elif not (k.boot_covered | bool) -%}
|
||||
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
|
||||
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
|
||||
the ring buffer, which wraps). Every memattr diagnostic is printed within the first
|
||||
fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
|
||||
pass. Read /var/log/kern.log for this boot, or re-run after a
|
||||
reboot.{{ memattr_corroboration }}
|
||||
{%- elif k.memattr_published is sameas true -%}
|
||||
the firmware publishes an EFI Memory Attributes table at
|
||||
{{ k.memattr_table_address | default('an unreported address') }} and the kernel
|
||||
accepted it without complaint, in a log that still reaches back to early boot. Not
|
||||
affected on this boot.
|
||||
{%- elif k.memattr_published is sameas false -%}
|
||||
this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
|
||||
lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
|
||||
guest.
|
||||
{%- else -%}
|
||||
no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
|
||||
so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
|
||||
boot, so that absence is measured, not assumed.
|
||||
{%- endif -%}
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
# The SECOND pass of the collector, run with escalation, for the two facts that are
|
||||
# genuinely root-gated:
|
||||
#
|
||||
# * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
|
||||
# unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
|
||||
# diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
|
||||
# between a node that has lost 13.3 GiB and one that has not.
|
||||
# * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
|
||||
# DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
|
||||
#
|
||||
# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
|
||||
# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
|
||||
# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
|
||||
# source of truth for the same fact, and the two would drift. `--privileged` makes the
|
||||
# collector emit exactly those subtrees and nothing else, so the parsing stays in one
|
||||
# place and is identical whichever pass produced it.
|
||||
#
|
||||
# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
|
||||
# actually failed to read one of them (main.yml checks both). On a node where
|
||||
# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
|
||||
# is that everything readable WITHOUT root is read without root.
|
||||
#
|
||||
# If escalation is unavailable the checks that consume these SKIP with the exact reason
|
||||
# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
|
||||
# and "could not read the kernel log" are different answers and the role says which.
|
||||
|
||||
- name: Re-collect the root-only facts with escalation
|
||||
ansible.builtin.script:
|
||||
cmd: node_health_collect.py --privileged
|
||||
executable: /usr/bin/python3
|
||||
become: true
|
||||
vars:
|
||||
ansible_become: true
|
||||
register: node_health_privileged_raw
|
||||
changed_when: false # strictly read-only; it can never change the node
|
||||
|
||||
- name: Assert the privileged payload is the schema this role understands
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
|
||||
fail_msg: >-
|
||||
Privileged collector pass returned an unexpected schema; role and collector are
|
||||
out of step.
|
||||
quiet: true
|
||||
vars:
|
||||
ansible_become: false
|
||||
|
||||
# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
|
||||
# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
|
||||
# would REPLACE it and throw away the reasons the first pass recorded.
|
||||
- name: Merge the privileged facts over the unprivileged ones
|
||||
ansible.builtin.set_fact:
|
||||
node_health_facts: >-
|
||||
{{ node_health_facts | combine({
|
||||
'klog': priv.klog,
|
||||
'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
|
||||
'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
|
||||
}) }}
|
||||
vars:
|
||||
ansible_become: false
|
||||
priv: "{{ node_health_privileged_raw.stdout | from_json }}"
|
||||
@@ -4,9 +4,11 @@
|
||||
# ORDER OF OPERATIONS
|
||||
# 1. Collect everything readable WITHOUT root (one script, one round trip).
|
||||
# 2. Probe whether privilege escalation actually works on this host.
|
||||
# 3. Only if it does AND smartmontools is installed: read SMART.
|
||||
# 4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
|
||||
# 5. Render the human summary + write the JSON artifact.
|
||||
# 3. Only if it does, and only for what the first pass could NOT read: re-run the same
|
||||
# collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
|
||||
# 4. Only if it does AND smartmontools is installed: read SMART.
|
||||
# 5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
|
||||
# 6. Render the human summary + write the JSON artifact.
|
||||
#
|
||||
# BECOME HANDLING — the subtle bit, read before editing.
|
||||
# `ansible_become` set as an inventory/group var (which is exactly what
|
||||
@@ -43,6 +45,16 @@
|
||||
ansible.builtin.include_tasks: probe_privilege.yml
|
||||
when: node_health_try_privileged | bool
|
||||
|
||||
# Only when the unprivileged pass actually failed to read one of them. On a node with
|
||||
# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
|
||||
# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
|
||||
- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
|
||||
ansible.builtin.include_tasks: collect_privileged.yml
|
||||
when:
|
||||
- node_health_root_available | bool
|
||||
- (not (node_health_facts.klog.available | bool))
|
||||
or (not (node_health_facts.memory.devices.available | bool))
|
||||
|
||||
- name: Read SMART data (privileged)
|
||||
ansible.builtin.include_tasks: collect_smart.yml
|
||||
when:
|
||||
|
||||
@@ -45,6 +45,15 @@
|
||||
{% endif %}
|
||||
Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
|
||||
ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
|
||||
{% set mz = node_health_facts.memory.zones %}
|
||||
{% set md = node_health_facts.memory.devices %}
|
||||
Present RAM {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
|
||||
DIMMs {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
|
||||
{% for dimm in md.devices | default([]) if dimm.size_kb %}
|
||||
{{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
|
||||
{% endfor %}
|
||||
{% set klog = node_health_facts.klog %}
|
||||
EFI memattr {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
|
||||
Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Reference in new issue
Block a user