Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven

Browse Source 
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.

*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.

THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.

BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
  - Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
    'skips some preboot initialization steps', which changes the heap top and hence where the MAT
    lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
  - Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
    'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
    'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
  - the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
    node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.

EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.

ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.

Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-03 22:32:25 -05:00
1 parent 6e4d311545
commit b19c5db9ae
3 files changed
+179 -33
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/PROD-PROVISIONING.md
+145 -20
View File
@@ -25,15 +25,15 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|---|---|---|---|
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
| `/dev/kvm` | present | present | |
| Serial | MXL1052NGN | MXL0073949 | |
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
| `/dev/kvm` | present | present | present |
| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
@@ -41,10 +41,21 @@ unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2
but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
NVMe is the drive node 1 should have had.
**The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is
THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy
modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life,
0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires
an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore
WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine.
Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so
`node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the
three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either.
Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
@@ -92,15 +103,42 @@ memblock_reserve(efi_mem_attr_table, tbl_size);
That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
### BOTH prod nodes are affected — node 1 has only been lucky
### ALL THREE prod nodes are affected — none is immune
Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every
node with settings held constant**, and each node has been seen on more than one path:
| Node | Boot | `MEMATTR=` | Kernel's read | Memory |
|---|---|---|---|---|
| 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ |
| 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ |
| 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ |
| 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
| 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
| 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ |
| 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ |
So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a
sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the
others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a
**valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox
SeaBIOS/OVMF) and are genuinely unaffected.
**All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.**
### The corrupting bytes are the OS boot chain, not the firmware or the DRAM
The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different
machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM
fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement),
writing the same data into the page the firmware pointed at. This is the documented failure mode —
the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates
there next overwrites it.
It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is
part of the causal chain, which matters for the experiment design below.
| Node | memattr version field | Outcome |
|---|---|---|
| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |
Node 1 is one unlucky boot away from the identical fault. **Fix both.**
### Mechanism, traced to the byte (second, independent confirmation)
@@ -212,6 +250,93 @@ action beyond `apt upgrade`.
heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not
required for this symptom and depends on the first commit anyway.
### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.**
The operator's hypothesis, tested properly now that there are two working nodes and one broken one.
All **240** attributes read on all three; identical key sets; identical firmware everywhere (same
BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings
or hardware, not firmware revision.
**The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ
in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature
Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven
remain:
| Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict |
|---|---|---|---|
| **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** |
| Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** |
| Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) |
| TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence |
| Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) |
**`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's
`Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy,
applied before firmware executes). It remains a §1 checklist violation on **two** nodes now.
**`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are
**identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` =
*"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All
UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the
master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is
built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time
allocation footprint at all**. Secure Boot is likewise not a differential — all three log
`Secure boot enabled` and load the same HP DB cert.
**`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this
document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by
bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped
preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒
**the MAT lands at a different address**. That is exactly the perturbation that flipped the
near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382),
where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training
theory stays dead — byte-identical `BIOS-e820` refuted it.)
#### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle
1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and
the MAT address moves every boot on all three machines.
2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence
that it does.
3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware
differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and
the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3
did not break the confound; it doubled it.** A different boot device means a different initrd
module set and size, hence a different EFI-stub allocation footprint — the very thing that decides
whether the MAT page gets overwritten.
**No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes
Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options
(`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No
HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated
05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase.
#### The experiment, if it is ever worth running
Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of
3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even
8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting
*elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per
arm** for 80% power. **Do not design for an odds shift — design for elimination and report
"inconclusive" otherwise.**
**Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with
*nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That
establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted
by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five
successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled
without touching anything.
**If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory
topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10
visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node
3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings
exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing.
Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.
**None of this changes the action.** The fix is already known, is not in dispute, and applies to all
three nodes: kernel ≥ 6.13.
### What NOT to do
- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is