diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index e4e9d24..8a9be4e 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -61,7 +61,46 @@ Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIO identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it is per-machine state. +**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` +kernel driver, which exposes **215 BIOS attributes** read/write at +`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords +are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see +§1, this changes how the firmware checklist can be enforced. + +What the comparison ruled OUT as the cause: +- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and + `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. +- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on + either node, so it cannot be a mis-set option. +- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, + which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. +- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory + blocks report `online`. The memory reaches the kernel and is then reserved. + +Real BIOS differences found (node 1 = works, node 2 = broken): + +| Attribute | node 1 | node 2 | +|---|---|---| +| **After Power Loss** | Power On | **Power Off** ← violates §1 | +| **Fast Boot** | Disable | **Enable** ← leading hypothesis | +| Legacy Boot Options | Disable | Enable | +| TPM Activation Policy | F1 to Boot | Allow user to reject | +| Microsoft/Windows UEFI CA 2023 | Enable | Disable | +| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | + +**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* +rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale +training data is a plausible route to memory being present, inventoried, handed over — and then +reserved. It is also the one memory-relevant setting that differs from the working node. This is +a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write +and a reboot, which needs operator sanction. + To investigate, cheapest first: +- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). + Can be done from the OS — see §1 — or in the F10 setup menu. +- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and + node 2 currently violates it. +- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. - [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G decoding" style option, and for any memory-limit or 32-bit-addressing setting. - [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. @@ -76,6 +115,15 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) +> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver +> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and +> with no BIOS password set they are **writable from the OS**. That means the checklist below can +> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever +> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power +> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend +> node_health to assert the firmware checklist, and add a role to converge it.** + + - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a