From 11b21f7347e49fa0bbfae5cd7be251555bd7c1ac Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 21:04:12 -0500 Subject: [PATCH] docs(prod): BIOS settings compared programmatically via hp-bioscfg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at /sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS. RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full 100000000-42f7fffff as System RAM and 127 memory blocks report online. REAL DIFFERENCES FOUND (node1 works / node2 broken): After Power Loss Power On / Power Off <- VIOLATES this document's own §1 Fast Boot Disable / Enable <- leading hypothesis Legacy Boot Options Disable / Enable TPM Activation Policy F1 to Boot / Allow user to reject MS+Windows UEFI CA 2023 Enable / Disable Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present, inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant setting differing from the working node. NOT TESTED: changing it is a firmware write plus a reboot and needs operator sanction. BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up recorded: extend node_health to assert the firmware checklist, add a role to converge it. Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same reason node-health artifacts are gitignored. Only the findings are recorded. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 48 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index e4e9d24..8a9be4e 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -61,7 +61,46 @@ Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIO identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it is per-machine state. +**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` +kernel driver, which exposes **215 BIOS attributes** read/write at +`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords +are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see +§1, this changes how the firmware checklist can be enforced. + +What the comparison ruled OUT as the cause: +- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and + `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. +- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on + either node, so it cannot be a mis-set option. +- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, + which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. +- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory + blocks report `online`. The memory reaches the kernel and is then reserved. + +Real BIOS differences found (node 1 = works, node 2 = broken): + +| Attribute | node 1 | node 2 | +|---|---|---| +| **After Power Loss** | Power On | **Power Off** ← violates §1 | +| **Fast Boot** | Disable | **Enable** ← leading hypothesis | +| Legacy Boot Options | Disable | Enable | +| TPM Activation Policy | F1 to Boot | Allow user to reject | +| Microsoft/Windows UEFI CA 2023 | Enable | Disable | +| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | + +**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* +rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale +training data is a plausible route to memory being present, inventoried, handed over — and then +reserved. It is also the one memory-relevant setting that differs from the working node. This is +a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write +and a reboot, which needs operator sanction. + To investigate, cheapest first: +- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). + Can be done from the OS — see §1 — or in the F10 setup menu. +- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and + node 2 currently violates it. +- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. - [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G decoding" style option, and for any memory-limit or 32-bit-addressing setting. - [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. @@ -76,6 +115,15 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) +> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver +> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and +> with no BIOS password set they are **writable from the OS**. That means the checklist below can +> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever +> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power +> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend +> node_health to assert the firmware checklist, and add a role to converge it.** + + - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a