docs(prod): BIOS settings compared programmatically via hp-bioscfg

Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at
/sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS.

RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself
reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at
all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full
100000000-42f7fffff as System RAM and 127 memory blocks report online.

REAL DIFFERENCES FOUND (node1 works / node2 broken):
  After Power Loss          Power On  / Power Off   <- VIOLATES this document's own §1
  Fast Boot                 Disable   / Enable      <- leading hypothesis
  Legacy Boot Options       Disable   / Enable
  TPM Activation Policy     F1 to Boot / Allow user to reject
  MS+Windows UEFI CA 2023   Enable    / Disable

Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each
boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present,
inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant
setting differing from the working node. NOT TESTED: changing it is a firmware write plus a
reboot and needs operator sanction.

BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an
eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have
After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up
recorded: extend node_health to assert the firmware checklist, add a role to converge it.

Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same
reason node-health artifacts are gitignored. Only the findings are recorded.

Serves: R-ENV-03. Task: T-7.1.

infra/PROD-PROVISIONING.md

@@ -61,7 +61,46 @@ Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIO
 identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it
 is per-machine state.
 
+**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg`
+kernel driver, which exposes **215 BIOS attributes** read/write at
+`/sys/class/firmware-attributes/hp-bioscfg/attributes/<name>/current_value`. No BIOS passwords
+are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see
+§1, this changes how the firmware checklist can be enforced.
+
+What the comparison ruled OUT as the cause:
+- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and
+  `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting.
+- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on
+  either node, so it cannot be a mis-set option.
+- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**,
+  which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path.
+- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory
+  blocks report `online`. The memory reaches the kernel and is then reserved.
+
+Real BIOS differences found (node 1 = works, node 2 = broken):
+
+| Attribute | node 1 | node 2 |
+|---|---|---|
+| **After Power Loss** | Power On | **Power Off** ← violates §1 |
+| **Fast Boot** | Disable | **Enable** ← leading hypothesis |
+| Legacy Boot Options | Disable | Enable |
+| TPM Activation Policy | F1 to Boot | Allow user to reject |
+| Microsoft/Windows UEFI CA 2023 | Enable | Disable |
+| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) |
+
+**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data*
+rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale
+training data is a plausible route to memory being present, inventoried, handed over — and then
+reserved. It is also the one memory-relevant setting that differs from the working node. This is
+a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write
+and a reboot, which needs operator sanction.
+
 To investigate, cheapest first:
+- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1).
+      Can be done from the OS — see §1 — or in the F10 setup menu.
+- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and
+      node 2 currently violates it.
+- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check.
 - [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G
       decoding" style option, and for any memory-limit or 32-bit-addressing setting.
 - [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`.
@@ -76,6 +115,15 @@ To investigate, cheapest first:
 
 ## 1. Firmware / BIOS (per node, before OS install)
 
+> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver
+> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and
+> with no BIOS password set they are **writable from the OS**. That means the checklist below can
+> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever
+> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power
+> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend
+> node_health to assert the firmware checklist, and add a role to converge it.**
+
+
 - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
       boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
 - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a