docs(prod): fill in node-2 inventory; drop stale investigate bullets

Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice). Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a ~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2 is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the disk decision will be made. Serves: R-ENV-03. Task: T-7.1.
docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable
2026-09-01 21:45:37 -05:00 · 2026-09-01 21:45:15 -05:00 · 2026-09-01 21:04:12 -05:00 · 2026-09-01 20:56:58 -05:00 · 2026-09-02 00:24:57 +00:00 · 2026-09-02 00:10:06 +00:00
2 changed files with 118 additions and 12 deletions
@@ -25,20 +25,124 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
 |---|---|---|---|
 | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
 | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
-| Model | HP EliteDesk 800 G5 Desktop Mini | | |
-| CPU | i7-9700T, 8c/8t, VT-x | | |
-| RAM | 31 GiB | | |
-| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
-| OS | Debian 13 (trixie), kernel 6.12 | | |
-| Firmware | HP R21 Ver. 02.26.00 | | |
-| `/dev/kvm` | present | | |
+| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
+| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
+| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
+| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
+| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
+| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
+| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
+| `/dev/kvm` | present | present | |
+| Serial | MXL1052NGN | MXL0073949 | |
+
+**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
+healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
+placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
+NVMe is the drive node 1 should have had.

 Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).

 ---

+## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB
+
+Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install
+more RAM until this is understood** — see the warning at the end.
+
+| Evidence | Reading |
+|---|---|
+| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty |
+| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` |
+| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` |
+| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot |
+| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** |
+| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** |
+| Memory blocks | 127 online, 128 MiB block size |
+
+**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The
+kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and
+everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping
+above 4 GB not being applied**, not of failing memory.
+
+Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00,
+identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it
+is per-machine state.
+
+**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg`
+kernel driver, which exposes **215 BIOS attributes** read/write at
+`/sys/class/firmware-attributes/hp-bioscfg/attributes/<name>/current_value`. No BIOS passwords
+are set (`authentication/*/is_enabled` = 0).
+
+**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was
+made on 2026-09-01 and the FIRMWARE refused it:
+`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`.
+HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent
+— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`.
+Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS
+Setup Password is deliberately provisioned (see §1).
+
+What the comparison ruled OUT as the cause:
+- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and
+  `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting.
+- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on
+  either node, so it cannot be a mis-set option.
+- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**,
+  which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path.
+- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory
+  blocks report `online`. The memory reaches the kernel and is then reserved.
+
+Real BIOS differences found (node 1 = works, node 2 = broken):
+
+| Attribute | node 1 | node 2 |
+|---|---|---|
+| **After Power Loss** | Power On | **Power Off** ← violates §1 |
+| **Fast Boot** | Disable | **Enable** ← leading hypothesis |
+| Legacy Boot Options | Disable | Enable |
+| TPM Activation Policy | F1 to Boot | Allow user to reject |
+| Microsoft/Windows UEFI CA 2023 | Enable | Disable |
+| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) |
+
+**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data*
+rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale
+training data is a plausible route to memory being present, inventoried, handed over — and then
+reserved. It is also the one memory-relevant setting that differs from the working node. This is
+a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write
+and a reboot, which needs operator sanction.
+
+To investigate, cheapest first:
+- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1).
+      **Must be done in F10** — the firmware refuses programmatic writes (above).
+- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and
+      node 2 currently violates it.
+- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check.
+- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot
+      vs board in one move.
+- [ ] `memtest86+` if the above do not resolve it.
+
+> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS
+> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted.
+> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod`
+> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share.
+
 ## 1. Firmware / BIOS (per node, before OS install)

+> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
+> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work
+> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
+> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
+>
+> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
+> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
+> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
+> F10 on the box.
+>
+> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
+> achievable today with reads alone, and would have caught node 2's drift on its first health run.
+> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
+> node, which is a real security decision (a stored credential that can rewrite firmware) and is
+> the operator's call, not a default.
+
+
 - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
      boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
 - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
@@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10

 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
-# no play can accidentally target a host that isn't there; uncomment each line only once
-# that node is actually on the bench and answering SSH.
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
+# node 3 remains commented until it is racked.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12

 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
Author	SHA1	Message	Date
jakman85	9f7052ab2b	docs(prod): fill in node-2 inventory; drop stale investigate bullets Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice). Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a ~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2 is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the disk decision will be made. Serves: R-ENV-03. Task: T-7.1.	2026-09-01 21:45:37 -05:00
jakman85	3c8f809aaf	docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the write was actually attempted: hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error) HP requires BIOS-admin authentication for programmatic changes and neither credential is provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the firmware is. What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots 'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today with reads alone, and node_health should assert it. What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite firmware. That is a real security decision and the operator's call, not something to default to. Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On must be done in F10 on the box. Updated §0b and §1 accordingly. Serves: R-ENV-03. Task: T-7.1.	2026-09-01 21:45:15 -05:00
jakman85	11b21f7347	docs(prod): BIOS settings compared programmatically via hp-bioscfg Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at /sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS. RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full 100000000-42f7fffff as System RAM and 127 memory blocks report online. REAL DIFFERENCES FOUND (node1 works / node2 broken): After Power Loss Power On / Power Off <- VIOLATES this document's own §1 Fast Boot Disable / Enable <- leading hypothesis Legacy Boot Options Disable / Enable TPM Activation Policy F1 to Boot / Allow user to reject MS+Windows UEFI CA 2023 Enable / Disable Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present, inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant setting differing from the working node. NOT TESTED: changing it is a firmware write plus a reboot and needs operator sanction. BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up recorded: extend node_health to assert the firmware checklist, add a role to converge it. Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same reason node-health artifacts are gitignored. Only the findings are recorded. Serves: R-ENV-03. Task: T-7.1.	2026-09-01 21:04:12 -05:00
jakman85andClaude Opus 5	cb69a53825	feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package floor installed, first health run completed. Vaulted become works on it unchanged. * OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB DIMM. * Recorded in PROD-PROVISIONING §0b. dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty) BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff kernel Memory: 2568544K/16546088K available (13958980K RESERVED) last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0 MemTotal 2,753,284 kB = 2.6 GiB The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21 Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a firmware-version defect. WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is wasted. Verify MemTotal after any RAM change. ALSO FOUND — the prod nodes are NOT hardware-symmetric: node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase failures, 32 unexpected power losses node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected power losses, overall-health PASSED Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote. Node 2's NVMe is the drive node 1 should have had. Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses), plus the same unknowable-ECC SKIP. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 20:56:58 -05:00
gitea-admin	f1daa8cae0	Merge pull request 'PROVEN: 1 → 2 — R-STATE-03 DR restore executed and proven' (#12 ) from feat/r-state-03-proven into main	2026-09-02 00:24:57 +00:00
gitea-admin	acae2ad36c	Merge pull request 'PROVEN: 0 → 1 — R-STATE-02 proven, plus two ledger bugs that made proof impossible' (#11 ) from feat/ledger-section-parsing into main	2026-09-02 00:10:06 +00:00
gitea-admin	14494e1947	Merge pull request 'feat(conformance): committed run ledger — prerequisite for moving PROVEN off zero' (#10 ) from feat/conformance-run-ledger into main	2026-09-01 23:50:28 +00:00