diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 55202bf..30714f1 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -25,20 +25,124 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. |---|---|---|---| | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` | -| Model | HP EliteDesk 800 G5 Desktop Mini | | | -| CPU | i7-9700T, 8c/8t, VT-x | | | -| RAM | 31 GiB | | | -| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | | -| OS | Debian 13 (trixie), kernel 6.12 | | | -| Firmware | HP R21 Ver. 02.26.00 | | | -| `/dev/kvm` | present | | | +| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | +| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | +| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | | +| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | | +| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | | +| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | | +| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | +| `/dev/kvm` | present | present | | +| Serial | MXL1052NGN | MXL0073949 | | + +**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a +healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica +placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's +NVMe is the drive node 1 should have had. Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). --- +## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB + +Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install +more RAM until this is understood** — see the warning at the end. + +| Evidence | Reading | +|---|---| +| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | +| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | +| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | +| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | +| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | +| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | +| Memory blocks | 127 online, 128 MiB block size | + +**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The +kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and +everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping +above 4 GB not being applied**, not of failing memory. + +Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, +identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it +is per-machine state. + +**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` +kernel driver, which exposes **215 BIOS attributes** read/write at +`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords +are set (`authentication/*/is_enabled` = 0). + +**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was +made on 2026-09-01 and the FIRMWARE refused it: +`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. +HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent +— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. +Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS +Setup Password is deliberately provisioned (see §1). + +What the comparison ruled OUT as the cause: +- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and + `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. +- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on + either node, so it cannot be a mis-set option. +- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, + which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. +- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory + blocks report `online`. The memory reaches the kernel and is then reserved. + +Real BIOS differences found (node 1 = works, node 2 = broken): + +| Attribute | node 1 | node 2 | +|---|---|---| +| **After Power Loss** | Power On | **Power Off** ← violates §1 | +| **Fast Boot** | Disable | **Enable** ← leading hypothesis | +| Legacy Boot Options | Disable | Enable | +| TPM Activation Policy | F1 to Boot | Allow user to reject | +| Microsoft/Windows UEFI CA 2023 | Enable | Disable | +| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | + +**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* +rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale +training data is a plausible route to memory being present, inventoried, handed over — and then +reserved. It is also the one memory-relevant setting that differs from the working node. This is +a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write +and a reboot, which needs operator sanction. + +To investigate, cheapest first: +- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). + **Must be done in F10** — the firmware refuses programmatic writes (above). +- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and + node 2 currently violates it. +- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. +- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot + vs board in one move. +- [ ] `memtest86+` if the above do not resolve it. + +> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS +> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. +> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` +> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. + ## 1. Firmware / BIOS (per node, before OS install) +> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes +> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work +> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found +> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. +> +> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, +> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is +> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means +> F10 on the box. +> +> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is +> achievable today with reads alone, and would have caught node 2's drift on its first health run. +> Converging it automatically additionally requires provisioning a BIOS Setup Password on every +> node, which is a real security decision (a stored credential that can rewrite firmware) and is +> the operator's call, not a default. + + - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 62b1c23..f13087c 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 [k3s_prod_secondary] -# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md -# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so -# no play can accidentally target a host that isn't there; uncomment each line only once -# that node is actually on the bench and answering SSH. -# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 +# Uncomment a node only once it is actually on the bench and answering SSH, so no play can +# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to +# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same +# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the +# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision. +# node 3 remains commented until it is racked. +k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]