diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 55202bf..e4e9d24 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -37,6 +37,43 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- +## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB + +Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install +more RAM until this is understood** — see the warning at the end. + +| Evidence | Reading | +|---|---| +| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | +| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | +| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | +| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | +| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | +| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | +| Memory blocks | 127 online, 128 MiB block size | + +**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The +kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and +everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping +above 4 GB not being applied**, not of failing memory. + +Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, +identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it +is per-machine state. + +To investigate, cheapest first: +- [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G + decoding" style option, and for any memory-limit or 32-bit-addressing setting. +- [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. +- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot + vs board in one move. +- [ ] `memtest86+` if the above do not resolve it. + +> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS +> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. +> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` +> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. + ## 1. Firmware / BIOS (per node, before OS install) - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 62b1c23..f13087c 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 [k3s_prod_secondary] -# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md -# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so -# no play can accidentally target a host that isn't there; uncomment each line only once -# that node is actually on the bench and answering SSH. -# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 +# Uncomment a node only once it is actually on the bench and answering SSH, so no play can +# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to +# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same +# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the +# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision. +# node 3 remains commented until it is racked. +k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]