From cb69a53825f7f662eccb298a77569d9c6e7c382c Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 20:56:58 -0500 Subject: [PATCH] =?UTF-8?q?feat(prod):=20enable=20node=202=20+=20HARDWARE?= =?UTF-8?q?=20FAULT=20=E2=80=94=20it=20addresses=202.6=20GiB=20of=20its=20?= =?UTF-8?q?16=20GB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package floor installed, first health run completed. Vaulted become works on it unchanged. *** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB DIMM. *** Recorded in PROD-PROVISIONING §0b. dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty) BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff kernel Memory: 2568544K/16546088K available (13958980K RESERVED) last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0 MemTotal 2,753,284 kB = 2.6 GiB The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21 Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a firmware-version defect. WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is wasted. Verify MemTotal after any RAM change. ALSO FOUND — the prod nodes are NOT hardware-symmetric: node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase failures, 32 unexpected power losses node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected power losses, overall-health PASSED Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote. Node 2's NVMe is the drive node 1 should have had. Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses), plus the same unknowable-ECC SKIP. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- infra/PROD-PROVISIONING.md | 37 +++++++++++++++++++++++++++++++ infra/ansible/inventory/hosts.ini | 12 +++++----- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 55202bf..e4e9d24 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -37,6 +37,43 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- +## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB + +Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install +more RAM until this is understood** — see the warning at the end. + +| Evidence | Reading | +|---|---| +| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | +| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | +| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | +| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | +| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | +| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | +| Memory blocks | 127 online, 128 MiB block size | + +**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The +kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and +everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping +above 4 GB not being applied**, not of failing memory. + +Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, +identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it +is per-machine state. + +To investigate, cheapest first: +- [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G + decoding" style option, and for any memory-limit or 32-bit-addressing setting. +- [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. +- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot + vs board in one move. +- [ ] `memtest86+` if the above do not resolve it. + +> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS +> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. +> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` +> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. + ## 1. Firmware / BIOS (per node, before OS install) - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 62b1c23..f13087c 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 [k3s_prod_secondary] -# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md -# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so -# no play can accidentally target a host that isn't there; uncomment each line only once -# that node is actually on the bench and answering SSH. -# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 +# Uncomment a node only once it is actually on the bench and answering SSH, so no play can +# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to +# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same +# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the +# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision. +# node 3 remains commented until it is racked. +k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]