From cb69a53825f7f662eccb298a77569d9c6e7c382c Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 20:56:58 -0500 Subject: [PATCH 1/4] =?UTF-8?q?feat(prod):=20enable=20node=202=20+=20HARDW?= =?UTF-8?q?ARE=20FAULT=20=E2=80=94=20it=20addresses=202.6=20GiB=20of=20its?= =?UTF-8?q?=2016=20GB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package floor installed, first health run completed. Vaulted become works on it unchanged. *** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB DIMM. *** Recorded in PROD-PROVISIONING §0b. dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty) BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff kernel Memory: 2568544K/16546088K available (13958980K RESERVED) last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0 MemTotal 2,753,284 kB = 2.6 GiB The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21 Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a firmware-version defect. WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is wasted. Verify MemTotal after any RAM change. ALSO FOUND — the prod nodes are NOT hardware-symmetric: node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase failures, 32 unexpected power losses node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected power losses, overall-health PASSED Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote. Node 2's NVMe is the drive node 1 should have had. Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses), plus the same unknowable-ECC SKIP. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- infra/PROD-PROVISIONING.md | 37 +++++++++++++++++++++++++++++++ infra/ansible/inventory/hosts.ini | 12 +++++----- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 55202bf..e4e9d24 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -37,6 +37,43 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di --- +## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB + +Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install +more RAM until this is understood** — see the warning at the end. + +| Evidence | Reading | +|---|---| +| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | +| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | +| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | +| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | +| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | +| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | +| Memory blocks | 127 online, 128 MiB block size | + +**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The +kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and +everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping +above 4 GB not being applied**, not of failing memory. + +Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, +identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it +is per-machine state. + +To investigate, cheapest first: +- [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G + decoding" style option, and for any memory-limit or 32-bit-addressing setting. +- [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. +- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot + vs board in one move. +- [ ] `memtest86+` if the above do not resolve it. + +> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS +> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. +> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` +> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. + ## 1. Firmware / BIOS (per node, before OS install) - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 62b1c23..f13087c 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 [k3s_prod_secondary] -# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md -# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so -# no play can accidentally target a host that isn't there; uncomment each line only once -# that node is actually on the bench and answering SSH. -# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 +# Uncomment a node only once it is actually on the bench and answering SSH, so no play can +# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to +# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same +# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the +# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision. +# node 3 remains commented until it is racked. +k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] From 11b21f7347e49fa0bbfae5cd7be251555bd7c1ac Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 21:04:12 -0500 Subject: [PATCH 2/4] docs(prod): BIOS settings compared programmatically via hp-bioscfg MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at /sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS. RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full 100000000-42f7fffff as System RAM and 127 memory blocks report online. REAL DIFFERENCES FOUND (node1 works / node2 broken): After Power Loss Power On / Power Off <- VIOLATES this document's own §1 Fast Boot Disable / Enable <- leading hypothesis Legacy Boot Options Disable / Enable TPM Activation Policy F1 to Boot / Allow user to reject MS+Windows UEFI CA 2023 Enable / Disable Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present, inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant setting differing from the working node. NOT TESTED: changing it is a firmware write plus a reboot and needs operator sanction. BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up recorded: extend node_health to assert the firmware checklist, add a role to converge it. Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same reason node-health artifacts are gitignored. Only the findings are recorded. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 48 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index e4e9d24..8a9be4e 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -61,7 +61,46 @@ Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIO identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it is per-machine state. +**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` +kernel driver, which exposes **215 BIOS attributes** read/write at +`/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords +are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see +§1, this changes how the firmware checklist can be enforced. + +What the comparison ruled OUT as the cause: +- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and + `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. +- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on + either node, so it cannot be a mis-set option. +- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, + which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. +- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory + blocks report `online`. The memory reaches the kernel and is then reserved. + +Real BIOS differences found (node 1 = works, node 2 = broken): + +| Attribute | node 1 | node 2 | +|---|---|---| +| **After Power Loss** | Power On | **Power Off** ← violates §1 | +| **Fast Boot** | Disable | **Enable** ← leading hypothesis | +| Legacy Boot Options | Disable | Enable | +| TPM Activation Policy | F1 to Boot | Allow user to reject | +| Microsoft/Windows UEFI CA 2023 | Enable | Disable | +| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | + +**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* +rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale +training data is a plausible route to memory being present, inventoried, handed over — and then +reserved. It is also the one memory-relevant setting that differs from the working node. This is +a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write +and a reboot, which needs operator sanction. + To investigate, cheapest first: +- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). + Can be done from the OS — see §1 — or in the F10 setup menu. +- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and + node 2 currently violates it. +- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. - [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G decoding" style option, and for any memory-limit or 32-bit-addressing setting. - [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. @@ -76,6 +115,15 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) +> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver +> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and +> with no BIOS password set they are **writable from the OS**. That means the checklist below can +> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever +> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power +> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend +> node_health to assert the firmware checklist, and add a role to converge it.** + + - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a From 3c8f809aaf4695c25205e337c7b23f68f832cd24 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 21:45:15 -0500 Subject: [PATCH 3/4] =?UTF-8?q?docs(prod):=20CORRECTION=20=E2=80=94=20BIOS?= =?UTF-8?q?=20attributes=20are=20read-only=20in=20practice,=20not=20writab?= =?UTF-8?q?le?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the write was actually attempted: hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error) HP requires BIOS-admin authentication for programmatic changes and neither credential is provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the firmware is. What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots 'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today with reads alone, and node_health should assert it. What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite firmware. That is a real security decision and the operator's call, not something to default to. Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On must be done in F10 on the box. Updated §0b and §1 accordingly. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 8a9be4e..295a536 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -64,8 +64,15 @@ is per-machine state. **BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` kernel driver, which exposes **215 BIOS attributes** read/write at `/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see -§1, this changes how the firmware checklist can be enforced. +are set (`authentication/*/is_enabled` = 0). + +**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was +made on 2026-09-01 and the FIRMWARE refused it: +`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. +HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent +— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. +Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS +Setup Password is deliberately provisioned (see §1). What the comparison ruled OUT as the cause: - **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and @@ -97,7 +104,7 @@ and a reboot, which needs operator sanction. To investigate, cheapest first: - [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - Can be done from the OS — see §1 — or in the F10 setup menu. + **Must be done in F10** — the firmware refuses programmatic writes (above). - [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and node 2 currently violates it. - [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. @@ -115,13 +122,21 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) -> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver -> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and -> with no BIOS password set they are **writable from the OS**. That means the checklist below can -> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever -> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power -> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend -> node_health to assert the firmware checklist, and add a role to converge it.** +> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes +> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work +> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found +> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. +> +> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, +> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is +> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means +> F10 on the box. +> +> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is +> achievable today with reads alone, and would have caught node 2's drift on its first health run. +> Converging it automatically additionally requires provisioning a BIOS Setup Password on every +> node, which is a real security decision (a stored credential that can rewrite firmware) and is +> the operator's call, not a default. - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after From 9f7052ab2b3e119ce54ddd1e9c2e467395418237 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 21:45:37 -0500 Subject: [PATCH 4/4] docs(prod): fill in node-2 inventory; drop stale investigate bullets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice). Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a ~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2 is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the disk decision will be made. Serves: R-ENV-03. Task: T-7.1. --- infra/PROD-PROVISIONING.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 295a536..30714f1 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -25,13 +25,20 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. |---|---|---|---| | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` | -| Model | HP EliteDesk 800 G5 Desktop Mini | | | -| CPU | i7-9700T, 8c/8t, VT-x | | | -| RAM | 31 GiB | | | -| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | | -| OS | Debian 13 (trixie), kernel 6.12 | | | -| Firmware | HP R21 Ver. 02.26.00 | | | -| `/dev/kvm` | present | | | +| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | +| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | +| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | | +| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | | +| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | | +| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | | +| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | +| `/dev/kvm` | present | present | | +| Serial | MXL1052NGN | MXL0073949 | | + +**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a +healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica +placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's +NVMe is the drive node 1 should have had. Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). @@ -108,9 +115,6 @@ To investigate, cheapest first: - [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and node 2 currently violates it. - [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. -- [ ] Compare node 2's BIOS settings against node 1 directly. Look for a memory remap / "Above 4G - decoding" style option, and for any memory-limit or 32-bit-addressing setting. -- [ ] Clear CMOS / load setup defaults on node 2, then re-check `MemTotal`. - [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot vs board in one move. - [ ] `memtest86+` if the above do not resolve it.