From df1d2ab44ddfa6ae5ddd8da20a1913901a3a1b54 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 22:50:45 -0500 Subject: [PATCH] =?UTF-8?q?docs(prod):=20node=201's=20drive=20was=20replac?= =?UTF-8?q?ed=20=E2=80=94=20and=20=C2=A72's=20exit=20condition=20still=20i?= =?UTF-8?q?s=20not=20met?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 --- infra/PROD-PROVISIONING.md | 47 +++++++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 682e980..89c4425 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -28,9 +28,9 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. | Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) | -| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) | -| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC | -| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 | +| Disks | 1× 466 GB SATA SSD (`Samsung SSD 850 EVO 500GB`) — **replaced 2026-09-07, was `C400-MTFDDAC512M` (~2011); see §0c** | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) | +| Drive health | PASSED, **0 reallocated / 0 grown bad blocks**, `Wear_Leveling_Count` raw 106, but **77,047 power-on hours** (~8.8 y) over only 219 power cycles | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC | +| OS | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | | Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | `/dev/kvm` | present | present | present | | Serial | MXL1052NGN | MXL0073949 | 8CC015307N | @@ -48,6 +48,47 @@ modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent h an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs** + +### §0c — node 1's drive was replaced (2026-09-07) + +**The paragraph above is history, not current state.** On 2026-09-07 node 1's kubelet stopped +posting status while the host stayed up and SSH-able. Root cause was **etcd starved of disk I/O**, +not memory: `slow fdatasync` at 3.5–6.7 s against an expected 1 s, `apply request took too long` at +8.3 s against 100 ms, then `etcdserver: too many requests`. No OOM, MCE or hung task in the kernel +log. All three nodes place etcd's WAL (`/data/k3s`) and every Longhorn replica (`/data/longhorn`) on +the SAME partition; node 1's was the 2011 C400 with 61,440 reallocated sectors and ~31% of its +endurance consumed, and its `/proc/pressure/io` `full avg300` ran **11.36** against ~1% steady state +on the other two. The failure is self-reinforcing: the node dropping out degrades Longhorn volumes, +whose rebuilds add write load to the very disk etcd needs. + +The operator replaced the drive with a **Samsung SSD 850 EVO 500GB** and reinstalled Debian. +Measured under the same load class — an active 18-replica Longhorn rebuild, with node 1 as the +*write target*, i.e. the heavier side: + +| node | drive | `full avg300` during rebuild | +|---|---|---| +| `k3s-prod-1` | Samsung 850 EVO (new) | **3.79** | +| `k3s-prod-2` | Samsung PM9A1 NVMe | 0.33 | +| `k3s-prod-3` | WDC WDS500G2B0A | 2.47 | +| *`k3s-prod-1` before* | *C400-MTFDDAC512MAM* | ***11.36*** | + +Roughly a **3× improvement**, moving node 1 from ~4× worse than its peers into the same band as +node 3, and it stayed `Ready` throughout. **The 850 EVO is not new**: 77,047 power-on hours over 219 +power cycles — it evidently idled in an always-on system — but its wear tells the opposite story to +the C400's (0 reallocated vs 61,440; wear-leveling 106 vs 949). + +**§2's exit condition is NOT met by this replacement.** etcd and Longhorn still share one SATA +spindle. The contention is reduced, not eliminated; giving etcd its own device remains the fix, and +node 1 still has exactly one disk. The three nodes also remain asymmetric — now 850 EVO / PM9A1 +NVMe / WD Blue. + +The rebuild itself surfaced two things worth carrying forward. Longhorn refused the new disk with +`DiskFilesystemChanged` — "record diskUUID doesn't match the one on the disk" — because its node CR +still held the old drive's `diskUUID`; the remedy is disable → remove the disk entry → re-add, in +that order (the validating webhook enforces it). And the Ansible inventory still listed node 1 in +`[k3s_prod_primary]`, which decides both which play gets `--cluster-init` *and* what `k3s_api_host` +resolves to — so re-provisioning it unchanged would have pointed a rebuilt, empty node at itself and +founded a second cluster beside the live one. The founder now sits on node 2. (2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to