docs(prod): node 1's drive was replaced — and §2's exit condition still is not met

Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was
replaced with a Samsung 850 EVO, and under the same load class (an active
18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell
from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band.

Updates the three §0 cells that had become false (disk model, drive health, kernel
— all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the
'NOT hardware-symmetric' analysis, which is still the correct reasoning about the
state it described.

Two things the rebuild taught, recorded so the next one is cheaper: Longhorn
refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove ->
re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH
--cluster-init and k3s_api_host, so rebuilding the founder without moving that
designation would have founded a second cluster.

The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PROD-PROVISIONING.md

@@ -28,9 +28,9 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
 | Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
 | CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
 | RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
-| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
-| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
-| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
+| Disks | 1× 466 GB SATA SSD (`Samsung SSD 850 EVO 500GB`) — **replaced 2026-09-07, was `C400-MTFDDAC512M` (~2011); see §0c** | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
+| Drive health | PASSED, **0 reallocated / 0 grown bad blocks**, `Wear_Leveling_Count` raw 106, but **77,047 power-on hours** (~8.8 y) over only 219 power cycles | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
+| OS | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** |
 | Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
 | `/dev/kvm` | present | present | present |
 | Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
@@ -48,6 +48,47 @@ modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent h
 an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
 this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
 have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
+
+### §0c — node 1's drive was replaced (2026-09-07)
+
+**The paragraph above is history, not current state.** On 2026-09-07 node 1's kubelet stopped
+posting status while the host stayed up and SSH-able. Root cause was **etcd starved of disk I/O**,
+not memory: `slow fdatasync` at 3.5–6.7 s against an expected 1 s, `apply request took too long` at
+8.3 s against 100 ms, then `etcdserver: too many requests`. No OOM, MCE or hung task in the kernel
+log. All three nodes place etcd's WAL (`/data/k3s`) and every Longhorn replica (`/data/longhorn`) on
+the SAME partition; node 1's was the 2011 C400 with 61,440 reallocated sectors and ~31% of its
+endurance consumed, and its `/proc/pressure/io` `full avg300` ran **11.36** against ~1% steady state
+on the other two. The failure is self-reinforcing: the node dropping out degrades Longhorn volumes,
+whose rebuilds add write load to the very disk etcd needs.
+
+The operator replaced the drive with a **Samsung SSD 850 EVO 500GB** and reinstalled Debian.
+Measured under the same load class — an active 18-replica Longhorn rebuild, with node 1 as the
+*write target*, i.e. the heavier side:
+
+| node | drive | `full avg300` during rebuild |
+|---|---|---|
+| `k3s-prod-1` | Samsung 850 EVO (new) | **3.79** |
+| `k3s-prod-2` | Samsung PM9A1 NVMe | 0.33 |
+| `k3s-prod-3` | WDC WDS500G2B0A | 2.47 |
+| *`k3s-prod-1` before* | *C400-MTFDDAC512MAM* | ***11.36*** |
+
+Roughly a **3× improvement**, moving node 1 from ~4× worse than its peers into the same band as
+node 3, and it stayed `Ready` throughout. **The 850 EVO is not new**: 77,047 power-on hours over 219
+power cycles — it evidently idled in an always-on system — but its wear tells the opposite story to
+the C400's (0 reallocated vs 61,440; wear-leveling 106 vs 949).
+
+**§2's exit condition is NOT met by this replacement.** etcd and Longhorn still share one SATA
+spindle. The contention is reduced, not eliminated; giving etcd its own device remains the fix, and
+node 1 still has exactly one disk. The three nodes also remain asymmetric — now 850 EVO / PM9A1
+NVMe / WD Blue.
+
+The rebuild itself surfaced two things worth carrying forward. Longhorn refused the new disk with
+`DiskFilesystemChanged` — "record diskUUID doesn't match the one on the disk" — because its node CR
+still held the old drive's `diskUUID`; the remedy is disable → remove the disk entry → re-add, in
+that order (the validating webhook enforces it). And the Ansible inventory still listed node 1 in
+`[k3s_prod_primary]`, which decides both which play gets `--cluster-init` *and* what `k3s_api_host`
+resolves to — so re-provisioning it unchanged would have pointed a rebuilt, empty node at itself and
+founded a second cluster beside the live one. The founder now sits on node 2.
 (2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
 usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
 too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to