docs(runbooks): node-memory-swap — replacing k3s-prod-1's mismatched DIMM pair

The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that
needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3
a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB
but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched
Crucial pair.

The procedure exists because this node is the awkward one to take down: it held
24 Longhorn replicas, it is one of three etcd members (so the drain leaves
quorum at 2 of 3 with no further margin), and its rebuilds are the load that
starved etcd once already. So the runbook front-loads the checks that decide
whether to start at all, and treats a BLOCKED drain as the safety property
working rather than an obstacle to force past.

Two things it deliberately does not assert:

- node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the
  runbook has the operator READ the live Setting and set it before draining if
  it is not block-if-contains-last-replica, instead of assuming the default.
- Configured Memory Speed, not Speed, is named as the field that proves the
  swap worked — a matched pair being clocked down still reports the rated Speed.

It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes
symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the
asymmetric one on memory as well as disk.

Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so
they can prove "two different modules" but cannot distinguish one module from
two identical ones — verification therefore collects dmidecode live. And node 2's
inventory is stale and self-contradictory (documented as the only single-DIMM
node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone
claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was
an EFI memattr kernel bug on node 2.

Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap
rather than existing only in git.

Co-Authored-By: Claude Opus 5 <[email protected]>

runbooks/kustomization.yaml

@@ -80,6 +80,7 @@ configMapGenerator:
       - node-down=node-down.md
       - node-io=node-io.md
       - node-memory=node-memory.md
+      - node-memory-swap=node-memory-swap.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md

runbooks/node-memory-swap.md

@@ -0,0 +1,184 @@
+# node-memory-swap — replacing a node's DIMM pair (planned maintenance)
+
+**Not alert-driven.** This is a planned hardware procedure. If a node is *reporting* a memory problem,
+start at `/runbooks/node-memory` instead.
+
+Written for the one known mismatch: **`k3s-prod-1` (192.168.2.10)**.
+
+## Why node 1, and what "mismatched" means here
+
+| slot | part number | size | rated speed | organisation |
+|---|---|---|---|---|
+| `DIMM1` | Micron **8**ATF2G64HZ-**3G2**E1 | 16 GB | DDR4-**3200** | 1Rx8 |
+| `DIMM3` | Micron **16**ATF2G64HZ-**2G6**E1 | 16 GB | DDR4-**2667** | 2Rx8 |
+
+Micron's part numbering carries the proof independently of any inventory doc: the `8ATF` / `16ATF`
+prefix encodes rank organisation, the `-3G2` / `-2G6` suffix encodes 3200 vs 2666 MT/s. Capacity is
+correct — 32 GB, ~31.1 GiB usable — so **nothing is faulty**. The pair simply runs at the slower 2667
+and the ranks differ. `k3s-prod-3` is a matched Crucial pair (`CT16G4SFD8266.C16FE`).
+
+Source: `infra/ansible/_artifacts/node-health/k3s-prod-1.json` records both part numbers; node 3's
+records one. **Caveat on the artifacts:** the extraction de-duplicates part numbers, so it can prove
+"two different modules" but cannot distinguish one module from two identical ones. Only `dmidecode`
+per-slot output settles that, and no raw per-slot block is recorded — which is why verification below
+collects it live.
+
+**Node 2's inventory is stale and contradicts itself** (`infra/PROD-PROVISIONING.md` calls it the only
+single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That
+does not change this procedure, but re-inventory node 2 before claiming three-way symmetry.
+
+**DO NOT confuse this with §0b.** The provisioning doc says, correctly, *"do not buy RAM"* — that is
+about node 2's old 2.6-GiB-of-16-GB symptom, which was an **EFI memattr kernel bug, not hardware**.
+Making node 1's pair matched is a different and legitimate change.
+
+## Before you touch anything
+
+*If `kubectl` hangs from the workstation rather than failing, the OIDC context is the reason — prefix
+these with `infra/devcontainer/dev.sh run`.*
+
+**1. Decide what capacity you are installing, because it is a symmetry decision.**
+
+- **2 × 16 GB** keeps all three nodes at 32 GB, which is what `infra/PROD-PROVISIONING.md` §2 assumes
+  (symmetric nodes; replica placement and the scheduler both lean on it).
+- **2 × 32 GB** gives node 1 **64 GB** while nodes 2 and 3 stay at 32 GB — a *new* asymmetry, on the
+  node that is already the asymmetric one on disk. Workloads will schedule there preferentially.
+  Allowed, but make it a decision rather than a side effect.
+
+**2. Confirm the cluster can lose this node.** All three nodes are control-plane **and** etcd members,
+so losing one leaves quorum at 2 of 3 — survivable, but with **zero further margin**. Do not start if
+another node is unhealthy.
+
+    kubectl --context k3s-prod get nodes -o wide
+    # etcd's own view, per node (the house check — embedded etcd metrics on 2381):
+    for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \
+      'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done
+
+Every node must report `etcd_server_has_leader 1`. Note which one reports `etcd_server_is_leader 1`:
+if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new
+leader afterwards rather than be surprised by it.
+
+**3. Confirm Longhorn is fully healthy first.** Node 1 held **24 replicas** when last measured. Every
+volume must be healthy *before* you drain, or you are draining into an already-degraded state.
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state
+
+Expect every attached volume `healthy` (a `detached` volume reporting `unknown` is normal for an unused
+PVC). **If anything reads `degraded`, stop and let it rebuild.**
+
+**4. Confirm the drain policy is what protects you**, rather than assuming it — it is not pinned in
+`platform/longhorn/helmrelease.yaml`, so it is whatever the chart defaults to on this version:
+
+    kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}'
+
+You want `block-if-contains-last-replica`. If it reads something more permissive, **set it before
+draining** — this is the setting that stops the drain from stranding a volume whose only healthy
+replica is on node 1.
+
+**5. Take a fresh backup.** There is player-authored state here now (the Minecraft world on
+`crafty-data`). Find the schedule's real resource name rather than guessing it:
+
+    kubectl --context k3s-prod -n velero get schedules
+    kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \
+      --from-schedule <the daily schedule from above>
+
+Wait for `Completed` with `errors: 0` before continuing.
+
+## Drain
+
+    kubectl --context k3s-prod cordon k3s-prod-1
+    kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m
+
+**If the drain blocks, that is the policy doing its job.** Do not reach for `--force` or
+`--disable-eviction`. Watch replicas rebuild onto the other two nodes and let the drain proceed:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy
+
+Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is
+deliberately throttled (see `/runbooks/node-io` for which limits are set and which do not exist on
+this Longhorn version). Watch the pressure it generates while it runs:
+
+    rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m])
+
+The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
+starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
+
+Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will
+reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice
+today, so expect it to be recreated, which also re-runs its cloud-init.
+
+## Power off and swap
+
+    sudo systemctl poweroff   # from the node itself, or via the console
+
+HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. **HP's supported population requires `DIMM1` to be
+populated** — fill `DIMM1` first, then `DIMM3`. Seat both fully; a partially seated SODIMM in these
+machines will POST with one module silently missing.
+
+## Post-swap verification
+
+In order. **Do not uncordon until 1–3 pass.**
+
+**1. The pair is actually matched** — the whole point of the exercise:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1   # read-only
+
+or directly on the node:
+
+    sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank'
+
+Both slots must report the **same Part Number**, the **same Rank**, and — the part that proves the
+mismatch is gone — the same `Speed` *and* the same `Configured Memory Speed`. **`Configured Memory
+Speed` is the field that matters:** a pair rated 3200 that is being clocked down still reports
+`Speed: 3200 MT/s` while `Configured Memory Speed` sits lower. Matched part numbers with a
+below-rated configured speed means the job is not done.
+
+**2. Capacity is what you installed.**
+
+    kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}'
+
+Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you
+installed, a module is not seated or not detected — power off and reseat rather than debugging it in
+software, because the scheduler will believe whatever capacity is reported.
+
+**3. No new memory errors.** These machines are non-ECC, so this is a sanity check, not a guarantee:
+
+    sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error'
+
+**4. Rejoin the cluster.**
+
+    kubectl --context k3s-prod uncordon k3s-prod-1
+    kubectl --context k3s-prod get nodes
+    ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
+
+`Ready` on the node and `etcd_server_has_leader 1` from node 1's own etcd before you call this done.
+
+**5. Longhorn returns to fully healthy.** Rebuilds run back onto node 1, and this is the phase that
+historically produced I/O pressure on this node — watch it rather than walking away:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy
+
+with the same PSI query from the drain section. `/runbooks/node-io` records what the post-SSD-swap
+18-replica rebuild measured, for comparison.
+
+**6. Alerts quiet.** The only thing that should be firing is `Watchdog`:
+
+    kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+      -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'
+
+## If it goes wrong
+
+- **Node does not POST**: reseat both modules; failing that, put the original pair back. The old DIMMs
+  are known-good — the mismatch was a consistency/performance issue, not a fault.
+- **Node POSTs with half the memory**: one module not detected. Power off and reseat.
+- **Node rejoins but volumes stay degraded**: `/runbooks/longhorn-degraded`. Do **not** drain a second
+  node while rebuilds are outstanding — that loses quorum and replicas at the same time.
+- **Abort mid-drain**: `kubectl --context k3s-prod uncordon k3s-prod-1` is safe any time before poweroff.
+
+## Update the record when you are done
+
+`infra/PROD-PROVISIONING.md` carries the DIMM inventory table and the "not a matched pair" paragraph.
+Both become wrong the moment this succeeds. Update them in the same change that records the swap —
+stale hardware documentation is exactly what made node 2's inventory untrustworthy.