diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 2682e79..6c1c1d8 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -80,6 +80,7 @@ configMapGenerator: - node-down=node-down.md - node-io=node-io.md - node-memory=node-memory.md + - node-memory-swap=node-memory-swap.md - pvc-capacity=pvc-capacity.md - targets-down=targets-down.md - update-rollback=update-rollback.md diff --git a/runbooks/node-memory-swap.md b/runbooks/node-memory-swap.md new file mode 100644 index 0000000..3e5c6be --- /dev/null +++ b/runbooks/node-memory-swap.md @@ -0,0 +1,184 @@ +# node-memory-swap — replacing a node's DIMM pair (planned maintenance) + +**Not alert-driven.** This is a planned hardware procedure. If a node is *reporting* a memory problem, +start at `/runbooks/node-memory` instead. + +Written for the one known mismatch: **`k3s-prod-1` (192.168.2.10)**. + +## Why node 1, and what "mismatched" means here + +| slot | part number | size | rated speed | organisation | +|---|---|---|---|---| +| `DIMM1` | Micron **8**ATF2G64HZ-**3G2**E1 | 16 GB | DDR4-**3200** | 1Rx8 | +| `DIMM3` | Micron **16**ATF2G64HZ-**2G6**E1 | 16 GB | DDR4-**2667** | 2Rx8 | + +Micron's part numbering carries the proof independently of any inventory doc: the `8ATF` / `16ATF` +prefix encodes rank organisation, the `-3G2` / `-2G6` suffix encodes 3200 vs 2666 MT/s. Capacity is +correct — 32 GB, ~31.1 GiB usable — so **nothing is faulty**. The pair simply runs at the slower 2667 +and the ranks differ. `k3s-prod-3` is a matched Crucial pair (`CT16G4SFD8266.C16FE`). + +Source: `infra/ansible/_artifacts/node-health/k3s-prod-1.json` records both part numbers; node 3's +records one. **Caveat on the artifacts:** the extraction de-duplicates part numbers, so it can prove +"two different modules" but cannot distinguish one module from two identical ones. Only `dmidecode` +per-slot output settles that, and no raw per-slot block is recorded — which is why verification below +collects it live. + +**Node 2's inventory is stale and contradicts itself** (`infra/PROD-PROVISIONING.md` calls it the only +single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That +does not change this procedure, but re-inventory node 2 before claiming three-way symmetry. + +**DO NOT confuse this with §0b.** The provisioning doc says, correctly, *"do not buy RAM"* — that is +about node 2's old 2.6-GiB-of-16-GB symptom, which was an **EFI memattr kernel bug, not hardware**. +Making node 1's pair matched is a different and legitimate change. + +## Before you touch anything + +*If `kubectl` hangs from the workstation rather than failing, the OIDC context is the reason — prefix +these with `infra/devcontainer/dev.sh run`.* + +**1. Decide what capacity you are installing, because it is a symmetry decision.** + +- **2 × 16 GB** keeps all three nodes at 32 GB, which is what `infra/PROD-PROVISIONING.md` §2 assumes + (symmetric nodes; replica placement and the scheduler both lean on it). +- **2 × 32 GB** gives node 1 **64 GB** while nodes 2 and 3 stay at 32 GB — a *new* asymmetry, on the + node that is already the asymmetric one on disk. Workloads will schedule there preferentially. + Allowed, but make it a decision rather than a side effect. + +**2. Confirm the cluster can lose this node.** All three nodes are control-plane **and** etcd members, +so losing one leaves quorum at 2 of 3 — survivable, but with **zero further margin**. Do not start if +another node is unhealthy. + + kubectl --context k3s-prod get nodes -o wide + # etcd's own view, per node (the house check — embedded etcd metrics on 2381): + for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \ + 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done + +Every node must report `etcd_server_has_leader 1`. Note which one reports `etcd_server_is_leader 1`: +if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new +leader afterwards rather than be surprised by it. + +**3. Confirm Longhorn is fully healthy first.** Node 1 held **24 replicas** when last measured. Every +volume must be healthy *before* you drain, or you are draining into an already-degraded state. + + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state + +Expect every attached volume `healthy` (a `detached` volume reporting `unknown` is normal for an unused +PVC). **If anything reads `degraded`, stop and let it rebuild.** + +**4. Confirm the drain policy is what protects you**, rather than assuming it — it is not pinned in +`platform/longhorn/helmrelease.yaml`, so it is whatever the chart defaults to on this version: + + kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}' + +You want `block-if-contains-last-replica`. If it reads something more permissive, **set it before +draining** — this is the setting that stops the drain from stranding a volume whose only healthy +replica is on node 1. + +**5. Take a fresh backup.** There is player-authored state here now (the Minecraft world on +`crafty-data`). Find the schedule's real resource name rather than guessing it: + + kubectl --context k3s-prod -n velero get schedules + kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \ + --from-schedule + +Wait for `Completed` with `errors: 0` before continuing. + +## Drain + + kubectl --context k3s-prod cordon k3s-prod-1 + kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m + +**If the drain blocks, that is the policy doing its job.** Do not reach for `--force` or +`--disable-eviction`. Watch replicas rebuild onto the other two nodes and let the drain proceed: + + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy + +Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is +deliberately throttled (see `/runbooks/node-io` for which limits are set and which do not exist on +this Longhorn version). Watch the pressure it generates while it runs: + + rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m]) + +The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that +starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not. + +Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will +reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice +today, so expect it to be recreated, which also re-runs its cloud-init. + +## Power off and swap + + sudo systemctl poweroff # from the node itself, or via the console + +HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. **HP's supported population requires `DIMM1` to be +populated** — fill `DIMM1` first, then `DIMM3`. Seat both fully; a partially seated SODIMM in these +machines will POST with one module silently missing. + +## Post-swap verification + +In order. **Do not uncordon until 1–3 pass.** + +**1. The pair is actually matched** — the whole point of the exercise: + + ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1 # read-only + +or directly on the node: + + sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank' + +Both slots must report the **same Part Number**, the **same Rank**, and — the part that proves the +mismatch is gone — the same `Speed` *and* the same `Configured Memory Speed`. **`Configured Memory +Speed` is the field that matters:** a pair rated 3200 that is being clocked down still reports +`Speed: 3200 MT/s` while `Configured Memory Speed` sits lower. Matched part numbers with a +below-rated configured speed means the job is not done. + +**2. Capacity is what you installed.** + + kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}' + +Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you +installed, a module is not seated or not detected — power off and reseat rather than debugging it in +software, because the scheduler will believe whatever capacity is reported. + +**3. No new memory errors.** These machines are non-ECC, so this is a sanity check, not a guarantee: + + sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error' + +**4. Rejoin the cluster.** + + kubectl --context k3s-prod uncordon k3s-prod-1 + kubectl --context k3s-prod get nodes + ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"' + +`Ready` on the node and `etcd_server_has_leader 1` from node 1's own etcd before you call this done. + +**5. Longhorn returns to fully healthy.** Rebuilds run back onto node 1, and this is the phase that +historically produced I/O pressure on this node — watch it rather than walking away: + + kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy + +with the same PSI query from the drain section. `/runbooks/node-io` records what the post-SSD-swap +18-replica rebuild measured, for comparison. + +**6. Alerts quiet.** The only thing that should be firing is `Watchdog`: + + kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \ + -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}' + +## If it goes wrong + +- **Node does not POST**: reseat both modules; failing that, put the original pair back. The old DIMMs + are known-good — the mismatch was a consistency/performance issue, not a fault. +- **Node POSTs with half the memory**: one module not detected. Power off and reseat. +- **Node rejoins but volumes stay degraded**: `/runbooks/longhorn-degraded`. Do **not** drain a second + node while rebuilds are outstanding — that loses quorum and replicas at the same time. +- **Abort mid-drain**: `kubectl --context k3s-prod uncordon k3s-prod-1` is safe any time before poweroff. + +## Update the record when you are done + +`infra/PROD-PROVISIONING.md` carries the DIMM inventory table and the "not a matched pair" paragraph. +Both become wrong the moment this succeeds. Update them in the same change that records the swap — +stale hardware documentation is exactly what made node 2's inventory untrustworthy.