home-network-k3s-impl/runbooks/node-memory-swap.md

node-memory-swap — replacing a node's DIMM pair (planned maintenance)

Not alert-driven. This is a planned hardware procedure. If a node is reporting a memory problem, start at /runbooks/node-memory instead.

Written for the one known mismatch: k3s-prod-1 (192.168.2.10).

Why node 1, and what "mismatched" means here

slot	part number	size	rated speed	organisation
DIMM1	Micron 8ATF2G64HZ-3G2E1	16 GB	DDR4-3200	1Rx8
DIMM3	Micron 16ATF2G64HZ-2G6E1	16 GB	DDR4-2667	2Rx8

Micron's part numbering carries the proof independently of any inventory doc: the 8ATF / 16ATF prefix encodes rank organisation, the -3G2 / -2G6 suffix encodes 3200 vs 2666 MT/s. Capacity is correct — 32 GB, ~31.1 GiB usable — so nothing is faulty. The pair simply runs at the slower 2667 and the ranks differ. k3s-prod-3 is a matched Crucial pair (CT16G4SFD8266.C16FE).

Source: infra/ansible/_artifacts/node-health/k3s-prod-1.json records both part numbers; node 3's records one. Caveat on the artifacts: the extraction de-duplicates part numbers, so it can prove "two different modules" but cannot distinguish one module from two identical ones. Only dmidecode per-slot output settles that, and no raw per-slot block is recorded — which is why verification below collects it live.

Node 2's inventory is stale and contradicts itself (infra/PROD-PROVISIONING.md calls it the only single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That does not change this procedure, but re-inventory node 2 before claiming three-way symmetry.

DO NOT confuse this with §0b. The provisioning doc says, correctly, "do not buy RAM" — that is about node 2's old 2.6-GiB-of-16-GB symptom, which was an EFI memattr kernel bug, not hardware. Making node 1's pair matched is a different and legitimate change.

Before you touch anything

If kubectl hangs from the workstation rather than failing, the OIDC context is the reason — prefix these with infra/devcontainer/dev.sh run.

1. Decide what capacity you are installing, because it is a symmetry decision.

• 2 × 16 GB keeps all three nodes at 32 GB, which is what infra/PROD-PROVISIONING.md §2 assumes (symmetric nodes; replica placement and the scheduler both lean on it).
• 2 × 32 GB gives node 1 64 GB while nodes 2 and 3 stay at 32 GB — a new asymmetry, on the node that is already the asymmetric one on disk. Workloads will schedule there preferentially. Allowed, but make it a decision rather than a side effect.

2. Confirm the cluster can lose this node. All three nodes are control-plane and etcd members, so losing one leaves quorum at 2 of 3 — survivable, but with zero further margin. Do not start if another node is unhealthy.

kubectl --context k3s-prod get nodes -o wide
# etcd's own view, per node (the house check — embedded etcd metrics on 2381):
for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \
  'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done

Every node must report etcd_server_has_leader 1. Note which one reports etcd_server_is_leader 1: if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new leader afterwards rather than be surprised by it.

3. Confirm Longhorn is fully healthy first. Node 1 held 24 replicas when last measured. Every volume must be healthy before you drain, or you are draining into an already-degraded state.

kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state

Expect every attached volume healthy (a detached volume reporting unknown is normal for an unused PVC). If anything reads degraded, stop and let it rebuild.

4. Confirm the drain policy is what protects you, rather than assuming it — it is not pinned in platform/longhorn/helmrelease.yaml, so it is whatever the chart defaults to on this version:

kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}'

You want block-if-contains-last-replica. If it reads something more permissive, set it before draining — this is the setting that stops the drain from stranding a volume whose only healthy replica is on node 1.

5. Take a fresh backup. There is player-authored state here now (the Minecraft world on crafty-data). Find the schedule's real resource name rather than guessing it:

kubectl --context k3s-prod -n velero get schedules
kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \
  --from-schedule <the daily schedule from above>

Wait for Completed with errors: 0 before continuing.

Drain

kubectl --context k3s-prod cordon k3s-prod-1
kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m

If the drain blocks, that is the policy doing its job. Do not reach for --force or --disable-eviction. Watch replicas rebuild onto the other two nodes and let the drain proceed:

kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy

Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is deliberately throttled (see /runbooks/node-io for which limits are set and which do not exist on this Longhorn version). Watch the pressure it generates while it runs:

rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m])

The warning threshold is 0.05 and critical is 0.10; node 1 sat at 0.10 during the incident that starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.

Note what moves, and one thing that happily does not. Verified live 2026-09-19:

kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role

No PostgreSQL primary is on node 1. All three CNPG clusters run 3 instances spread one per node by required anti-affinity, and node 1 holds a replica of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). Draining it evicts three replicas and forces no failover at all — the best state this maintenance could ask for. Re-run the command above before you start rather than trusting this paragraph: primaries move on their own after any restart or failover.

CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. Do not "fix" that before the swap. Rebalancing means promoting a replica, and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on the very node you are about to power off and convert a no-failover drain into a two-database failover. Rebalance after node 1 is back and its replicas have caught up.

sonarr, qbittorrent and jackett do sit on node 1 and will reschedule. The UniFi VM (virt-launcher-unifi-*) is there too — it is not live-migratable in practice today, so expect it to be recreated, which also re-runs its cloud-init.

Power off and swap

sudo systemctl poweroff   # from the node itself, or via the console

HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. HP's supported population requires DIMM1 to be populated — fill DIMM1 first, then DIMM3. Seat both fully; a partially seated SODIMM in these machines will POST with one module silently missing.

Post-swap verification

In order. Do not uncordon until 1–3 pass.

1. The pair is actually matched — the whole point of the exercise:

ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1   # read-only

or directly on the node:

sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank'

Both slots must report the same Part Number, the same Rank, and — the part that proves the mismatch is gone — the same Speed and the same Configured Memory Speed. Configured Memory Speed is the field that matters: a pair rated 3200 that is being clocked down still reports Speed: 3200 MT/s while Configured Memory Speed sits lower. Matched part numbers with a below-rated configured speed means the job is not done.

2. Capacity is what you installed.

kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}'

Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you installed, a module is not seated or not detected — power off and reseat rather than debugging it in software, because the scheduler will believe whatever capacity is reported.

3. No new memory errors. These machines are non-ECC, so this is a sanity check, not a guarantee:

sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error'

4. Rejoin the cluster.

kubectl --context k3s-prod uncordon k3s-prod-1
kubectl --context k3s-prod get nodes
ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'

Ready on the node and etcd_server_has_leader 1 from node 1's own etcd before you call this done.

5. Longhorn returns to fully healthy. Rebuilds run back onto node 1, and this is the phase that historically produced I/O pressure on this node — watch it rather than walking away:

kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy

with the same PSI query from the drain section. /runbooks/node-io records what the post-SSD-swap 18-replica rebuild measured, for comparison.

6. Alerts quiet. The only thing that should be firing is Watchdog:

kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
  -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'

If it goes wrong

• Node does not POST: reseat both modules; failing that, put the original pair back. The old DIMMs are known-good — the mismatch was a consistency/performance issue, not a fault.
• Node POSTs with half the memory: one module not detected. Power off and reseat.
• Node rejoins but volumes stay degraded: /runbooks/longhorn-degraded. Do not drain a second node while rebuilds are outstanding — that loses quorum and replicas at the same time.
• Abort mid-drain: kubectl --context k3s-prod uncordon k3s-prod-1 is safe any time before poweroff.

Update the record when you are done

infra/PROD-PROVISIONING.md carries the DIMM inventory table and the "not a matched pair" paragraph. Both become wrong the moment this succeeds. Update them in the same change that records the swap — stale hardware documentation is exactly what made node 2's inventory untrustworthy.