home-network-k3s-impl/runbooks/node-memory-swap.md

# node-memory-swap — replacing a node's DIMM pair (planned maintenance)

**Not alert-driven.** This is a planned hardware procedure. If a node is *reporting* a memory problem,
start at `/runbooks/node-memory` instead.

Written for the one known mismatch: **`k3s-prod-1` (192.168.2.10)**.

## Why node 1, and what "mismatched" means here

| slot | part number | size | rated speed | organisation |
|---|---|---|---|---|
| `DIMM1` | Micron **8**ATF2G64HZ-**3G2**E1 | 16 GB | DDR4-**3200** | 1Rx8 |
| `DIMM3` | Micron **16**ATF2G64HZ-**2G6**E1 | 16 GB | DDR4-**2667** | 2Rx8 |

Micron's part numbering carries the proof independently of any inventory doc: the `8ATF` / `16ATF`
prefix encodes rank organisation, the `-3G2` / `-2G6` suffix encodes 3200 vs 2666 MT/s. Capacity is
correct — 32 GB, ~31.1 GiB usable — so **nothing is faulty**. The pair simply runs at the slower 2667
and the ranks differ. `k3s-prod-3` is a matched Crucial pair (`CT16G4SFD8266.C16FE`).

Source: `infra/ansible/_artifacts/node-health/k3s-prod-1.json` records both part numbers; node 3's
records one. **Caveat on the artifacts:** the extraction de-duplicates part numbers, so it can prove
"two different modules" but cannot distinguish one module from two identical ones. Only `dmidecode`
per-slot output settles that, and no raw per-slot block is recorded — which is why verification below
collects it live.

**Node 2's inventory is stale and contradicts itself** (`infra/PROD-PROVISIONING.md` calls it the only
single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That
does not change this procedure, but re-inventory node 2 before claiming three-way symmetry.

**DO NOT confuse this with §0b.** The provisioning doc says, correctly, *"do not buy RAM"* — that is
about node 2's old 2.6-GiB-of-16-GB symptom, which was an **EFI memattr kernel bug, not hardware**.
Making node 1's pair matched is a different and legitimate change.

## Before you touch anything

*If `kubectl` hangs from the workstation rather than failing, the OIDC context is the reason — prefix
these with `infra/devcontainer/dev.sh run`.*

**1. Decide what capacity you are installing, because it is a symmetry decision.**

- **2 × 16 GB** keeps all three nodes at 32 GB, which is what `infra/PROD-PROVISIONING.md` §2 assumes
  (symmetric nodes; replica placement and the scheduler both lean on it).
- **2 × 32 GB** gives node 1 **64 GB** while nodes 2 and 3 stay at 32 GB — a *new* asymmetry, on the
  node that is already the asymmetric one on disk. Workloads will schedule there preferentially.
  Allowed, but make it a decision rather than a side effect.

**2. Confirm the cluster can lose this node.** All three nodes are control-plane **and** etcd members,
so losing one leaves quorum at 2 of 3 — survivable, but with **zero further margin**. Do not start if
another node is unhealthy.

    kubectl --context k3s-prod get nodes -o wide
    # etcd's own view, per node (the house check — embedded etcd metrics on 2381):
    for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \
      'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done

Every node must report `etcd_server_has_leader 1`. Note which one reports `etcd_server_is_leader 1`:
if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new
leader afterwards rather than be surprised by it.

**3. Confirm Longhorn is fully healthy first.** Node 1 held **24 replicas** when last measured. Every
volume must be healthy *before* you drain, or you are draining into an already-degraded state.

    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state

Expect every attached volume `healthy` (a `detached` volume reporting `unknown` is normal for an unused
PVC). **If anything reads `degraded`, stop and let it rebuild.**

**4. Confirm the drain policy is what protects you**, rather than assuming it — it is not pinned in
`platform/longhorn/helmrelease.yaml`, so it is whatever the chart defaults to on this version:

    kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}'

You want `block-if-contains-last-replica`. If it reads something more permissive, **set it before
draining** — this is the setting that stops the drain from stranding a volume whose only healthy
replica is on node 1.

**5. Take a fresh backup.** There is player-authored state here now (the Minecraft world on
`crafty-data`). Find the schedule's real resource name rather than guessing it:

    kubectl --context k3s-prod -n velero get schedules
    kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \
      --from-schedule <the daily schedule from above>

Wait for `Completed` with `errors: 0` before continuing.

## Drain

    kubectl --context k3s-prod cordon k3s-prod-1
    kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m

**If the drain blocks, that is the policy doing its job.** Do not reach for `--force` or
`--disable-eviction`. Watch replicas rebuild onto the other two nodes and let the drain proceed:

    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy

Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is
deliberately throttled (see `/runbooks/node-io` for which limits are set and which do not exist on
this Longhorn version). Watch the pressure it generates while it runs:

    rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m])

The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.

Note what moves, and one thing that happily does not. Verified live 2026-09-19:

    kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
      -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role

**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
this maintenance could ask for. Re-run the command above before you start rather than trusting this
paragraph: primaries move on their own after any restart or failover.

`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
the very node you are about to power off and convert a no-failover drain into a two-database failover.
Rebalance *after* node 1 is back and its replicas have caught up.

`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
recreated, which also re-runs its cloud-init.

## Power off and swap

    sudo systemctl poweroff   # from the node itself, or via the console

HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. **HP's supported population requires `DIMM1` to be
populated** — fill `DIMM1` first, then `DIMM3`. Seat both fully; a partially seated SODIMM in these
machines will POST with one module silently missing.

## Post-swap verification

In order. **Do not uncordon until 1–3 pass.**

**1. The pair is actually matched** — the whole point of the exercise:

    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1   # read-only

or directly on the node:

    sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank'

Both slots must report the **same Part Number**, the **same Rank**, and — the part that proves the
mismatch is gone — the same `Speed` *and* the same `Configured Memory Speed`. **`Configured Memory
Speed` is the field that matters:** a pair rated 3200 that is being clocked down still reports
`Speed: 3200 MT/s` while `Configured Memory Speed` sits lower. Matched part numbers with a
below-rated configured speed means the job is not done.

**2. Capacity is what you installed.**

    kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}'

Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you
installed, a module is not seated or not detected — power off and reseat rather than debugging it in
software, because the scheduler will believe whatever capacity is reported.

**3. No new memory errors.** These machines are non-ECC, so this is a sanity check, not a guarantee:

    sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error'

**4. Rejoin the cluster.**

    kubectl --context k3s-prod uncordon k3s-prod-1
    kubectl --context k3s-prod get nodes
    ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'

`Ready` on the node and `etcd_server_has_leader 1` from node 1's own etcd before you call this done.

**5. Longhorn returns to fully healthy.** Rebuilds run back onto node 1, and this is the phase that
historically produced I/O pressure on this node — watch it rather than walking away:

    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy

with the same PSI query from the drain section. `/runbooks/node-io` records what the post-SSD-swap
18-replica rebuild measured, for comparison.

**6. Alerts quiet.** The only thing that should be firing is `Watchdog`:

    kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
      -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'

## If it goes wrong

- **Node does not POST**: reseat both modules; failing that, put the original pair back. The old DIMMs
  are known-good — the mismatch was a consistency/performance issue, not a fault.
- **Node POSTs with half the memory**: one module not detected. Power off and reseat.
- **Node rejoins but volumes stay degraded**: `/runbooks/longhorn-degraded`. Do **not** drain a second
  node while rebuilds are outstanding — that loses quorum and replicas at the same time.
- **Abort mid-drain**: `kubectl --context k3s-prod uncordon k3s-prod-1` is safe any time before poweroff.

## Update the record when you are done

`infra/PROD-PROVISIONING.md` carries the DIMM inventory table and the "not a matched pair" paragraph.
Both become wrong the moment this succeeds. Update them in the same change that records the swap —
stale hardware documentation is exactly what made node 2's inventory untrustworthy.