diff --git a/runbooks/node-memory-swap.md b/runbooks/node-memory-swap.md index 3e5c6be..3783668 100644 --- a/runbooks/node-memory-swap.md +++ b/runbooks/node-memory-swap.md @@ -104,9 +104,26 @@ this Longhorn version). Watch the pressure it generates while it runs: The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not. -Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will -reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice -today, so expect it to be recreated, which also re-runs its cloud-init. +Note what moves, and one thing that happily does not. Verified live 2026-09-19: + + kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \ + -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role + +**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by +required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`, +`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state +this maintenance could ask for. Re-run the command above before you start rather than trusting this +paragraph: primaries move on their own after any restart or failover. + +`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and +`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica, +and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on +the very node you are about to power off and convert a no-failover drain into a two-database failover. +Rebalance *after* node 1 is back and its replicas have caught up. + +`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM +(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be +recreated, which also re-runs its cloud-init. ## Power off and swap