fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1

They do not, and the correction changes the operator's plan rather than just a
detail.

Verified live: all three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2,
nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces
no failover at all — the best state this maintenance could ask for. The original
sentence was carried over from earlier session state and was simply stale; the
runbook now prints the command to re-derive it, because primaries move on their
own after any restart or failover.

Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing
for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The
instinct is to tidy that before a maintenance window. Doing so would be actively
harmful here: rebalancing means promoting a replica, the only node with spare
primary capacity is node 1, and so "fixing" the alert would put a primary on the
very node about to be powered off — converting a no-failover drain into a
two-database failover. Rebalance after node 1 is back and caught up.

Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-19 00:01:19 -05:00
1 parent 05a8094785
commit cde02d19d2
1 file changed
+20 -3
+20 -3
View File
@@ -104,9 +104,26 @@ this Longhorn version). Watch the pressure it generates while it runs:
The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will
reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice
today, so expect it to be recreated, which also re-runs its cloud-init.
Note what moves, and one thing that happily does not. Verified live 2026-09-19:
kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
-o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role
**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
this maintenance could ask for. Re-run the command above before you start rather than trusting this
paragraph: primaries move on their own after any restart or failover.
`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
the very node you are about to power off and convert a no-failover drain into a two-database failover.
Rebalance *after* node 1 is back and its replicas have caught up.
`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
recreated, which also re-runs its cloud-init.
## Power off and swap