fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1
They do not, and the correction changes the operator's plan rather than just a detail. Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces no failover at all — the best state this maintenance could ask for. The original sentence was carried over from earlier session state and was simply stale; the runbook now prints the command to re-derive it, because primaries move on their own after any restart or failover. Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The instinct is to tidy that before a maintenance window. Doing so would be actively harmful here: rebalancing means promoting a replica, the only node with spare primary capacity is node 1, and so "fixing" the alert would put a primary on the very node about to be powered off — converting a no-failover drain into a two-database failover. Rebalance after node 1 is back and caught up. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
05a8094785
commit
cde02d19d2
1 file changed
+20
-3
@@ -104,9 +104,26 @@ this Longhorn version). Watch the pressure it generates while it runs:
|
||||
The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
|
||||
starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
|
||||
|
||||
Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will
|
||||
reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice
|
||||
today, so expect it to be recreated, which also re-runs its cloud-init.
|
||||
Note what moves, and one thing that happily does not. Verified live 2026-09-19:
|
||||
|
||||
kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
|
||||
-o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role
|
||||
|
||||
**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
|
||||
required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
|
||||
`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
|
||||
this maintenance could ask for. Re-run the command above before you start rather than trusting this
|
||||
paragraph: primaries move on their own after any restart or failover.
|
||||
|
||||
`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
|
||||
`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
|
||||
and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
|
||||
the very node you are about to power off and convert a no-failover drain into a two-database failover.
|
||||
Rebalance *after* node 1 is back and its replicas have caught up.
|
||||
|
||||
`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
|
||||
(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
|
||||
recreated, which also re-runs its cloud-init.
|
||||
|
||||
## Power off and swap
|
||||
|
||||
|
||||
Reference in new issue
Block a user