From cde02d19d29762788b868838a10613854b339452 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 19 Sep 2026 00:01:19 -0500 Subject: [PATCH] fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit They do not, and the correction changes the operator's plan rather than just a detail. Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces no failover at all — the best state this maintenance could ask for. The original sentence was carried over from earlier session state and was simply stale; the runbook now prints the command to re-derive it, because primaries move on their own after any restart or failover. Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The instinct is to tidy that before a maintenance window. Doing so would be actively harmful here: rebalancing means promoting a replica, the only node with spare primary capacity is node 1, and so "fixing" the alert would put a primary on the very node about to be powered off — converting a no-failover drain into a two-database failover. Rebalance after node 1 is back and caught up. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 --- runbooks/node-memory-swap.md | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/runbooks/node-memory-swap.md b/runbooks/node-memory-swap.md index 3e5c6be..3783668 100644 --- a/runbooks/node-memory-swap.md +++ b/runbooks/node-memory-swap.md @@ -104,9 +104,26 @@ this Longhorn version). Watch the pressure it generates while it runs: The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not. -Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will -reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice -today, so expect it to be recreated, which also re-runs its cloud-init. +Note what moves, and one thing that happily does not. Verified live 2026-09-19: + + kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \ + -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role + +**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by +required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`, +`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state +this maintenance could ask for. Re-run the command above before you start rather than trusting this +paragraph: primaries move on their own after any restart or failover. + +`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and +`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica, +and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on +the very node you are about to power off and convert a no-failover drain into a two-database failover. +Rebalance *after* node 1 is back and its replicas have caught up. + +`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM +(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be +recreated, which also re-runs its cloud-init. ## Power off and swap