merge: fix/node-memory-swap-cnpg-placement
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

This commit is contained in:
jakman85 committed 2026-09-19 00:01:19 -05:00
commit 6eaaaae334
1 file changed
+20 -3
+20 -3
View File
@@ -104,9 +104,26 @@ this Longhorn version). Watch the pressure it generates while it runs:
The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
Note what moves: the CNPG primaries, `sonarr`, `qbittorrent` and `jackett` all sit on node 1 and will
reschedule. The UniFi VM (`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice
today, so expect it to be recreated, which also re-runs its cloud-init.
Note what moves, and one thing that happily does not. Verified live 2026-09-19:
kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
-o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role
**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
this maintenance could ask for. Re-run the command above before you start rather than trusting this
paragraph: primaries move on their own after any restart or failover.
`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
the very node you are about to power off and convert a no-failover drain into a two-database failover.
Rebalance *after* node 1 is back and its replicas have caught up.
`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
recreated, which also re-runs its cloud-init.
## Power off and swap