# cnpg-primary-spread — Two or more CNPG primaries share one node **Alert:** `CNPGPrimariesCoLocated` (warning, 30m) **File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` **Expression:** `count by (node) () > 1` **and** gated on every node being Ready: `count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)` ## What this means here More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`, `nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is one primary per node, and this alert says that shape has drifted. **This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node (`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`, `dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a `warning` with a long `for`. **Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run `instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage — but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same moment as two others, on nodes that are also absorbing Longhorn replica rebuilds. **Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.** The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade. So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it will be reported again rather than silently persisting. **Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node returns. This rule was proven in both directions on live prod data (2026-09-13): the expression returned `{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going `pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty vector while the inner join still resolved all three primaries** — quiet because the condition is false, not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml` (run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up` join, the readiness gate, or shortening `for:` makes a test FAIL. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Where the primaries are — the one command this alert is about: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \ -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName' ``` Are all three Clusters actually healthy enough to move one? (Fully-qualified: `clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' ``` Is every node Ready (the gate the rule itself applies)? ``` infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide' ``` Is there storage or I/O trouble that makes a switchover a bad idea right now? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' ``` ``` ssh k3s-prod-1 'cat /proc/pressure/io' ``` And confirm DR is intact before you deliberately move a primary — the primary owns the backup catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the short name resolves to Longhorn's (zero objects): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \ kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' ``` ## Likely causes, most likely first ### 1. A node rebooted and the primaries never moved back The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason to move them back afterwards. Remediation: a deliberate switchover, below. ### 2. A rolling operator or Postgres upgrade `primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on `keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever they landed. ### 3. One node was cordoned or unschedulable for a while Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why the *instances* stay spread even when the *primaries* do not. ### 4. A Longhorn or I/O event caused a health-check failover A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing. ### 5. The alert is firing during a genuine node outage It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue. ## Remediation — move one primary, deliberately The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the operator version (CNPG **1.30.0** here) before you start. ``` kubectl cnpg promote -n ``` **Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable compound one. Pick the target by reading the primaries table above and choosing an instance that already sits on a node holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear on its own. ## When to escalate or stop - **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption, and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent. - **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a cluster that has no standby left to fall back on. - **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict quorum, and one degraded standby would then **block writes** entirely on the SSO database. - **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the thing `primaryUpdateMethod: switchover` exists to avoid. - **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius tidying completely. - Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea. --- Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/` Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.