home-network-k3s-impl/runbooks/cnpg-primary-spread.md

cnpg-primary-spread — Two or more CNPG primaries share one node

Alert: CNPGPrimariesCoLocated (warning, 30m) File: platform/cnpg/alerts.yaml, group platform-health.cnpg Expression: count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1 and gated on every node being Ready: count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)

What this means here

More than one CloudNativePG primary has been running on the same node for 30 minutes, while all three nodes are Ready. There are exactly three CNPG Clusters on this platform — keycloak/keycloak-db, nextcloud-db/nextcloud-db and semaphore-db/semaphore-db — so on a 3-node cluster the healthy shape is one primary per node, and this alert says that shape has drifted.

This is BLAST RADIUS, not an outage. Each Cluster has instances: 3 with one pod per node (podAntiAffinityType: required) and one synchronous standby (method: any, number: 1, dataDurability: preferred), so every one of them survives losing the shared node — you just get two or three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a warning with a long for.

Why it matters anyway: this exact drift has already caused an outage. All three Clusters used to run instances: 1 and all three primaries had drifted onto one node, and on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together. instances: 3 means the same drift no longer causes an outage — but if keycloak-db's primary is co-located and that node dies, the SSO failover happens at the same moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.

Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field. The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade. So this is avoidable drift that must be corrected, not prevented, and the alert re-arms itself so it will be reported again rather than silently persisting.

Why the node-readiness gate is in the expression. During a real node outage, primaries legitimately pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst moment. The and on () clause suppresses the rule unless every node is Ready, and re-arms when the node returns.

This rule was proven in both directions on live prod data (2026-09-13): the expression returned {node="k3s-prod-1"} => 3 before the rebalance and a throwaway twin with for: 1m was observed going pending at +70s and firing at +130s; after two switchovers the same expression returned an empty vector while the inner join still resolved all three primaries — quiet because the condition is false, not because the query went blind. It is also unit-tested in alerts.promtool-test.yaml (run ./run-alert-tests.sh) with four cases, mutation-checked so that deleting the cnpg_collector_up join, the readiness gate, or shortening for: makes a test FAIL.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Where the primaries are — the one command this alert is about:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
     -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'

Are all three Clusters actually healthy enough to move one? (Fully-qualified: clusters.postgresql.cnpg.io — cluster is an ambiguous short name.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'

Is every node Ready (the gate the rule itself applies)?

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'

Is there storage or I/O trouble that makes a switchover a bad idea right now?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'

ssh k3s-prod-1 'cat /proc/pressure/io'

And confirm DR is intact before you deliberately move a primary — the primary owns the backup catalogue, and fully-qualified names matter: kubectl get backup -A prints "No resources found" on this cluster even with 27 CNPG backups present, because three CRDs register the kind Backup and the short name resolves to Longhorn's (zero objects):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'

Likely causes, most likely first

1. A node rebooted and the primaries never moved back

The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason to move them back afterwards. Remediation: a deliberate switchover, below.

2. A rolling operator or Postgres upgrade

primaryUpdateMethod: switchover is set deliberately (the CRD default restart is a delete-and-recreate of the primary, i.e. tens of seconds with no Postgres — an SSO outage on keycloak-db). Switchovers during an upgrade are correct behaviour and leave the primaries wherever they landed.

3. One node was cordoned or unschedulable for a while

Same mechanism. Note that podAntiAffinityType: required means an instance stays Pending until a node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why the instances stay spread even when the primaries do not.

4. A Longhorn or I/O event caused a health-check failover

A primary whose storage stalled gets demoted. Cross-read /runbooks/node-io and /runbooks/longhorn-degraded — if that is why, fix the node before rebalancing.

5. The alert is firing during a genuine node outage

It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.

Remediation — move one primary, deliberately

The cnpg plugin is NOT in the devcontainer image; fetch the kubectl-cnpg release matching the operator version (CNPG 1.30.0 here) before you start.

kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>

Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next. A switchover interrupts writes to that database for about 9 seconds — measured on both clusters moved on 2026-09-13. Nine seconds of keycloak-db unavailability is a brief SSO interruption; doing two at once is an avoidable compound one.

Pick the target by reading the primaries table above and choosing an instance that already sits on a node holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear on its own.

When to escalate or stop

• Do not rebalance during any other incident. A switchover is a small, deliberate write interruption, and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with degraded Longhorn volumes. This alert has a 30-minute for precisely because it is never urgent.
• Never switch over a Cluster whose readyInstances is below 3. You would be promoting into a cluster that has no standby left to fall back on.
• Never change dataDurability to required while thinking about durability here: it is a strict quorum, and one degraded standby would then block writes entirely on the SSO database.
• Do not kubectl delete pod the primary to force a failover. That is restart semantics — the thing primaryUpdateMethod: switchover exists to avoid.
• Check ContinuousArchiving and firstRecoverabilityPoint first (/runbooks/cnpg-archiving). If archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius tidying completely.
• Escalate to the operator for the switchover itself if the affected Cluster is keycloak-db: the brief write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.

---

Source of truth: runbooks/cnpg-primary-spread.md. Index: /runbooks/ Records: platform/cnpg/alerts.yaml (the rule's own both-directions proof on live prod data and its promtool tests); commits f338e91 (instances 3), e8d8601 (required anti-affinity + switchover), cac413e (synchronous replication, dataDurability: preferred); the 2026-09-07 single-node SSO outage.