home-network-k3s-impl/runbooks/cnpg-primary-spread.md

# cnpg-primary-spread — Two or more CNPG primaries share one node

**Alert:** `CNPGPrimariesCoLocated` (warning, 30m)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:** `count by (node) (<ready *-rw endpoints joined to cnpg_collector_up>) > 1`
**and** gated on every node being Ready:
`count(kube_node_status_condition{condition="Ready",status="true"} == 1) == count(kube_node_info)`

## What this means here

More than one CloudNativePG **primary** has been running on the same node for 30 minutes, while all
three nodes are Ready. There are exactly three CNPG Clusters on this platform — `keycloak/keycloak-db`,
`nextcloud-db/nextcloud-db` and `semaphore-db/semaphore-db` — so on a 3-node cluster the healthy shape is
one primary per node, and this alert says that shape has drifted.

**This is BLAST RADIUS, not an outage.** Each Cluster has `instances: 3` with one pod per node
(`podAntiAffinityType: required`) and one synchronous standby (`method: any`, `number: 1`,
`dataDurability: preferred`), so every one of them survives losing the shared node — you just get two or
three simultaneous PostgreSQL failovers instead of one. Nothing is broken right now, which is why it is a
`warning` with a long `for`.

**Why it matters anyway: this exact drift has already caused an outage.** All three Clusters used to run
`instances: 1` and all three primaries had drifted onto one node, and **on 2026-09-07 that node took SSO,
Nextcloud and Semaphore down together.** `instances: 3` means the same drift no longer causes an outage —
but if `keycloak-db`'s primary is co-located and that node dies, the SSO failover happens at the same
moment as two others, on nodes that are also absorbing Longhorn replica rebuilds.

**Why the drift happens and cannot be declared away: CNPG has no declarative primary-placement field.**
The primary lands wherever the last promotion put it — a node reboot, a switchover, an operator upgrade.
So this is avoidable drift that must be *corrected*, not prevented, and the alert re-arms itself so it
will be reported again rather than silently persisting.

**Why the node-readiness gate is in the expression.** During a real node outage, primaries legitimately
pile onto the survivors and there is nothing to do about it; alerting then would be noise at the worst
moment. The `and on ()` clause suppresses the rule unless every node is Ready, and re-arms when the node
returns.

This rule was proven in both directions on live prod data (2026-09-13): the expression returned
`{node="k3s-prod-1"} => 3` before the rebalance and a **throwaway twin with `for: 1m`** was observed going
`pending` at +70s and `firing` at +130s; after two switchovers the same expression returned an **empty
vector while the inner join still resolved all three primaries** — quiet because the condition is false,
not because the query went blind. It is also unit-tested in `alerts.promtool-test.yaml`
(run `./run-alert-tests.sh`) with four cases, mutation-checked so that deleting the `cnpg_collector_up`
join, the readiness gate, or shortening `for:` makes a test FAIL.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Where the primaries are — the one command this alert is about:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A -l cnpg.io/instanceRole=primary \
     -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName'
```

Are all three Clusters actually healthy enough to move one? (Fully-qualified:
`clusters.postgresql.cnpg.io` — `cluster` is an ambiguous short name.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```

Is every node Ready (the gate the rule itself applies)?

```
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes -o wide'
```

Is there storage or I/O trouble that makes a switchover a bad idea right now?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
```

```
ssh k3s-prod-1 'cat /proc/pressure/io'
```

And confirm DR is intact before you deliberately move a primary — the primary owns the backup
catalogue, and **fully-qualified names matter**: `kubectl get backup -A` prints "No resources found" on
this cluster even with 27 CNPG backups present, because three CRDs register the kind `Backup` and the
short name resolves to Longhorn's (zero objects):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A | tail -10; echo; \
   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```

## Likely causes, most likely first

### 1. A node rebooted and the primaries never moved back
The dominant cause. Promotions during the reboot put two primaries on a survivor, and CNPG has no reason
to move them back afterwards. Remediation: a deliberate switchover, below.

### 2. A rolling operator or Postgres upgrade
`primaryUpdateMethod: switchover` is set deliberately (the CRD default `restart` is a
delete-and-recreate of the *primary*, i.e. tens of seconds with no Postgres — an SSO outage on
`keycloak-db`). Switchovers during an upgrade are correct behaviour and leave the primaries wherever
they landed.

### 3. One node was cordoned or unschedulable for a while
Same mechanism. Note that `podAntiAffinityType: required` means an instance stays **Pending** until a
node returns rather than co-locating — that is the accepted cost of the 2026-09-07 fix, and it is why
the *instances* stay spread even when the *primaries* do not.

### 4. A Longhorn or I/O event caused a health-check failover
A primary whose storage stalled gets demoted. Cross-read `/runbooks/node-io` and
`/runbooks/longhorn-degraded` — if that is why, fix the node before rebalancing.

### 5. The alert is firing during a genuine node outage
It should not be able to: the readiness gate suppresses it. If it fires anyway with a node NotReady, the
gate is broken — that is a monitoring bug worth fixing, since a rule that cannot stop is fatigue.

## Remediation — move one primary, deliberately

The cnpg plugin is **NOT in the devcontainer image**; fetch the `kubectl-cnpg` release matching the
operator version (CNPG **1.30.0** here) before you start.

```
kubectl cnpg promote -n <namespace> <cluster> <instance-on-an-idle-node>
```

**Do ONE CLUSTER AT A TIME and wait for `readyInstances: 3` before the next.** A switchover interrupts
writes to that database for about **9 seconds** — measured on both clusters moved on 2026-09-13. Nine
seconds of `keycloak-db` unavailability is a brief SSO interruption; doing two at once is an avoidable
compound one.

Pick the target by reading the primaries table above and choosing an instance that already sits on a node
holding no other primary. Then re-run the table to confirm the new distribution, and let the alert clear
on its own.

## When to escalate or stop

- **Do not rebalance during any other incident.** A switchover is a small, deliberate write interruption,
  and it is the wrong thing to add to a node that is already under I/O pressure or to a cluster with
  degraded Longhorn volumes. This alert has a 30-minute `for` precisely because it is never urgent.
- **Never switch over a Cluster whose `readyInstances` is below 3.** You would be promoting into a
  cluster that has no standby left to fall back on.
- **Never change `dataDurability` to `required`** while thinking about durability here: it is a strict
  quorum, and one degraded standby would then **block writes** entirely on the SSO database.
- **Do not `kubectl delete pod` the primary to force a failover.** That is `restart` semantics — the
  thing `primaryUpdateMethod: switchover` exists to avoid.
- **Check `ContinuousArchiving` and `firstRecoverabilityPoint` first** (`/runbooks/cnpg-archiving`). If
  archiving is broken, that is a real countdown to a disk-full PANIC and it outranks blast-radius
  tidying completely.
- Escalate to the operator for the switchover itself if the affected Cluster is `keycloak-db`: the brief
  write interruption is user-visible across SSO, kubectl OIDC, Grafana, Headlamp and Gitea.

---
Source of truth: `runbooks/cnpg-primary-spread.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the rule's own both-directions proof on live prod data and its
promtool tests); commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover),
`cac413e` (synchronous replication, `dataDurability: preferred`); the 2026-09-07 single-node SSO outage.