feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent

`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.

There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.

So this commits the part that IS durable — visibility — and rebalances live.

CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.

A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.

Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.

Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.

The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2561,3 +2561,130 @@ own status conditions and find out why the list is lying before acting.
 **Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a
 backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`,
 `prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.
+## 2026-09-13 — CNPG primaries spread across nodes, and the honest answer: **there is no declarative fix** (branch `fix/cnpg-primary-spread`)
+
+The original ask was *"fix the database primaries being solely on prod 2"*. The instance half of
+that was already done (`instances: 3` + `affinity.podAntiAffinityType: required` +
+`primaryUpdateMethod: switchover`, commits `f338e91`/`e8d8601`/`cac413e`): every Cluster now has
+exactly one pod per node and a synchronous standby, so no single node loss can take a database
+down. The primary half was **not** done — and worse, that roll had *moved* the concentration
+rather than removed it:
+
+```
+BEFORE (2026-09-13 04:19 UTC)          AFTER (04:26 UTC)
+keycloak-db-2   -> k3s-prod-1  primary  keycloak-db-1   -> k3s-prod-2  primary
+nextcloud-db-2  -> k3s-prod-1  primary  nextcloud-db-1  -> k3s-prod-3  primary
+semaphore-db-2  -> k3s-prod-1  primary  semaphore-db-2  -> k3s-prod-1  primary
+```
+
+All three primaries had landed on `k3s-prod-1` — the node whose failing drive was just replaced and
+whose etcd WAL shares a 2011 SSD with Longhorn. Losing it meant three simultaneous failovers.
+
+### The judgment call, made BEFORE touching anything
+
+**CNPG 1.30.0 has no primary-placement primitive, and the obvious workaround is worse than
+nothing.** Established by walking the openAPIV3Schema of the CRD *actually installed on prod*
+(`clusters.postgresql.cnpg.io/v1`, controller-gen v0.21.0) for
+`/primar|promot|spread|topolog|prefer.*node|switchover/`, not from memory:
+
+- `spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
+  additionalPodAffinity,additionalPodAntiAffinity}` — every one is documented as applying to *"all
+  the cluster's pods"*. None is role-aware.
+- `spec.topologySpreadConstraints` — spreads a Cluster's **own** pods; says nothing about how two
+  independent Clusters' primaries relate.
+- `spec.replica.primary` — names which **Cluster object** is primary in a distributed
+  (`externalClusters`) topology. Not a node.
+- `spec.primaryUpdateMethod` / `primaryUpdateStrategy` / `switchoverDelay` / `primaryLease` /
+  `postgresql.promotionTimeout` — govern **how and when** a promotion happens, never **where**.
+- No other CNPG CRD is about scheduling (`backups`, `clusterimagecatalogs`, `databaseroles`,
+  `databases`, `failoverquorums`, `imagecatalogs`, `poolers`, `publications`, `scheduledbackups`,
+  `subscriptions`).
+- **`targetPrimary` exists only under `status`, never `spec`** (confirmed against the live schema).
+  So `cnpg promote` is an imperative status patch by construction — there is nothing to write in git.
+
+And `additionalPodAntiAffinity` against `cnpg.io/instanceRole=primary` fails **twice**: (1) Kubernetes
+pod anti-affinity is only ever `...IgnoredDuringExecution` — there is no `RequiredDuringExecution`
+variant — while CNPG promotes by **relabelling an already-running pod**, so a scheduling-time
+constraint cannot bind a promotion at all; (2) even at scheduling time, with 3 instances on 3 nodes
+and `required` intra-cluster anti-affinity, excluding the node holding another primary leaves the
+third instance **unschedulable** — trading blast radius for a permanent `readyInstances: 2`.
+
+**So option (a) — "accept it, every cluster already survives node loss" — was rejected as
+incomplete, and a git-only "fix" was rejected as impossible.** What was done instead: reduce the
+blast radius today, and make the drift **visible** so it is a known cheap runbook step rather than
+something found by hand months later. The placement below is **current, not durable** — say so.
+
+### Why the drift is not hypothetical
+
+It already happened, from a routine declarative change: the `required` anti-affinity roll is
+precisely what re-promoted all three clusters onto one node. `keycloak-db` reported
+`Primary promotion time: 2026-09-08` — a re-promotion nobody asked for. No prod outage was staged to
+"prove" non-durability; the repo's own history is the evidence.
+
+### The rebalance: two switchovers, not three
+
+`semaphore-db` was **deliberately left** with its primary on `k3s-prod-1`. One primary per node is
+the goal, and semaphore is by far the lightest writer (its WAL backlog sat at 3 segments because it
+writes almost nothing), so it is the right tenant for the node with the known etcd/Longhorn I/O
+contention. Two switchovers instead of three = one less write interruption.
+
+**Disruption, measured — not estimated.** A `select txid_current()` probe (a real write: it assigns
+an xid) ran against each `-rw` Service from a pod that stayed a replica throughout, password fed via
+stdin so it never reached argv:
+
+| cluster | target | write outage | detail |
+|---|---|---|---|
+| `keycloak-db` (SSO) | `keycloak-db-1` on k3s-prod-2 | **~9.4 s** | last OK 04:20:24.634 → fail burst to 04:20:33.731; healthy at t+30s |
+| `nextcloud-db` | `nextcloud-db-1` on k3s-prod-3 | **9.43 s** | 1402 samples @44 ms; ONE burst 04:25:46.854 → 04:25:56.287; healthy at t+25s |
+| `semaphore-db` | *(none — kept on k3s-prod-1)* | **0 s** | stayed `readyInstances: 3` throughout |
+
+One contiguous failure burst each, no flapping, no second dip. Note the 30 s scrape interval showed
+**no gap at all** for keycloak-db — the `-rw` endpoint was `ready` at every sample. A ~9 s write
+outage is invisible to Prometheus at this resolution; only the sub-second probe sees it.
+
+After: all three `readyInstances: 3`, `Cluster in healthy state`, synchronous replication intact —
+`synchronous_standby_names = ANY 1 (...)` in force on each primary with both standbys
+`state=streaming sync_state=quorum`.
+
+### The durable part: `CNPGPrimariesCoLocated`
+
+Added to `platform/cnpg/alerts.yaml` (already wired via `platform-config/kustomization.yaml`, so
+Flux reconciles it on merge — no hand-apply).
+
+**Neither metric family can answer the question alone**, which is the whole difficulty:
+`cnpg_*` metrics carry `namespace`/`pod` but **no node**; and `kube_pod_labels` is **empty** on this
+cluster (kube-state-metrics has no `--metric-labels-allowlist`), so `cnpg.io/instanceRole` is not in
+Prometheus at all. The rule instead reads **CNPG's own definition of "who is primary"** — the
+`<cluster>-rw` Service selects the primary and nothing else, and
+`kube_endpointslice_endpoints` carries both `endpoint_nodename` and `targetref_name`. Two
+`label_replace`s lift those into `node`/`pod`; `* on (namespace, pod) group_left () cnpg_collector_up`
+restricts the count to real CNPG instances so an unrelated `*-rw` Service cannot trigger it.
+`cnpg_collector_up` is the right-hand side deliberately — it exists for **every** instance, unlike
+the `cnpg_pg_stat_*` family which is missing entirely for `nextcloud-db`.
+
+**A Ready-node gate keeps it from becoming the next permanently-firing alert.** With three clusters
+on three nodes, losing a node *necessarily* co-locates two primaries — correct behaviour, not a
+defect — so the rule only speaks while every node is `Ready`, and re-arms itself when the node
+returns. `for: 30m` rides out a switchover (seconds) and a full rolling update (minutes).
+`severity: warning`, like everything else in that file: a resilience posture, never a page.
+
+**Proven in both directions, because this repo has been burned by both.** Live: the expression
+returned `{node="k3s-prod-1"} => 3` before, and an empty vector after (while the inner join still
+resolved all three primaries — quiet because the condition is false, not because the query went
+blind). Loaded into prod's Prometheus as a throwaway twin (`for: 1m`), it was observed going
+`pending` at +70 s and `firing` at +130 s, then deleted. And `platform/cnpg/alerts.promtool-test.yaml`
+(`./run-alert-tests.sh`) pins all four behaviours under promtool 3.14.0 — the exact Prometheus
+version on prod. **The tests were mutation-checked:** removing the `cnpg_collector_up` join,
+removing the readiness gate, or shortening `for:` each makes a test *fail*, so they are not vacuous.
+
+### Debt / follow-ups
+
+- **Placement is current, not durable.** The next failover, node reboot or CNPG rolling update can
+  re-concentrate the primaries. That is now *reported*, not *prevented*. Rebalancing is a ~9 s
+  write interruption per cluster.
+- **`promtool` is not in the baked toolchain** (`infra/devcontainer/versions.env`), so
+  `run-alert-tests.sh` fetches it on demand and no CI stage runs the alert unit tests. Adding
+  `PROMTOOL_VERSION` to `ci/versions.env` + the Containerfile would let `make -C ci gate` run them.
+- **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`.
+  `cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description`
+  carries the full actionable procedure so the missing page is not load-bearing.