feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent

`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.

There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.

So this commits the part that IS durable — visibility — and rebalances live.

CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.

A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.

Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.

Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.

The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/cnpg/alerts.yaml

@@ -265,3 +265,116 @@ spec:
             summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
             description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
+        # ---------------------------------------------------------------------------------------
+        # BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
+        #
+        # WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
+        # The operator's original ask was "the database primaries are all on one node". `instances: 3`
+        # + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
+        # exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
+        # PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
+        # semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
+        # failovers — survivable, but the worst single-node blast radius the cluster can have.
+        #
+        # THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
+        # (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
+        # the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
+        #   * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
+        #     additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
+        #     the cluster's pods". None is role-aware: there is no way to say "this constraint is for
+        #     whichever pod is currently primary".
+        #   * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
+        #     Cluster that is the job podAntiAffinityType already does; it says nothing about how two
+        #     INDEPENDENT Clusters' primaries relate.
+        #   * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
+        #     (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
+        #   * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
+        #     postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
+        #     primary lands.
+        # Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
+        # scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
+        # imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
+        #
+        # AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
+        # cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
+        # (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
+        # RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
+        # (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
+        # constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
+        # actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
+        # excluding the node that holds another cluster's primary leaves the third instance
+        # UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
+        #
+        # SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
+        # status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
+        # rebalances today and drifts again on the next failover, node reboot or rolling update.
+        # A switchover that silently un-does itself is not a fix, so what is committed here is the
+        # part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
+        # a known, cheap runbook step instead of something discovered by hand months later.
+        #
+        # WHY THIS PROMQL. Neither family alone can answer the question:
+        #   * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
+        #   * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
+        #     --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
+        # The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
+        # Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
+        # and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
+        # `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
+        # `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
+        # that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
+        # cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
+        # the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
+        # is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
+        #
+        # THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
+        # three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
+        # is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
+        # rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
+        # the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
+        #
+        # for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
+        # 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
+        # still reporting real drift the same hour. severity: warning, like every rule in this file:
+        # this is a resilience posture to correct at leisure, never a page.
+        #
+        # PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
+        # fire is decoration and one that cannot stop is fatigue:
+        #   * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
+        #     {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
+        #     throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
+        #     observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
+        #     was deleted afterwards; it was never committed.
+        #   * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
+        #     empty vector, while the inner join still resolved all three primaries — semaphore-db-2
+        #     on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
+        #     the condition is false, NOT because the query went blind.
+        #   * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
+        #     ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
+        #     version running here: co-located fires after 30m, one-per-node is silent, a NotReady
+        #     node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
+        #     Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
+        #     readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
+        - alert: CNPGPrimariesCoLocated
+          expr: |
+            count by (node) (
+              max by (namespace, pod, node) (
+                label_replace(
+                  label_replace(
+                    kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
+                    "pod", "$1", "targetref_name", "(.+)"
+                  ),
+                  "node", "$1", "endpoint_nodename", "(.+)"
+                )
+              )
+              * on (namespace, pod) group_left ()
+              max by (namespace, pod) (cnpg_collector_up)
+            ) > 1
+            and on ()
+            count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
+            description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"