home-network-k3s-impl/platform/cnpg/alerts.yaml

---
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
#
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
#
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
# warning for a storage outage. The generic PVC-fill backstop is the separate
# platform/monitoring/pvc-capacity-alerts.yaml.
#
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
# failure this repo keeps hitting.
#
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
# 2026-09-06). CNPG's exporter publishes TWO independent families:
#
#   * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
#     and the backup catalogue). Present for EVERY instance.
#   * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
#
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
# exporter cannot log in to its own database:
#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)  [user cnpg_metrics_exporter]
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
# itself so the blind spot cannot recur unnoticed.
#
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
#   cnpg_collector_pg_wal_archive_status{value="ready"}   prod: keycloak 17, nextcloud 7, semaphore 3
#                                                          dev: 0, 0, 0  (min over 24h also 0)
#   cnpg_collector_first_recoverability_point             prod: 0, 0, 0
#                                                          dev: 1788018752 / 1788018762 / 1788018770
#   cnpg_collector_last_available_backup_timestamp        prod: 0, 0, 0   dev: 0.1-3.1h old
#   cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64  dev: 36 vs 64  (neither fires yet)
#   cnpg_last_error                                       prod+dev: nextcloud-db = 1
#
# AGGREGATION SCOPE — WHICH RULES MAY KEY ON `pod`, AND WHY (incident, 2026-09-12). The first cut of
# this file grouped EVERY rule `by (namespace, pod)`. That is correct only for metrics every instance
# reports independently, and WRONG for the two backup-catalogue metrics, which ONLY THE PRIMARY
# populates. Verified live on prod 2026-09-12 (`cnpg_collector_first_recoverability_point`, raw):
#     keycloak-db-2  = 1788764922   nextcloud-db-2 = 1788764955   semaphore-db-2 = 1788764954  <- PRIMARIES
#     keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 = 0                               <- REPLICAS
# Since the Clusters went to `instances: 3`, `CNPGNoRecoverabilityPoint` was therefore PERMANENTLY
# FIRING on all six replicas while all three Clusters reported firstRecoverabilityPoint=2026-09-07,
# lastSuccessfulBackup within hours, ContinuousArchiving=True and LastBackupSucceeded=True. Backups
# were fine; the alert was noise, and noise on a DR alarm is worse than no alarm at all — a REAL loss
# of recoverability would have arrived as a seventh identical email nobody reads.
#
# THE RULE OF THUMB, applied as an audit to every rule below:
#   * `== 0` / staleness shapes ("nothing reports a value") MUST aggregate `pod` AWAY, or every pod
#     that structurally never reports the value fires forever.
#   * `> 0` shapes ("some pod reports a bad value") are SAFE per-pod: a pod sitting at 0 simply does
#     not fire, and keeping `pod` names the instance to go look at.
#
# THE GROUPING LABEL IS `job`, NOT `namespace`. The metric carries NO `cnpg.io/cluster` label — its
# full label set on prod is {container, endpoint, instance, job, namespace, pod} — but the CNPG
# operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace,
# so prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
# `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. That identifies the CNPG CLUSTER
# uniquely. Grouping by `namespace` alone happens to work today (one Cluster per namespace, verified)
# but would let two Clusters in one namespace MASK each other — the healthy one's non-zero value would
# satisfy `max` for both. `job` cannot do that, so `job` is what the DR rules group by.
#
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
# which fires only when an outage is genuinely imminent.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: platform-health-cnpg
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
    platform-health: "true"
spec:
  groups:
    - name: platform-health.cnpg
      rules:
        # ---------------------------------------------------------------------------------------
        # THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
        # the hour instead of 38 hours later.
        #
        # pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
        # archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
        # seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
        # segment is 16 MiB that Postgres may not recycle.
        #
        # THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
        # condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
        # queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
        # returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
        # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
        # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
        # backlog sits at 3 because it writes almost nothing.
        #
        # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately. Every instance has its own pg_wal on its
        # own PVC and every instance reports this metric (verified: all 9 prod pods publish
        # `{value="ready"}` = 0 today), and this is a `> 0` shape — a replica sitting at 0 never fires,
        # so there is no permanent-firing hazard and no masking (`> 0` fires if ANY pod is bad). `pod`
        # earns its place here: it names the instance whose archiver is stuck.
        - alert: CNPGWALArchiveBacklog
          expr: |
            max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
            description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
        # ---------------------------------------------------------------------------------------
        # The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
        # time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
        # is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
        # be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
        # and an old base backup can exist while archiving has just broken (only the other fires).
        #
        # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
        # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
        # start quiet while still catching a bootstrap that never produced a recoverability point.
        #
        # SCOPE — `by (namespace, job)`, NOT `by (namespace, pod)`. THIS IS THE FIX FOR THE 2026-09-12
        # permanently-firing incident (see the AGGREGATION SCOPE block in the header). The recoverability
        # point is a property of the CLUSTER's backup catalogue, which only the PRIMARY reads; a replica
        # reports a hard 0 forever. Grouped by `pod`, each replica was evaluated on its own and this
        # `== 0` fired permanently on all six of them. Aggregating `pod` away makes `max` take the
        # PRIMARY's non-zero value as the cluster's answer, so the alert says what its own summary always
        # claimed to say — "this Postgres CLUSTER has no recoverability point" — and fires only when NO
        # instance of that Cluster reports one.
        #
        # PROVEN LIVE, read-only, prod, 2026-09-12 (promtool query instant):
        #   OLD  max by (namespace, pod) (...) == 0  -> 6 series: keycloak-db-1/-3, nextcloud-db-1/-3,
        #                                              semaphore-db-1/-3   (and 6 ALERTS firing)
        #   NEW  max by (namespace, job) (...) == 0  -> EMPTY
        # and it is not merely vacuous — forcing the metric to 0 for one cluster's primary only
        # (`* on(pod) group_left() ...{pod="keycloak-db-2"}`) yields exactly one series,
        # {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster.
        #
        # A switchover cannot flap this: while the primary restarts, `max` briefly sees only replicas' 0,
        # but `for: 6h` requires the condition at EVERY evaluation for six hours and a promotion takes
        # seconds. Annotations key on `job` (= "<namespace>/<cluster>") because `pod` no longer exists on
        # these series — a `{{ $labels.pod }}` left behind here would render as an empty string.
        - alert: CNPGNoRecoverabilityPoint
          expr: |
            max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0
          for: 6h
          labels:
            severity: warning
          annotations:
            summary: "CNPG cluster has NO recoverability point: {{ $labels.job }}"
            description: "PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) reports firstRecoverabilityPoint = null on EVERY instance for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). This is a per-CLUSTER statement: the primary owns the backup catalogue, so a healthy cluster clears it even though its replicas report 0. Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
        # ---------------------------------------------------------------------------------------
        # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
        # platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
        # databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
        # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
        # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
        # ~57 years and always true). 36h = one missed daily run plus slack.
        #
        # SCOPE — `by (namespace, job)`, fixed alongside CNPGNoRecoverabilityPoint (2026-09-12).
        # `cnpg_collector_last_available_backup_timestamp` has the SAME primary-only defect, measured on
        # prod the same day: keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and
        # ALL SIX REPLICAS = 0, i.e. `time() - 0` = 497020h ≈ 56.7 YEARS. Unlike the rule above this one
        # was not firing, because the `> 0` guard was load-bearing — it was the single thing standing
        # between this rule and six more permanent false positives, which is far too thin a margin for a
        # DR alarm. Aggregating `pod` away removes the dependence: `max` picks the PRIMARY's timestamp as
        # the cluster's answer (a demoted primary drops to 0 and can never win the `max`, and a fresher
        # backup always beats a staler one), the `> 0` guard then means exactly "this cluster has had at
        # least one backup", and the result is a real per-cluster staleness statement.
        #
        # PROVEN LIVE, prod, 2026-09-12: OLD -> empty, NEW -> empty (so the fix changes no live verdict),
        # while the per-CLUSTER ages the new aggregation actually evaluates are keycloak 3.09h /
        # nextcloud 2.09h / semaphore 0.09h — the primaries' real values, all far inside 36h.
        - alert: CNPGNoRecentBaseBackup
          expr: |
            (time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
            and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
          for: 2h
          labels:
            severity: warning
          annotations:
            summary: "No recent CNPG base backup: {{ $labels.job }}"
            description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
        # ---------------------------------------------------------------------------------------
        # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
        # which means checkpoints are no longer reclaiming space — the mechanism that actually fills
        # the PVC. Both series come from the same metric under different `value` labels, so each side
        # is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
        # label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
        # both clusters), `count` = segments currently in pg_wal.
        #
        # This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
        # CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
        # to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
        # at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
        # runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
        # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
        # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
        # therefore NOT usable as a signal.
        #
        # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately — and here it is REQUIRED. Every instance
        # reports its own count/max (verified on all 9 prod pods: count 16-37 against max 64), pg_wal is
        # per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself:
        # aggregating `pod` away would compare one instance's count with another's max. `>` is also a
        # per-pod shape, so no instance can sit permanently true.
        - alert: CNPGWALVolumeNotRecycling
          expr: |
            max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
              > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
          for: 2h
          labels:
            severity: warning
          annotations:
            summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
            description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
        # ---------------------------------------------------------------------------------------
        # THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
        # queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
        # `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
        #
        # THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
        # 2026-09-06: `cnpg_errors_total` carries
        #   pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
        #   database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
        # i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
        # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
        # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
        # permanently-firing `critical` this repo has been burned by.
        #
        # SCOPE AUDIT (2026-09-12): keeps `pod`, UNCHANGED — this rule's meaning and firing set are
        # deliberately left exactly as they were. `cnpg_last_error` is genuinely per-instance: each
        # instance's own exporter reports its own SQL-collection health, and prod proves the metric
        # discriminates rather than tracking role — all THREE nextcloud-db pods (primary -2 included)
        # report 1 while all six keycloak/semaphore pods report 0. So there is no primary-only defect to
        # fix, and per-pod is the right granularity: the remedy is a GRANT applied per database, and you
        # want to see it clear on every instance. It is a TRUE POSITIVE right now (nextcloud-db, the
        # 42501 CONNECT grant) and is owned by separate work; nothing here silences or rescopes it.
        - alert: CNPGMetricsCollectorErroring
          expr: |
            max by (namespace, pod) (cnpg_last_error) == 1
          for: 6h
          labels:
            severity: warning
          annotations:
            summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
            description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
        # ---------------------------------------------------------------------------------------
        # BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
        #
        # WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
        # The operator's original ask was "the database primaries are all on one node". `instances: 3`
        # + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
        # exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
        # PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
        # semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
        # failovers — survivable, but the worst single-node blast radius the cluster can have.
        #
        # THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
        # (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
        # the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
        #   * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
        #     additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
        #     the cluster's pods". None is role-aware: there is no way to say "this constraint is for
        #     whichever pod is currently primary".
        #   * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
        #     Cluster that is the job podAntiAffinityType already does; it says nothing about how two
        #     INDEPENDENT Clusters' primaries relate.
        #   * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
        #     (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
        #   * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
        #     postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
        #     primary lands.
        # Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
        # scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
        # imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
        #
        # AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
        # cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
        # (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
        # RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
        # (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
        # constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
        # actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
        # excluding the node that holds another cluster's primary leaves the third instance
        # UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
        #
        # SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
        # status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
        # rebalances today and drifts again on the next failover, node reboot or rolling update.
        # A switchover that silently un-does itself is not a fix, so what is committed here is the
        # part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
        # a known, cheap runbook step instead of something discovered by hand months later.
        #
        # WHY THIS PROMQL. Neither family alone can answer the question:
        #   * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
        #   * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
        #     --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
        # The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
        # Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
        # and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
        # `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
        # `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
        # that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
        # cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
        # the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
        # is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
        #
        # THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
        # three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
        # is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
        # rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
        # the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
        #
        # for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
        # 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
        # still reporting real drift the same hour. severity: warning, like every rule in this file:
        # this is a resilience posture to correct at leisure, never a page.
        #
        # PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
        # fire is decoration and one that cannot stop is fatigue:
        #   * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
        #     {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
        #     throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
        #     observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
        #     was deleted afterwards; it was never committed.
        #   * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
        #     empty vector, while the inner join still resolved all three primaries — semaphore-db-2
        #     on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
        #     the condition is false, NOT because the query went blind.
        #   * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
        #     platform/monitoring/tests/run.sh — the one repo-wide harness; the merge gate runs the
        #     same path via conformance/tests/l0-static/alert-rules-unit-tested) covers four cases
        #     under promtool 3.14.0, the exact Prometheus
        #     version running here: co-located fires after 30m, one-per-node is silent, a NotReady
        #     node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
        #     Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
        #     readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
        - alert: CNPGPrimariesCoLocated
          expr: |
            count by (node) (
              max by (namespace, pod, node) (
                label_replace(
                  label_replace(
                    kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
                    "pod", "$1", "targetref_name", "(.+)"
                  ),
                  "node", "$1", "endpoint_nodename", "(.+)"
                )
              )
              * on (namespace, pod) group_left ()
              max by (namespace, pod) (cnpg_collector_up)
            ) > 1
            and on ()
            count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
            description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"