--- # CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals. # # WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db, # nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with # `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP # (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert # of any kind in platform/ or clusters/ (verified by grep, 2026-09-06). # # WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres # REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the # postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB # PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO # database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it. # So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early # warning for a storage outage. The generic PVC-fill backstop is the separate # platform/monitoring/pvc-capacity-alerts.yaml. # # WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, # so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the # controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and # platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the # operator's Prometheus select these rules (ruleSelector). It IS referenced from # platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't" # failure this repo keeps hitting. # # WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only, # 2026-09-06). CNPG's exporter publishes TWO independent families: # # * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status # and the backup catalogue). Present for EVERY instance. # * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED. # # Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db # exporter cannot log in to its own database: # FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter] # so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all. # An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND # to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition # itself so the blind spot cannot recur unnoticed. # # LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not): # cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3 # dev: 0, 0, 0 (min over 24h also 0) # cnpg_collector_first_recoverability_point prod: 0, 0, 0 # dev: 1788018752 / 1788018762 / 1788018770 # cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old # cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet) # cnpg_last_error prod+dev: nextcloud-db = 1 # # AGGREGATION SCOPE — WHICH RULES MAY KEY ON `pod`, AND WHY (incident, 2026-09-12). The first cut of # this file grouped EVERY rule `by (namespace, pod)`. That is correct only for metrics every instance # reports independently, and WRONG for the two backup-catalogue metrics, which ONLY THE PRIMARY # populates. Verified live on prod 2026-09-12 (`cnpg_collector_first_recoverability_point`, raw): # keycloak-db-2 = 1788764922 nextcloud-db-2 = 1788764955 semaphore-db-2 = 1788764954 <- PRIMARIES # keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 = 0 <- REPLICAS # Since the Clusters went to `instances: 3`, `CNPGNoRecoverabilityPoint` was therefore PERMANENTLY # FIRING on all six replicas while all three Clusters reported firstRecoverabilityPoint=2026-09-07, # lastSuccessfulBackup within hours, ContinuousArchiving=True and LastBackupSucceeded=True. Backups # were fine; the alert was noise, and noise on a DR alarm is worse than no alarm at all — a REAL loss # of recoverability would have arrived as a seventh identical email nobody reads. # # THE RULE OF THUMB, applied as an audit to every rule below: # * `== 0` / staleness shapes ("nothing reports a value") MUST aggregate `pod` AWAY, or every pod # that structurally never reports the value fires forever. # * `> 0` shapes ("some pod reports a bad value") are SAFE per-pod: a pod sitting at 0 simply does # not fire, and keeping `pod` names the instance to go look at. # # THE GROUPING LABEL IS `job`, NOT `namespace`. The metric carries NO `cnpg.io/cluster` label — its # full label set on prod is {container, endpoint, instance, job, namespace, pod} — but the CNPG # operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace, # so prometheus-operator derives `job="/"`: `keycloak/keycloak-db`, # `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. That identifies the CNPG CLUSTER # uniquely. Grouping by `namespace` alone happens to work today (one Cluster per namespace, verified) # but would let two Clusters in one namespace MASK each other — the healthy one's non-zero value would # satisfy `max` for both. `job` cannot do that, so `job` is what the DR rules group by. # # SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate. # A database heading for disk-full deserves a warning with weeks of lead time, and this repo already # has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in # platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually # running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml — # which fires only when an outage is genuinely imminent. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-cnpg namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.cnpg rules: # --------------------------------------------------------------------------------------- # THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within # the hour instead of 38 hours later. # # pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet # archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within # seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck # segment is 16 MiB that Postgres may not recycle. # # THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the # condition to hold at EVERY evaluation for the full hour, so a write burst that transiently # queues a segment resets the timer and never notifies; but a genuinely stuck archiver never # returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3). # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its # backlog sits at 3 because it writes almost nothing. # # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately. Every instance has its own pg_wal on its # own PVC and every instance reports this metric (verified: all 9 prod pods publish # `{value="ready"}` = 0 today), and this is a `> 0` shape — a replica sitting at 0 never fires, # so there is no permanent-firing hazard and no masking (`> 0` fires if ANY pod is bad). `pod` # earns its place here: it names the instance whose archiver is stuck. - alert: CNPGWALArchiveBacklog expr: | max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 for: 1h labels: severity: warning annotations: summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)" description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" # --------------------------------------------------------------------------------------- # The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in # time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint # is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can # be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires), # and an old base backup can exist while archiving has just broken (only the other fires). # # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold # start quiet while still catching a bootstrap that never produced a recoverability point. # # SCOPE — `by (namespace, job)`, NOT `by (namespace, pod)`. THIS IS THE FIX FOR THE 2026-09-12 # permanently-firing incident (see the AGGREGATION SCOPE block in the header). The recoverability # point is a property of the CLUSTER's backup catalogue, which only the PRIMARY reads; a replica # reports a hard 0 forever. Grouped by `pod`, each replica was evaluated on its own and this # `== 0` fired permanently on all six of them. Aggregating `pod` away makes `max` take the # PRIMARY's non-zero value as the cluster's answer, so the alert says what its own summary always # claimed to say — "this Postgres CLUSTER has no recoverability point" — and fires only when NO # instance of that Cluster reports one. # # PROVEN LIVE, read-only, prod, 2026-09-12 (promtool query instant): # OLD max by (namespace, pod) (...) == 0 -> 6 series: keycloak-db-1/-3, nextcloud-db-1/-3, # semaphore-db-1/-3 (and 6 ALERTS firing) # NEW max by (namespace, job) (...) == 0 -> EMPTY # and it is not merely vacuous — forcing the metric to 0 for one cluster's primary only # (`* on(pod) group_left() ...{pod="keycloak-db-2"}`) yields exactly one series, # {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster. # # A switchover cannot flap this: while the primary restarts, `max` briefly sees only replicas' 0, # but `for: 6h` requires the condition at EVERY evaluation for six hours and a promotion takes # seconds. Annotations key on `job` (= "/") because `pod` no longer exists on # these series — a `{{ $labels.pod }}` left behind here would render as an empty string. - alert: CNPGNoRecoverabilityPoint expr: | max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0 for: 6h labels: severity: warning annotations: summary: "CNPG cluster has NO recoverability point: {{ $labels.job }}" description: "PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) reports firstRecoverabilityPoint = null on EVERY instance for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). This is a per-CLUSTER statement: the primary owns the backup catalogue, so a healthy cluster clears it even though its replicas report 0. Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" # --------------------------------------------------------------------------------------- # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in # platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these # databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be # ~57 years and always true). 36h = one missed daily run plus slack. # # SCOPE — `by (namespace, job)`, fixed alongside CNPGNoRecoverabilityPoint (2026-09-12). # `cnpg_collector_last_available_backup_timestamp` has the SAME primary-only defect, measured on # prod the same day: keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and # ALL SIX REPLICAS = 0, i.e. `time() - 0` = 497020h ≈ 56.7 YEARS. Unlike the rule above this one # was not firing, because the `> 0` guard was load-bearing — it was the single thing standing # between this rule and six more permanent false positives, which is far too thin a margin for a # DR alarm. Aggregating `pod` away removes the dependence: `max` picks the PRIMARY's timestamp as # the cluster's answer (a demoted primary drops to 0 and can never win the `max`, and a fresher # backup always beats a staler one), the `> 0` guard then means exactly "this cluster has had at # least one backup", and the result is a real per-cluster staleness statement. # # PROVEN LIVE, prod, 2026-09-12: OLD -> empty, NEW -> empty (so the fix changes no live verdict), # while the per-CLUSTER ages the new aggregation actually evaluates are keycloak 3.09h / # nextcloud 2.09h / semaphore 0.09h — the primaries' real values, all far inside 36h. - alert: CNPGNoRecentBaseBackup expr: | (time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0 for: 2h labels: severity: warning annotations: summary: "No recent CNPG base backup: {{ $labels.job }}" description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed" # --------------------------------------------------------------------------------------- # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for, # which means checkpoints are no longer reclaiming space — the mechanism that actually fills # the PVC. Both series come from the same metric under different `value` labels, so each side # is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching # label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on # both clusters), `count` = segments currently in pg_wal. # # This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to: # CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough # to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks — # at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC # runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is # therefore NOT usable as a signal. # # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately — and here it is REQUIRED. Every instance # reports its own count/max (verified on all 9 prod pods: count 16-37 against max 64), pg_wal is # per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself: # aggregating `pod` away would compare one instance's count with another's max. `>` is also a # per-pod shape, so no instance can sit permanently true. - alert: CNPGWALVolumeNotRecycling expr: | max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) for: 2h labels: severity: warning annotations: summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)" description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth" # --------------------------------------------------------------------------------------- # THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user # queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while # `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health. # # THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live # 2026-09-06: `cnpg_errors_total` carries # pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter # database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) # i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all. # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the # permanently-firing `critical` this repo has been burned by. # # SCOPE AUDIT (2026-09-12): keeps `pod`, UNCHANGED — this rule's meaning and firing set are # deliberately left exactly as they were. `cnpg_last_error` is genuinely per-instance: each # instance's own exporter reports its own SQL-collection health, and prod proves the metric # discriminates rather than tracking role — all THREE nextcloud-db pods (primary -2 included) # report 1 while all six keycloak/semaphore pods report 0. So there is no primary-only defect to # fix, and per-pod is the right granularity: the remedy is a GRANT applied per database, and you # want to see it clear on every instance. It is a TRUE POSITIVE right now (nextcloud-db, the # 42501 CONNECT grant) and is owned by separate work; nothing here silences or rescopes it. - alert: CNPGMetricsCollectorErroring expr: | max by (namespace, pod) (cnpg_last_error) == 1 for: 6h labels: severity: warning annotations: summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}" description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter" # --------------------------------------------------------------------------------------- # BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE. # # WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12). # The operator's original ask was "the database primaries are all on one node". `instances: 3` # + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has # exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the # PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2, # semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous # failovers — survivable, but the worst single-node blast radius the cluster can have. # # THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here # (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking # the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/: # * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey, # additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all # the cluster's pods". None is role-aware: there is no way to say "this constraint is for # whichever pod is currently primary". # * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one # Cluster that is the job podAntiAffinityType already does; it says nothing about how two # INDEPENDENT Clusters' primaries relate. # * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology # (externalClusters / cross-k8s replica clusters). Not a node, not applicable here. # * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease / # postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the # primary lands. # Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about # scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums, # imagecatalogs, poolers, publications, scheduledbackups, subscriptions). # # AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled # cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod # (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no # RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod # (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time # constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be # actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity, # excluding the node that holds another cluster's primary leaves the third instance # UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2. # # SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches # status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live) # rebalances today and drifts again on the next failover, node reboot or rolling update. # A switchover that silently un-does itself is not a fix, so what is committed here is the # part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is # a known, cheap runbook step instead of something discovered by hand months later. # # WHY THIS PROMQL. Neither family alone can answer the question: # * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live). # * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no # --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all. # The join below instead reads CNPG's OWN definition of "who is primary": the `-rw` # Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint # and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and # `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the # `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods # that really are CNPG instances — so an unrelated Service that happens to be named `*-rw` # cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by # the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which # is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above). # # THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With # three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that # is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the # rule only speaks when every node is Ready, i.e. when the spread is actually achievable and # the co-location is therefore avoidable drift. It re-arms by itself once the node returns. # # for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so # 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while # still reporting real drift the same hour. severity: warning, like every rule in this file: # this is a resilience posture to correct at leisure, never a page. # # PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot # fire is decoration and one that cannot stop is fatigue: # * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned # {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a # throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and # observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin # was deleted afterwards; it was never committed. # * GOES QUIET once rebalanced. After the two switchovers the same expression returned an # empty vector, while the inner join still resolved all three primaries — semaphore-db-2 # on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because # the condition is false, NOT because the query went blind. # * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with # platform/monitoring/tests/run.sh — the one repo-wide harness; the merge gate runs the # same path via conformance/tests/l0-static/alert-rules-unit-tested) covers four cases # under promtool 3.14.0, the exact Prometheus # version running here: co-located fires after 30m, one-per-node is silent, a NotReady # node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw` # Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the # readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly. - alert: CNPGPrimariesCoLocated expr: | count by (node) ( max by (namespace, pod, node) ( label_replace( label_replace( kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, "pod", "$1", "targetref_name", "(.+)" ), "node", "$1", "endpoint_nodename", "(.+)" ) ) * on (namespace, pod) group_left () max by (namespace, pod) (cnpg_collector_up) ) > 1 and on () count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info) for: 30m labels: severity: warning annotations: summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}" description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n ` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"