--- # CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals. # # WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db, # nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with # `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP # (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert # of any kind in platform/ or clusters/ (verified by grep, 2026-09-06). # # WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres # REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the # postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB # PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO # database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it. # So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early # warning for a storage outage. The generic PVC-fill backstop is the separate # platform/monitoring/pvc-capacity-alerts.yaml. # # WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, # so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the # controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and # platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the # operator's Prometheus select these rules (ruleSelector). It IS referenced from # platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't" # failure this repo keeps hitting. # # WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only, # 2026-09-06). CNPG's exporter publishes TWO independent families: # # * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status # and the backup catalogue). Present for EVERY instance. # * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED. # # Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db # exporter cannot log in to its own database: # FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter] # so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all. # An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND # to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition # itself so the blind spot cannot recur unnoticed. # # LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not): # cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3 # dev: 0, 0, 0 (min over 24h also 0) # cnpg_collector_first_recoverability_point prod: 0, 0, 0 # dev: 1788018752 / 1788018762 / 1788018770 # cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old # cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet) # cnpg_last_error prod+dev: nextcloud-db = 1 # # SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate. # A database heading for disk-full deserves a warning with weeks of lead time, and this repo already # has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in # platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually # running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml — # which fires only when an outage is genuinely imminent. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-cnpg namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.cnpg rules: # --------------------------------------------------------------------------------------- # THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within # the hour instead of 38 hours later. # # pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet # archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within # seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck # segment is 16 MiB that Postgres may not recycle. # # THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the # condition to hold at EVERY evaluation for the full hour, so a write burst that transiently # queues a segment resets the timer and never notifies; but a genuinely stuck archiver never # returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3). # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its # backlog sits at 3 because it writes almost nothing. - alert: CNPGWALArchiveBacklog expr: | max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 for: 1h labels: severity: warning annotations: summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)" description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" # --------------------------------------------------------------------------------------- # The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in # time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint # is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can # be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires), # and an old base backup can exist while archiving has just broken (only the other fires). # # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold # start quiet while still catching a bootstrap that never produced a recoverability point. - alert: CNPGNoRecoverabilityPoint expr: | max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0 for: 6h labels: severity: warning annotations: summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}" description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" # --------------------------------------------------------------------------------------- # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in # platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these # databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be # ~57 years and always true). 36h = one missed daily run plus slack. - alert: CNPGNoRecentBaseBackup expr: | (time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0 for: 2h labels: severity: warning annotations: summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}" description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed" # --------------------------------------------------------------------------------------- # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for, # which means checkpoints are no longer reclaiming space — the mechanism that actually fills # the PVC. Both series come from the same metric under different `value` labels, so each side # is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching # label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on # both clusters), `count` = segments currently in pg_wal. # # This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to: # CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough # to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks — # at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC # runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is # therefore NOT usable as a signal. - alert: CNPGWALVolumeNotRecycling expr: | max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) for: 2h labels: severity: warning annotations: summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)" description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth" # --------------------------------------------------------------------------------------- # THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user # queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while # `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health. # # THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live # 2026-09-06: `cnpg_errors_total` carries # pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter # database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) # i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all. # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the # permanently-firing `critical` this repo has been burned by. - alert: CNPGMetricsCollectorErroring expr: | max by (namespace, pod) (cnpg_last_error) == 1 for: 6h labels: severity: warning annotations: summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}" description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"