jakman85andClaude Opus 5
bacb4c61ee
CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`. On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register kind Backup, so the bare short name resolves to Longhorn's empty set and prints 'No resources found' while CNPG backups exist. An on-call following the alert that fires BECAUSE backups look stale would have been told, with authority, that no backups exist at all — and might re-bootstrap a database over good ones. Fully qualify it. Co-Authored-By: Claude Opus 5 <[email protected]>
268 lines
22 KiB
YAML
268 lines
22 KiB
YAML
---
|
|
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
|
|
#
|
|
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
|
|
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
|
|
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
|
|
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
|
|
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
|
|
#
|
|
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
|
|
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
|
|
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
|
|
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
|
|
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
|
|
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
|
|
# warning for a storage outage. The generic PVC-fill backstop is the separate
|
|
# platform/monitoring/pvc-capacity-alerts.yaml.
|
|
#
|
|
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
|
|
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
|
|
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
|
|
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
|
|
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
|
|
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
|
|
# failure this repo keeps hitting.
|
|
#
|
|
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
|
|
# 2026-09-06). CNPG's exporter publishes TWO independent families:
|
|
#
|
|
# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
|
|
# and the backup catalogue). Present for EVERY instance.
|
|
# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
|
|
#
|
|
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
|
|
# exporter cannot log in to its own database:
|
|
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter]
|
|
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
|
|
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
|
|
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
|
|
# itself so the blind spot cannot recur unnoticed.
|
|
#
|
|
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
|
|
# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3
|
|
# dev: 0, 0, 0 (min over 24h also 0)
|
|
# cnpg_collector_first_recoverability_point prod: 0, 0, 0
|
|
# dev: 1788018752 / 1788018762 / 1788018770
|
|
# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old
|
|
# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet)
|
|
# cnpg_last_error prod+dev: nextcloud-db = 1
|
|
#
|
|
# AGGREGATION SCOPE — WHICH RULES MAY KEY ON `pod`, AND WHY (incident, 2026-09-12). The first cut of
|
|
# this file grouped EVERY rule `by (namespace, pod)`. That is correct only for metrics every instance
|
|
# reports independently, and WRONG for the two backup-catalogue metrics, which ONLY THE PRIMARY
|
|
# populates. Verified live on prod 2026-09-12 (`cnpg_collector_first_recoverability_point`, raw):
|
|
# keycloak-db-2 = 1788764922 nextcloud-db-2 = 1788764955 semaphore-db-2 = 1788764954 <- PRIMARIES
|
|
# keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 = 0 <- REPLICAS
|
|
# Since the Clusters went to `instances: 3`, `CNPGNoRecoverabilityPoint` was therefore PERMANENTLY
|
|
# FIRING on all six replicas while all three Clusters reported firstRecoverabilityPoint=2026-09-07,
|
|
# lastSuccessfulBackup within hours, ContinuousArchiving=True and LastBackupSucceeded=True. Backups
|
|
# were fine; the alert was noise, and noise on a DR alarm is worse than no alarm at all — a REAL loss
|
|
# of recoverability would have arrived as a seventh identical email nobody reads.
|
|
#
|
|
# THE RULE OF THUMB, applied as an audit to every rule below:
|
|
# * `== 0` / staleness shapes ("nothing reports a value") MUST aggregate `pod` AWAY, or every pod
|
|
# that structurally never reports the value fires forever.
|
|
# * `> 0` shapes ("some pod reports a bad value") are SAFE per-pod: a pod sitting at 0 simply does
|
|
# not fire, and keeping `pod` names the instance to go look at.
|
|
#
|
|
# THE GROUPING LABEL IS `job`, NOT `namespace`. The metric carries NO `cnpg.io/cluster` label — its
|
|
# full label set on prod is {container, endpoint, instance, job, namespace, pod} — but the CNPG
|
|
# operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace,
|
|
# so prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
|
|
# `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. That identifies the CNPG CLUSTER
|
|
# uniquely. Grouping by `namespace` alone happens to work today (one Cluster per namespace, verified)
|
|
# but would let two Clusters in one namespace MASK each other — the healthy one's non-zero value would
|
|
# satisfy `max` for both. `job` cannot do that, so `job` is what the DR rules group by.
|
|
#
|
|
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
|
|
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
|
|
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
|
|
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
|
|
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
|
|
# which fires only when an outage is genuinely imminent.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-cnpg
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.cnpg
|
|
rules:
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
|
|
# the hour instead of 38 hours later.
|
|
#
|
|
# pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
|
|
# archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
|
|
# seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
|
|
# segment is 16 MiB that Postgres may not recycle.
|
|
#
|
|
# THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
|
|
# condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
|
|
# queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
|
|
# returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
|
|
# 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
|
|
# A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
|
|
# backlog sits at 3 because it writes almost nothing.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately. Every instance has its own pg_wal on its
|
|
# own PVC and every instance reports this metric (verified: all 9 prod pods publish
|
|
# `{value="ready"}` = 0 today), and this is a `> 0` shape — a replica sitting at 0 never fires,
|
|
# so there is no permanent-firing hazard and no masking (`> 0` fires if ANY pod is bad). `pod`
|
|
# earns its place here: it names the instance whose archiver is stuck.
|
|
- alert: CNPGWALArchiveBacklog
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
|
|
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
|
|
# time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
|
|
# is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
|
|
# be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
|
|
# and an old base backup can exist while archiving has just broken (only the other fires).
|
|
#
|
|
# `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
|
|
# (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
|
|
# start quiet while still catching a bootstrap that never produced a recoverability point.
|
|
#
|
|
# SCOPE — `by (namespace, job)`, NOT `by (namespace, pod)`. THIS IS THE FIX FOR THE 2026-09-12
|
|
# permanently-firing incident (see the AGGREGATION SCOPE block in the header). The recoverability
|
|
# point is a property of the CLUSTER's backup catalogue, which only the PRIMARY reads; a replica
|
|
# reports a hard 0 forever. Grouped by `pod`, each replica was evaluated on its own and this
|
|
# `== 0` fired permanently on all six of them. Aggregating `pod` away makes `max` take the
|
|
# PRIMARY's non-zero value as the cluster's answer, so the alert says what its own summary always
|
|
# claimed to say — "this Postgres CLUSTER has no recoverability point" — and fires only when NO
|
|
# instance of that Cluster reports one.
|
|
#
|
|
# PROVEN LIVE, read-only, prod, 2026-09-12 (promtool query instant):
|
|
# OLD max by (namespace, pod) (...) == 0 -> 6 series: keycloak-db-1/-3, nextcloud-db-1/-3,
|
|
# semaphore-db-1/-3 (and 6 ALERTS firing)
|
|
# NEW max by (namespace, job) (...) == 0 -> EMPTY
|
|
# and it is not merely vacuous — forcing the metric to 0 for one cluster's primary only
|
|
# (`* on(pod) group_left() ...{pod="keycloak-db-2"}`) yields exactly one series,
|
|
# {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster.
|
|
#
|
|
# A switchover cannot flap this: while the primary restarts, `max` briefly sees only replicas' 0,
|
|
# but `for: 6h` requires the condition at EVERY evaluation for six hours and a promotion takes
|
|
# seconds. Annotations key on `job` (= "<namespace>/<cluster>") because `pod` no longer exists on
|
|
# these series — a `{{ $labels.pod }}` left behind here would render as an empty string.
|
|
- alert: CNPGNoRecoverabilityPoint
|
|
expr: |
|
|
max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG cluster has NO recoverability point: {{ $labels.job }}"
|
|
description: "PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) reports firstRecoverabilityPoint = null on EVERY instance for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). This is a per-CLUSTER statement: the primary owns the backup catalogue, so a healthy cluster clears it even though its replicas report 0. Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
|
|
# platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
|
|
# databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
|
|
# backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
|
|
# CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
|
|
# ~57 years and always true). 36h = one missed daily run plus slack.
|
|
#
|
|
# SCOPE — `by (namespace, job)`, fixed alongside CNPGNoRecoverabilityPoint (2026-09-12).
|
|
# `cnpg_collector_last_available_backup_timestamp` has the SAME primary-only defect, measured on
|
|
# prod the same day: keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and
|
|
# ALL SIX REPLICAS = 0, i.e. `time() - 0` = 497020h ≈ 56.7 YEARS. Unlike the rule above this one
|
|
# was not firing, because the `> 0` guard was load-bearing — it was the single thing standing
|
|
# between this rule and six more permanent false positives, which is far too thin a margin for a
|
|
# DR alarm. Aggregating `pod` away removes the dependence: `max` picks the PRIMARY's timestamp as
|
|
# the cluster's answer (a demoted primary drops to 0 and can never win the `max`, and a fresher
|
|
# backup always beats a staler one), the `> 0` guard then means exactly "this cluster has had at
|
|
# least one backup", and the result is a real per-cluster staleness statement.
|
|
#
|
|
# PROVEN LIVE, prod, 2026-09-12: OLD -> empty, NEW -> empty (so the fix changes no live verdict),
|
|
# while the per-CLUSTER ages the new aggregation actually evaluates are keycloak 3.09h /
|
|
# nextcloud 2.09h / semaphore 0.09h — the primaries' real values, all far inside 36h.
|
|
- alert: CNPGNoRecentBaseBackup
|
|
expr: |
|
|
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
|
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "No recent CNPG base backup: {{ $labels.job }}"
|
|
description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
|
|
# ---------------------------------------------------------------------------------------
|
|
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
|
|
# which means checkpoints are no longer reclaiming space — the mechanism that actually fills
|
|
# the PVC. Both series come from the same metric under different `value` labels, so each side
|
|
# is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
|
|
# label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
|
|
# both clusters), `count` = segments currently in pg_wal.
|
|
#
|
|
# This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
|
|
# CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
|
|
# to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
|
|
# at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
|
|
# runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
|
|
# any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
|
|
# `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
|
|
# therefore NOT usable as a signal.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately — and here it is REQUIRED. Every instance
|
|
# reports its own count/max (verified on all 9 prod pods: count 16-37 against max 64), pg_wal is
|
|
# per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself:
|
|
# aggregating `pod` away would compare one instance's count with another's max. `>` is also a
|
|
# per-pod shape, so no instance can sit permanently true.
|
|
- alert: CNPGWALVolumeNotRecycling
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
|
|
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
|
|
description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
|
|
# queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
|
|
# `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
|
|
#
|
|
# THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
|
|
# 2026-09-06: `cnpg_errors_total` carries
|
|
# pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
|
|
# database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
|
# i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
|
|
# Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
|
|
# restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
|
|
# permanently-firing `critical` this repo has been burned by.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, UNCHANGED — this rule's meaning and firing set are
|
|
# deliberately left exactly as they were. `cnpg_last_error` is genuinely per-instance: each
|
|
# instance's own exporter reports its own SQL-collection health, and prod proves the metric
|
|
# discriminates rather than tracking role — all THREE nextcloud-db pods (primary -2 included)
|
|
# report 1 while all six keycloak/semaphore pods report 0. So there is no primary-only defect to
|
|
# fix, and per-pod is the right granularity: the remedy is a GRANT applied per database, and you
|
|
# want to see it clear on every instance. It is a TRUE POSITIVE right now (nextcloud-db, the
|
|
# 42501 CONNECT grant) and is owned by separate work; nothing here silences or rescopes it.
|
|
- alert: CNPGMetricsCollectorErroring
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_last_error) == 1
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|