jakman85andClaude Opus 5
505083fded
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
381 lines
31 KiB
YAML
381 lines
31 KiB
YAML
---
|
|
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
|
|
#
|
|
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
|
|
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
|
|
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
|
|
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
|
|
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
|
|
#
|
|
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
|
|
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
|
|
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
|
|
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
|
|
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
|
|
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
|
|
# warning for a storage outage. The generic PVC-fill backstop is the separate
|
|
# platform/monitoring/pvc-capacity-alerts.yaml.
|
|
#
|
|
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
|
|
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
|
|
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
|
|
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
|
|
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
|
|
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
|
|
# failure this repo keeps hitting.
|
|
#
|
|
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
|
|
# 2026-09-06). CNPG's exporter publishes TWO independent families:
|
|
#
|
|
# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
|
|
# and the backup catalogue). Present for EVERY instance.
|
|
# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
|
|
#
|
|
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
|
|
# exporter cannot log in to its own database:
|
|
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter]
|
|
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
|
|
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
|
|
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
|
|
# itself so the blind spot cannot recur unnoticed.
|
|
#
|
|
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
|
|
# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3
|
|
# dev: 0, 0, 0 (min over 24h also 0)
|
|
# cnpg_collector_first_recoverability_point prod: 0, 0, 0
|
|
# dev: 1788018752 / 1788018762 / 1788018770
|
|
# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old
|
|
# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet)
|
|
# cnpg_last_error prod+dev: nextcloud-db = 1
|
|
#
|
|
# AGGREGATION SCOPE — WHICH RULES MAY KEY ON `pod`, AND WHY (incident, 2026-09-12). The first cut of
|
|
# this file grouped EVERY rule `by (namespace, pod)`. That is correct only for metrics every instance
|
|
# reports independently, and WRONG for the two backup-catalogue metrics, which ONLY THE PRIMARY
|
|
# populates. Verified live on prod 2026-09-12 (`cnpg_collector_first_recoverability_point`, raw):
|
|
# keycloak-db-2 = 1788764922 nextcloud-db-2 = 1788764955 semaphore-db-2 = 1788764954 <- PRIMARIES
|
|
# keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 = 0 <- REPLICAS
|
|
# Since the Clusters went to `instances: 3`, `CNPGNoRecoverabilityPoint` was therefore PERMANENTLY
|
|
# FIRING on all six replicas while all three Clusters reported firstRecoverabilityPoint=2026-09-07,
|
|
# lastSuccessfulBackup within hours, ContinuousArchiving=True and LastBackupSucceeded=True. Backups
|
|
# were fine; the alert was noise, and noise on a DR alarm is worse than no alarm at all — a REAL loss
|
|
# of recoverability would have arrived as a seventh identical email nobody reads.
|
|
#
|
|
# THE RULE OF THUMB, applied as an audit to every rule below:
|
|
# * `== 0` / staleness shapes ("nothing reports a value") MUST aggregate `pod` AWAY, or every pod
|
|
# that structurally never reports the value fires forever.
|
|
# * `> 0` shapes ("some pod reports a bad value") are SAFE per-pod: a pod sitting at 0 simply does
|
|
# not fire, and keeping `pod` names the instance to go look at.
|
|
#
|
|
# THE GROUPING LABEL IS `job`, NOT `namespace`. The metric carries NO `cnpg.io/cluster` label — its
|
|
# full label set on prod is {container, endpoint, instance, job, namespace, pod} — but the CNPG
|
|
# operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace,
|
|
# so prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
|
|
# `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. That identifies the CNPG CLUSTER
|
|
# uniquely. Grouping by `namespace` alone happens to work today (one Cluster per namespace, verified)
|
|
# but would let two Clusters in one namespace MASK each other — the healthy one's non-zero value would
|
|
# satisfy `max` for both. `job` cannot do that, so `job` is what the DR rules group by.
|
|
#
|
|
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
|
|
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
|
|
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
|
|
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
|
|
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
|
|
# which fires only when an outage is genuinely imminent.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-cnpg
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.cnpg
|
|
rules:
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
|
|
# the hour instead of 38 hours later.
|
|
#
|
|
# pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
|
|
# archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
|
|
# seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
|
|
# segment is 16 MiB that Postgres may not recycle.
|
|
#
|
|
# THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
|
|
# condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
|
|
# queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
|
|
# returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
|
|
# 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
|
|
# A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
|
|
# backlog sits at 3 because it writes almost nothing.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately. Every instance has its own pg_wal on its
|
|
# own PVC and every instance reports this metric (verified: all 9 prod pods publish
|
|
# `{value="ready"}` = 0 today), and this is a `> 0` shape — a replica sitting at 0 never fires,
|
|
# so there is no permanent-firing hazard and no masking (`> 0` fires if ANY pod is bad). `pod`
|
|
# earns its place here: it names the instance whose archiver is stuck.
|
|
- alert: CNPGWALArchiveBacklog
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
|
|
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
|
|
# time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
|
|
# is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
|
|
# be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
|
|
# and an old base backup can exist while archiving has just broken (only the other fires).
|
|
#
|
|
# `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
|
|
# (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
|
|
# start quiet while still catching a bootstrap that never produced a recoverability point.
|
|
#
|
|
# SCOPE — `by (namespace, job)`, NOT `by (namespace, pod)`. THIS IS THE FIX FOR THE 2026-09-12
|
|
# permanently-firing incident (see the AGGREGATION SCOPE block in the header). The recoverability
|
|
# point is a property of the CLUSTER's backup catalogue, which only the PRIMARY reads; a replica
|
|
# reports a hard 0 forever. Grouped by `pod`, each replica was evaluated on its own and this
|
|
# `== 0` fired permanently on all six of them. Aggregating `pod` away makes `max` take the
|
|
# PRIMARY's non-zero value as the cluster's answer, so the alert says what its own summary always
|
|
# claimed to say — "this Postgres CLUSTER has no recoverability point" — and fires only when NO
|
|
# instance of that Cluster reports one.
|
|
#
|
|
# PROVEN LIVE, read-only, prod, 2026-09-12 (promtool query instant):
|
|
# OLD max by (namespace, pod) (...) == 0 -> 6 series: keycloak-db-1/-3, nextcloud-db-1/-3,
|
|
# semaphore-db-1/-3 (and 6 ALERTS firing)
|
|
# NEW max by (namespace, job) (...) == 0 -> EMPTY
|
|
# and it is not merely vacuous — forcing the metric to 0 for one cluster's primary only
|
|
# (`* on(pod) group_left() ...{pod="keycloak-db-2"}`) yields exactly one series,
|
|
# {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster.
|
|
#
|
|
# A switchover cannot flap this: while the primary restarts, `max` briefly sees only replicas' 0,
|
|
# but `for: 6h` requires the condition at EVERY evaluation for six hours and a promotion takes
|
|
# seconds. Annotations key on `job` (= "<namespace>/<cluster>") because `pod` no longer exists on
|
|
# these series — a `{{ $labels.pod }}` left behind here would render as an empty string.
|
|
- alert: CNPGNoRecoverabilityPoint
|
|
expr: |
|
|
max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG cluster has NO recoverability point: {{ $labels.job }}"
|
|
description: "PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) reports firstRecoverabilityPoint = null on EVERY instance for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). This is a per-CLUSTER statement: the primary owns the backup catalogue, so a healthy cluster clears it even though its replicas report 0. Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
|
|
# platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
|
|
# databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
|
|
# backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
|
|
# CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
|
|
# ~57 years and always true). 36h = one missed daily run plus slack.
|
|
#
|
|
# SCOPE — `by (namespace, job)`, fixed alongside CNPGNoRecoverabilityPoint (2026-09-12).
|
|
# `cnpg_collector_last_available_backup_timestamp` has the SAME primary-only defect, measured on
|
|
# prod the same day: keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and
|
|
# ALL SIX REPLICAS = 0, i.e. `time() - 0` = 497020h ≈ 56.7 YEARS. Unlike the rule above this one
|
|
# was not firing, because the `> 0` guard was load-bearing — it was the single thing standing
|
|
# between this rule and six more permanent false positives, which is far too thin a margin for a
|
|
# DR alarm. Aggregating `pod` away removes the dependence: `max` picks the PRIMARY's timestamp as
|
|
# the cluster's answer (a demoted primary drops to 0 and can never win the `max`, and a fresher
|
|
# backup always beats a staler one), the `> 0` guard then means exactly "this cluster has had at
|
|
# least one backup", and the result is a real per-cluster staleness statement.
|
|
#
|
|
# PROVEN LIVE, prod, 2026-09-12: OLD -> empty, NEW -> empty (so the fix changes no live verdict),
|
|
# while the per-CLUSTER ages the new aggregation actually evaluates are keycloak 3.09h /
|
|
# nextcloud 2.09h / semaphore 0.09h — the primaries' real values, all far inside 36h.
|
|
- alert: CNPGNoRecentBaseBackup
|
|
expr: |
|
|
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
|
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "No recent CNPG base backup: {{ $labels.job }}"
|
|
description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
|
|
# ---------------------------------------------------------------------------------------
|
|
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
|
|
# which means checkpoints are no longer reclaiming space — the mechanism that actually fills
|
|
# the PVC. Both series come from the same metric under different `value` labels, so each side
|
|
# is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
|
|
# label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
|
|
# both clusters), `count` = segments currently in pg_wal.
|
|
#
|
|
# This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
|
|
# CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
|
|
# to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
|
|
# at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
|
|
# runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
|
|
# any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
|
|
# `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
|
|
# therefore NOT usable as a signal.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately — and here it is REQUIRED. Every instance
|
|
# reports its own count/max (verified on all 9 prod pods: count 16-37 against max 64), pg_wal is
|
|
# per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself:
|
|
# aggregating `pod` away would compare one instance's count with another's max. `>` is also a
|
|
# per-pod shape, so no instance can sit permanently true.
|
|
- alert: CNPGWALVolumeNotRecycling
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
|
|
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
|
|
description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
|
|
# queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
|
|
# `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
|
|
#
|
|
# THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
|
|
# 2026-09-06: `cnpg_errors_total` carries
|
|
# pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
|
|
# database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
|
# i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
|
|
# Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
|
|
# restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
|
|
# permanently-firing `critical` this repo has been burned by.
|
|
#
|
|
# SCOPE AUDIT (2026-09-12): keeps `pod`, UNCHANGED — this rule's meaning and firing set are
|
|
# deliberately left exactly as they were. `cnpg_last_error` is genuinely per-instance: each
|
|
# instance's own exporter reports its own SQL-collection health, and prod proves the metric
|
|
# discriminates rather than tracking role — all THREE nextcloud-db pods (primary -2 included)
|
|
# report 1 while all six keycloak/semaphore pods report 0. So there is no primary-only defect to
|
|
# fix, and per-pod is the right granularity: the remedy is a GRANT applied per database, and you
|
|
# want to see it clear on every instance. It is a TRUE POSITIVE right now (nextcloud-db, the
|
|
# 42501 CONNECT grant) and is owned by separate work; nothing here silences or rescopes it.
|
|
- alert: CNPGMetricsCollectorErroring
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_last_error) == 1
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|
|
# ---------------------------------------------------------------------------------------
|
|
# BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
|
|
#
|
|
# WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
|
|
# The operator's original ask was "the database primaries are all on one node". `instances: 3`
|
|
# + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
|
|
# exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
|
|
# PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
|
|
# semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
|
|
# failovers — survivable, but the worst single-node blast radius the cluster can have.
|
|
#
|
|
# THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
|
|
# (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
|
|
# the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
|
|
# * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
|
# additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
|
|
# the cluster's pods". None is role-aware: there is no way to say "this constraint is for
|
|
# whichever pod is currently primary".
|
|
# * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
|
|
# Cluster that is the job podAntiAffinityType already does; it says nothing about how two
|
|
# INDEPENDENT Clusters' primaries relate.
|
|
# * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
|
|
# (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
|
|
# * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
|
|
# postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
|
|
# primary lands.
|
|
# Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
|
|
# scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
|
|
# imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
|
|
#
|
|
# AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
|
|
# cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
|
|
# (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
|
|
# RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
|
|
# (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
|
|
# constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
|
|
# actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
|
|
# excluding the node that holds another cluster's primary leaves the third instance
|
|
# UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
|
|
#
|
|
# SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
|
|
# status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
|
|
# rebalances today and drifts again on the next failover, node reboot or rolling update.
|
|
# A switchover that silently un-does itself is not a fix, so what is committed here is the
|
|
# part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
|
|
# a known, cheap runbook step instead of something discovered by hand months later.
|
|
#
|
|
# WHY THIS PROMQL. Neither family alone can answer the question:
|
|
# * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
|
|
# * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
|
|
# --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
|
|
# The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
|
|
# Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
|
|
# and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
|
|
# `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
|
|
# `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
|
|
# that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
|
|
# cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
|
|
# the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
|
|
# is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
|
|
#
|
|
# THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
|
|
# three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
|
|
# is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
|
|
# rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
|
|
# the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
|
|
#
|
|
# for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
|
|
# 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
|
|
# still reporting real drift the same hour. severity: warning, like every rule in this file:
|
|
# this is a resilience posture to correct at leisure, never a page.
|
|
#
|
|
# PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
|
|
# fire is decoration and one that cannot stop is fatigue:
|
|
# * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
|
|
# {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
|
|
# throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
|
|
# observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
|
|
# was deleted afterwards; it was never committed.
|
|
# * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
|
|
# empty vector, while the inner join still resolved all three primaries — semaphore-db-2
|
|
# on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
|
|
# the condition is false, NOT because the query went blind.
|
|
# * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
|
|
# ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
|
|
# version running here: co-located fires after 30m, one-per-node is silent, a NotReady
|
|
# node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
|
|
# Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
|
|
# readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
|
|
- alert: CNPGPrimariesCoLocated
|
|
expr: |
|
|
count by (node) (
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
) > 1
|
|
and on ()
|
|
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
|
|
description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"
|