jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
183 lines
14 KiB
YAML
183 lines
14 KiB
YAML
---
|
|
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
|
|
#
|
|
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
|
|
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
|
|
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
|
|
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
|
|
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
|
|
#
|
|
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
|
|
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
|
|
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
|
|
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
|
|
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
|
|
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
|
|
# warning for a storage outage. The generic PVC-fill backstop is the separate
|
|
# platform/monitoring/pvc-capacity-alerts.yaml.
|
|
#
|
|
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
|
|
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
|
|
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
|
|
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
|
|
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
|
|
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
|
|
# failure this repo keeps hitting.
|
|
#
|
|
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
|
|
# 2026-09-06). CNPG's exporter publishes TWO independent families:
|
|
#
|
|
# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
|
|
# and the backup catalogue). Present for EVERY instance.
|
|
# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
|
|
#
|
|
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
|
|
# exporter cannot log in to its own database:
|
|
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter]
|
|
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
|
|
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
|
|
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
|
|
# itself so the blind spot cannot recur unnoticed.
|
|
#
|
|
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
|
|
# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3
|
|
# dev: 0, 0, 0 (min over 24h also 0)
|
|
# cnpg_collector_first_recoverability_point prod: 0, 0, 0
|
|
# dev: 1788018752 / 1788018762 / 1788018770
|
|
# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old
|
|
# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet)
|
|
# cnpg_last_error prod+dev: nextcloud-db = 1
|
|
#
|
|
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
|
|
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
|
|
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
|
|
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
|
|
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
|
|
# which fires only when an outage is genuinely imminent.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-cnpg
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.cnpg
|
|
rules:
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
|
|
# the hour instead of 38 hours later.
|
|
#
|
|
# pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
|
|
# archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
|
|
# seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
|
|
# segment is 16 MiB that Postgres may not recycle.
|
|
#
|
|
# THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
|
|
# condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
|
|
# queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
|
|
# returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
|
|
# 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
|
|
# A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
|
|
# backlog sits at 3 because it writes almost nothing.
|
|
- alert: CNPGWALArchiveBacklog
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
|
|
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
|
|
# time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
|
|
# is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
|
|
# be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
|
|
# and an old base backup can exist while archiving has just broken (only the other fires).
|
|
#
|
|
# `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
|
|
# (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
|
|
# start quiet while still catching a bootstrap that never produced a recoverability point.
|
|
- alert: CNPGNoRecoverabilityPoint
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
|
# ---------------------------------------------------------------------------------------
|
|
# The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
|
|
# platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
|
|
# databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
|
|
# backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
|
|
# CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
|
|
# ~57 years and always true). 36h = one missed daily run plus slack.
|
|
- alert: CNPGNoRecentBaseBackup
|
|
expr: |
|
|
(time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
|
and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
|
|
# ---------------------------------------------------------------------------------------
|
|
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
|
|
# which means checkpoints are no longer reclaiming space — the mechanism that actually fills
|
|
# the PVC. Both series come from the same metric under different `value` labels, so each side
|
|
# is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
|
|
# label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
|
|
# both clusters), `count` = segments currently in pg_wal.
|
|
#
|
|
# This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
|
|
# CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
|
|
# to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
|
|
# at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
|
|
# runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
|
|
# any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
|
|
# `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
|
|
# therefore NOT usable as a signal.
|
|
- alert: CNPGWALVolumeNotRecycling
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
|
|
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
|
|
description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
|
|
# queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
|
|
# `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
|
|
#
|
|
# THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
|
|
# 2026-09-06: `cnpg_errors_total` carries
|
|
# pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
|
|
# database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
|
# i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
|
|
# Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
|
|
# restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
|
|
# permanently-firing `critical` this repo has been burned by.
|
|
- alert: CNPGMetricsCollectorErroring
|
|
expr: |
|
|
max by (namespace, pod) (cnpg_last_error) == 1
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|