From bacb4c61eefef701a2592bac3420c86a7d835477 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 23:21:49 -0500 Subject: [PATCH] fix(cnpg): the alert told the on-call to run the command that lies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`. On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register kind Backup, so the bare short name resolves to Longhorn's empty set and prints 'No resources found' while CNPG backups exist. An on-call following the alert that fires BECAUSE backups look stale would have been told, with authority, that no backups exist at all — and might re-bootstrap a database over good ones. Fully qualify it. Co-Authored-By: Claude Opus 5 --- platform/cnpg/alerts.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform/cnpg/alerts.yaml b/platform/cnpg/alerts.yaml index b177c95..3c0621d 100644 --- a/platform/cnpg/alerts.yaml +++ b/platform/cnpg/alerts.yaml @@ -198,7 +198,7 @@ spec: severity: warning annotations: summary: "No recent CNPG base backup: {{ $labels.job }}" - description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backup -n {{ $labels.namespace }}`." + description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed" # --------------------------------------------------------------------------------------- # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,