Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

fix(cnpg): the alert told the on-call to run the command that lies

Browse Source 
CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`.
On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register
kind Backup, so the bare short name resolves to Longhorn's empty set and
prints 'No resources found' while CNPG backups exist. An on-call following
the alert that fires BECAUSE backups look stale would have been told, with
authority, that no backups exist at all — and might re-bootstrap a database
over good ones. Fully qualify it.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 23:28:12 -05:00
1 parent 21491f901b
commit bacb4c61ee
1 file changed
+1 -1
Download Patch File Download Diff File Expand all files Collapse all files
platform/cnpg/alerts.yaml
+1 -1
View File
@@ -198,7 +198,7 @@ spec:
severity: warning
annotations:
summary: "No recent CNPG base backup: {{ $labels.job }}"
description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backup -n {{ $labels.namespace }}`."
description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
# ---------------------------------------------------------------------------------------
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,