home-network-k3s-impl/runbooks/cnpg-backup-missed.md

cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster

Alert: CNPGNoRecentBaseBackup (warning, 2h) File: platform/cnpg/alerts.yaml, group platform-health.cnpg Expression:

(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0

What this means here

The newest completed base backup for one Postgres cluster is over 36 hours old — one missed daily run plus slack. The daily ScheduledBackup has silently stopped producing backups, so the PITR window is ageing out against the 30d retention policy.

Nothing else covers this. The Velero rules in platform/monitoring/alerts.yaml watch Velero's own schedules, and Velero does not back up these databases — CNPG owns their DR path, because a Velero PVC snapshot of a live Postgres is not a backup. So /runbooks/backup-missed being clear says nothing about the databases, and vice versa.

The three Clusters and their staggered base-backup times: keycloak 01:00, nextcloud 02:00, semaphore 04:00 (Velero's own daily runs at 03:00, deliberately between them).

Two expression details that matter when you read the alert

The > 0 guard is load-bearing. A cluster that has never had a backup reports 0, and time() - 0 is about 57 years — always true. Without the guard this rule would double-notify with CNPGNoRecoverabilityPoint, which is the correct alert for "never had a backup". So this rule means specifically: this cluster has had at least one backup, and the newest one is stale.

It is aggregated by (namespace, job), not by (namespace, pod) — fixed 2026-09-12 alongside CNPGNoRecoverabilityPoint. cnpg_collector_last_available_backup_timestamp has the same primary-only defect: measured on prod the same day, keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and all six replicas = 0, i.e. time() - 0 ≈ 56.7 years. Unlike its sibling this rule was not firing — because the > 0 guard was the single thing standing between it and six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating pod away removes the dependence: max picks the primary's timestamp as the cluster's answer (a demoted primary drops to 0 and can never win the max; a fresher backup always beats a staler one). job is the grouping label because the metric carries no cnpg.io/cluster label — the operator creates one PodMonitor per Cluster, so prometheus-operator derives job="<namespace>/<cluster>": keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db.

Severity is warning on purpose — this repo has been burned by a permanently-firing critical (the VPNTunnelDown post-mortem), and "noise on a DR alarm is worse than no alarm at all".

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and defaults its current-context to k3s-dev, so a command without --context silently answers about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.) Do not run two dev.sh invocations at once — they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Fully-qualified names, always. kubectl get backup -A prints "No resources found" on this cluster even with 27 CNPG backups present: three CRDs register the kind Backup (longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1) and the bare short name resolves to Longhorn's, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone" from that and acting on it is how a CLI quirk becomes data loss.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'

Are the schedules there and enabled?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'

The Cluster's own verdict (and the condition that tells you whether archiving is also broken):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'

Why the last attempt failed — the barman error is in the primary's logs:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
   kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'

Likely causes, most likely first

1. The ScheduledBackup is suspended, missing, or unapplied

Check spec.suspend, and whether the object exists at all for that Cluster. A ScheduledBackup that was never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier that owns it is Ready (/runbooks/flux-reconcile) and not suspended (/runbooks/flux-suspended).

2. The object store is rejecting the write

Same root cause family as /runbooks/cnpg-archiving: endpoint, path, region or credentials. Live prod values to expect are CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman, S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com, S3_BACKUP_REGION = auto, BACKUP_CREDENTIALS_SUFFIX = -prod. If ContinuousArchiving is also False, fix that first — a cluster whose archiver is broken has a disk-full countdown running, which is more urgent than a stale base backup.

3. A failed run that nobody noticed

There is precedent: one historical failure, semaphore-db-daily-20260905130918. A single failed run does not clear itself; the next night's run should. If two consecutive nights fail, the cause is configuration, not luck.

4. The primary changed and the new primary cannot write

The backup catalogue is the primary's job. After a switchover, a credential or path problem scoped to one instance shows up here. Check which pod is currentPrimary and read that pod's logs, not another's.

5. Node or storage trouble during the backup window

Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read /runbooks/node-io.

6. The metric is missing, not the backup

If cnpg_collector_last_available_backup_timestamp has no series, the rule cannot fire. Note that a related blind spot is real and separate: the CNPG SQL exporter can fail while up stays 1 — see /runbooks/cnpg-exporter.

When to escalate or stop

• A stale base backup narrows, but does not eliminate, recoverability — provided WAL archiving is healthy, PITR still works from the older base. That is why this is a warning. If both this and CNPGNoRecoverabilityPoint / CNPGWALArchiveBacklog are lit, DR for that database is effectively gone: escalate.
• Never delete WAL, .ready markers, or anything under a Postgres data directory.
• Do not delete old CNPG Backup objects to tidy up. Retention is 30d and the operator manages it; deleting a Backup object can orphan objects in R2.
• Do not trigger a base backup as a reflex on a cluster whose archiver is failing — it adds write load to a volume that may already be filling.
• Be honest about restore: the CNPG restore path from R2 has not been rehearsed on this cluster (dr-restore is authored and never executed). Free R2 egress makes a rehearsal cheap; it is outstanding. A restore is an operator operation.

---

Source of truth: runbooks/cnpg-backup-missed.md. Index: /runbooks/ Records: platform/cnpg/alerts.yaml (the 2026-09-12 aggregation-scope fix with its live measurements, the > 0 guard rationale, and the staggered schedule).