# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster **Alert:** `CNPGNoRecentBaseBackup` (warning, 2h) **File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` **Expression:** ``` (time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0 ``` ## What this means here The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is ageing out against the `30d` retention policy. **Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing about the databases, and vice versa. The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**, **semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them). ## Two expression details that matter when you read the alert **The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and `time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with `CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means specifically: *this cluster has had at least one backup, and the newest one is stale.* **It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside `CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h, `semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary drops to 0 and can never win the `max`; a fresher backup always beats a staler one). `job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator creates one PodMonitor per Cluster, so prometheus-operator derives `job="/"`: `keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. **Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands **Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup` (`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone" from that and acting on it is how a CLI quirk becomes data loss. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20' ``` Are the schedules there and enabled? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime' ``` The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' ``` Why the last attempt failed — the barman error is in the primary's logs: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \ kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20' ``` ## Likely causes, most likely first ### 1. The ScheduledBackup is suspended, missing, or unapplied Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`). ### 2. The object store is rejecting the write Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, `S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, `S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is more urgent than a stale base backup. ### 3. A failed run that nobody noticed There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does not clear itself; the next night's run should. If two consecutive nights fail, the cause is configuration, not luck. ### 4. The primary changed and the new primary cannot write The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's. ### 5. Node or storage trouble during the backup window Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`. ### 6. The metric is missing, not the backup If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a **related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see `/runbooks/cnpg-exporter`. ## When to escalate or stop - **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is healthy, PITR still works from the older base. That is why this is a warning. If **both** this and `CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively gone: escalate. - **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.** - **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it; deleting a Backup object can orphan objects in R2. - **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write load to a volume that may already be filling. - **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster (`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is outstanding. A restore is an operator operation. --- Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/` Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements, the `> 0` guard rationale, and the staggered schedule).