home-network-k3s-impl/runbooks/cnpg-backup-missed.md

# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster

**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:**
```
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
```

## What this means here

The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run
plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is
ageing out against the `30d` retention policy.

**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own
schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero
PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing
about the databases, and vice versa.

The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**,
**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them).

## Two expression details that matter when you read the alert

**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and
`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with
`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means
specifically: *this cluster has had at least one backup, and the newest one is stale.*

**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside
`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only
defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h,
`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its
sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and
six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away
removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary
drops to 0 and can never win the `max`; a fresher backup always beats a staler one).
`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator
creates one PodMonitor per Cluster, so prometheus-operator derives `job="<namespace>/<cluster>"`:
`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`.

**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical`
(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this
cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup`
(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to
**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone"
from that and acting on it is how a CLI quirk becomes data loss.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'
```

Are the schedules there and enabled?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'
```

The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```

Why the last attempt failed — the barman error is in the primary's logs:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
   kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'
```

## Likely causes, most likely first

### 1. The ScheduledBackup is suspended, missing, or unapplied
Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was
never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier
that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`).

### 2. The object store is rejecting the write
Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod
values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also
False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is
more urgent than a stale base backup.

### 3. A failed run that nobody noticed
There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does
not clear itself; the next night's run should. If two consecutive nights fail, the cause is
configuration, not luck.

### 4. The primary changed and the new primary cannot write
The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to
one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's.

### 5. Node or storage trouble during the backup window
Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`.

### 6. The metric is missing, not the backup
If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a
**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see
`/runbooks/cnpg-exporter`.

## When to escalate or stop

- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is
  healthy, PITR still works from the older base. That is why this is a warning. If **both** this and
  `CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively
  gone: escalate.
- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.**
- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it;
  deleting a Backup object can orphan objects in R2.
- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write
  load to a volume that may already be filling.
- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster
  (`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is
  outstanding. A restore is an operator operation.

---
Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements,
the `> 0` guard rationale, and the staggered schedule).