jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
166 lines
10 KiB
Markdown
166 lines
10 KiB
Markdown
# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster
|
|
|
|
**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h)
|
|
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
|
|
**Expression:**
|
|
```
|
|
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
|
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
|
|
```
|
|
|
|
## What this means here
|
|
|
|
The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run
|
|
plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is
|
|
ageing out against the `30d` retention policy.
|
|
|
|
**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own
|
|
schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero
|
|
PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing
|
|
about the databases, and vice versa.
|
|
|
|
The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**,
|
|
**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them).
|
|
|
|
## Two expression details that matter when you read the alert
|
|
|
|
**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and
|
|
`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with
|
|
`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means
|
|
specifically: *this cluster has had at least one backup, and the newest one is stale.*
|
|
|
|
**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside
|
|
`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only
|
|
defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h,
|
|
`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its
|
|
sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and
|
|
six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away
|
|
removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary
|
|
drops to 0 and can never win the `max`; a fresher backup always beats a staler one).
|
|
`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator
|
|
creates one PodMonitor per Cluster, so prometheus-operator derives `job="<namespace>/<cluster>"`:
|
|
`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`.
|
|
|
|
**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical`
|
|
(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this
|
|
cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup`
|
|
(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to
|
|
**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone"
|
|
from that and acting on it is how a CLI quirk becomes data loss.
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'
|
|
```
|
|
|
|
Are the schedules there and enabled?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'
|
|
```
|
|
|
|
The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
|
|
```
|
|
|
|
Why the last attempt failed — the barman error is in the primary's logs:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
|
|
kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The ScheduledBackup is suspended, missing, or unapplied
|
|
Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was
|
|
never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier
|
|
that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`).
|
|
|
|
### 2. The object store is rejecting the write
|
|
Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod
|
|
values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
|
|
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
|
|
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also
|
|
False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is
|
|
more urgent than a stale base backup.
|
|
|
|
### 3. A failed run that nobody noticed
|
|
There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does
|
|
not clear itself; the next night's run should. If two consecutive nights fail, the cause is
|
|
configuration, not luck.
|
|
|
|
### 4. The primary changed and the new primary cannot write
|
|
The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to
|
|
one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's.
|
|
|
|
### 5. Node or storage trouble during the backup window
|
|
Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`.
|
|
|
|
### 6. The metric is missing, not the backup
|
|
If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a
|
|
**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see
|
|
`/runbooks/cnpg-exporter`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is
|
|
healthy, PITR still works from the older base. That is why this is a warning. If **both** this and
|
|
`CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively
|
|
gone: escalate.
|
|
- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.**
|
|
- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it;
|
|
deleting a Backup object can orphan objects in R2.
|
|
- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write
|
|
load to a volume that may already be filling.
|
|
- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster
|
|
(`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is
|
|
outstanding. A restore is an operator operation.
|
|
|
|
---
|
|
Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/`
|
|
Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements,
|
|
the `> 0` guard rationale, and the staggered schedule).
|