jakman85andClaude Opus 5
44983095e1
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>
160 lines
9.6 KiB
Markdown
160 lines
9.6 KiB
Markdown
# cnpg-backup-missed — No recent CNPG base backup for a Postgres cluster
|
|
|
|
**Alert:** `CNPGNoRecentBaseBackup` (warning, 2h)
|
|
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
|
|
**Expression:**
|
|
```
|
|
(time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
|
and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
|
|
```
|
|
|
|
## What this means here
|
|
|
|
The newest completed **base backup** for one Postgres cluster is over 36 hours old — one missed daily run
|
|
plus slack. The daily `ScheduledBackup` has silently stopped producing backups, so the PITR window is
|
|
ageing out against the `30d` retention policy.
|
|
|
|
**Nothing else covers this.** The Velero rules in `platform/monitoring/alerts.yaml` watch Velero's own
|
|
schedules, and **Velero does not back up these databases** — CNPG owns their DR path, because a Velero
|
|
PVC snapshot of a live Postgres is not a backup. So `/runbooks/backup-missed` being clear says nothing
|
|
about the databases, and vice versa.
|
|
|
|
The three Clusters and their staggered base-backup times: **keycloak 01:00**, **nextcloud 02:00**,
|
|
**semaphore 04:00** (Velero's own daily runs at 03:00, deliberately between them).
|
|
|
|
## Two expression details that matter when you read the alert
|
|
|
|
**The `> 0` guard is load-bearing.** A cluster that has *never* had a backup reports `0`, and
|
|
`time() - 0` is about **57 years** — always true. Without the guard this rule would double-notify with
|
|
`CNPGNoRecoverabilityPoint`, which is the correct alert for "never had a backup". So this rule means
|
|
specifically: *this cluster has had at least one backup, and the newest one is stale.*
|
|
|
|
**It is aggregated `by (namespace, job)`, not `by (namespace, pod)`** — fixed 2026-09-12 alongside
|
|
`CNPGNoRecoverabilityPoint`. `cnpg_collector_last_available_backup_timestamp` has the same primary-only
|
|
defect: measured on prod the same day, `keycloak-db-2` = 3.1h old, `nextcloud-db-2` = 2.1h,
|
|
`semaphore-db-2` = 0.1h, and **all six replicas = 0**, i.e. `time() - 0` ≈ **56.7 years**. Unlike its
|
|
sibling this rule was *not* firing — because the `> 0` guard was the single thing standing between it and
|
|
six more permanent false positives, which is far too thin a margin for a DR alarm. Aggregating `pod` away
|
|
removes the dependence: `max` picks the primary's timestamp as the cluster's answer (a demoted primary
|
|
drops to 0 and can never win the `max`; a fresher backup always beats a staler one).
|
|
`job` is the grouping label because the metric carries **no `cnpg.io/cluster` label** — the operator
|
|
creates one PodMonitor per Cluster, so prometheus-operator derives `job="<namespace>/<cluster>"`:
|
|
`keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`.
|
|
|
|
**Severity is `warning` on purpose** — this repo has been burned by a permanently-firing `critical`
|
|
(the `VPNTunnelDown` post-mortem), and *"noise on a DR alarm is worse than no alarm at all"*.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
|
defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
|
|
about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
|
|
Do not run two `dev.sh` invocations at once — they share one container and interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
**Fully-qualified names, always.** `kubectl get backup -A` prints **"No resources found"** on this
|
|
cluster even with **27 CNPG backups** present: three CRDs register the kind `Backup`
|
|
(`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1`, `velero.io/v1`) and the bare short name resolves to
|
|
**Longhorn's**, which holds zero objects. Verified on prod 2026-09-13. Concluding "the backups are gone"
|
|
from that and acting on it is how a CLI quirk becomes data loss.
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLUSTER:.spec.cluster.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt --sort-by=.metadata.creationTimestamp | tail -20'
|
|
```
|
|
|
|
Are the schedules there and enabled?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,CLUSTER:.spec.cluster.name,LAST:.status.lastScheduleTime'
|
|
```
|
|
|
|
The Cluster's own verdict (and the condition that tells you whether *archiving* is also broken):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
|
|
```
|
|
|
|
Why the last attempt failed — the barman error is in the primary's logs:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -iE "barman|backup|archive" | tail -30; echo ====; \
|
|
kubectl --context k3s-prod -n cnpg-system logs deploy/cnpg-controller-manager --tail=120 2>&1 | grep -iE "backup|error" | tail -20'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The ScheduledBackup is suspended, missing, or unapplied
|
|
Check `spec.suspend`, and whether the object exists at all for that Cluster. A `ScheduledBackup` that was
|
|
never applied is the "looks done, is absent" failure class this repo keeps finding; verify the Flux tier
|
|
that owns it is Ready (`/runbooks/flux-reconcile`) and not suspended (`/runbooks/flux-suspended`).
|
|
|
|
### 2. The object store is rejecting the write
|
|
Same root cause family as `/runbooks/cnpg-archiving`: endpoint, path, region or credentials. Live prod
|
|
values to expect are `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
|
|
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
|
|
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`. **If `ContinuousArchiving` is also
|
|
False, fix that first** — a cluster whose archiver is broken has a disk-full countdown running, which is
|
|
more urgent than a stale base backup.
|
|
|
|
### 3. A failed run that nobody noticed
|
|
There is precedent: one historical failure, `semaphore-db-daily-20260905130918`. A single failed run does
|
|
not clear itself; the next night's run should. If two consecutive nights fail, the cause is
|
|
configuration, not luck.
|
|
|
|
### 4. The primary changed and the new primary cannot write
|
|
The backup catalogue is the **primary's** job. After a switchover, a credential or path problem scoped to
|
|
one instance shows up here. Check which pod is `currentPrimary` and read that pod's logs, not another's.
|
|
|
|
### 5. Node or storage trouble during the backup window
|
|
Base backups are I/O-heavy and run at 01:00-04:00 alongside Velero. Cross-read `/runbooks/node-io`.
|
|
|
|
### 6. The metric is missing, not the backup
|
|
If `cnpg_collector_last_available_backup_timestamp` has no series, the rule cannot fire. Note that a
|
|
**related** blind spot is real and separate: the CNPG SQL exporter can fail while `up` stays 1 — see
|
|
`/runbooks/cnpg-exporter`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **A stale base backup narrows, but does not eliminate, recoverability** — provided WAL archiving is
|
|
healthy, PITR still works from the older base. That is why this is a warning. If **both** this and
|
|
`CNPGNoRecoverabilityPoint` / `CNPGWALArchiveBacklog` are lit, DR for that database is effectively
|
|
gone: escalate.
|
|
- **Never delete WAL, `.ready` markers, or anything under a Postgres data directory.**
|
|
- **Do not delete old CNPG Backup objects to tidy up.** Retention is `30d` and the operator manages it;
|
|
deleting a Backup object can orphan objects in R2.
|
|
- **Do not trigger a base backup as a reflex** on a cluster whose archiver is failing — it adds write
|
|
load to a volume that may already be filling.
|
|
- **Be honest about restore:** the CNPG restore path from R2 has not been rehearsed on this cluster
|
|
(`dr-restore` is authored and never executed). Free R2 egress makes a rehearsal cheap; it is
|
|
outstanding. A restore is an operator operation.
|
|
|
|
---
|
|
Source of truth: `runbooks/cnpg-backup-missed.md`. Index: `/runbooks/`
|
|
Records: `platform/cnpg/alerts.yaml` (the 2026-09-12 aggregation-scope fix with its live measurements,
|
|
the `> 0` guard rationale, and the staggered schedule).
|