# backup-missed — No successful Velero backup in 24 hours **Alert:** `VeleroNoRecentBackup` (critical, 1h) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` **Expression:** `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)` ## What this means here A Velero **schedule** has not produced a successful backup in over a day. This is the alert that catches the quiet failure — not "a backup broke" (that is `/runbooks/backup-failure`) but "backups stopped happening and nobody noticed". The `{schedule!=""}` filter means it only watches scheduled backups, so ad-hoc ones do not mask a dead schedule. Live shape on prod: two schedules, both `0 3 * * *` — `schedule.velero.io/velero-daily` and `schedule.velero.io/velero-daily-resources-only` — retention `720h`, target Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, region `auto`). **Why this alert is worth its `critical`:** the precedent is that these rules *could not fire at all*. Velero shipped with three backup alerts and the chart default `metrics.serviceMonitor.enabled: false`, confirmed live on prod — `kubectl -n velero get servicemonitor` returned no resources. Recorded verbatim: *"A nightly backup could fail — or silently stop running — for weeks with nothing to say so."* The ServiceMonitor now exists (`velero` and `velero/node-agent` are both live scrape jobs as of 2026-09-13), which is the only reason this alert is real. ## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE `kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** while the short name returned nothing. This false negative has already produced two wrong "the backups are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk becomes real data loss.** Always: ``` backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io ``` ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Are the schedules enabled, and when did each last run? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get schedules.velero.io \ -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase' ``` What backups actually exist (fully-qualified — the short name lies): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.velero.io -A \ -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp' ``` Is the storage location usable at all? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \ -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default' ``` Is Velero's own Flux tier reconciling, or was it suspended? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"' ``` And the controller: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get pods; \ kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40' ``` ## Likely causes, most likely first ### 1. The Velero Flux tier was suspended and nobody resumed it This has happened: **Velero was found suspended live, with the suspension never recorded in git**, and was resumed with `flux resume`. A suspended backup controller is a silent DR gap and is precisely the class this alert exists for. Remediation (operator): `flux resume kustomization velero`, and see `/runbooks/flux-suspended` for why an undocumented live suspension is itself a problem. ### 2. The schedule is paused `schedules.velero.io` has `spec.paused`. Note from the live listing that `velero-daily-resources-only` shows an empty PAUSED column while `velero-daily` shows `false` — read the field, not the column width. A paused schedule is a git change to unpause. ### 3. Every run is failing rather than not running Then `backups.velero.io` will show recent `Failed` entries and this alert is a downstream symptom. Go to `/runbooks/backup-failure` — and in particular check the plugin pin (`velero/velero-plugin-for-aws:v1.14.3-rc.2`), because the R2 tagging bug produces exactly this: volume data uploads, the backup goes Failed, and no successful backup ever lands. ### 4. The BackupStorageLocation is not Available A wrong endpoint, a bad credential, or the placeholder. `clusters/prod` deliberately carried the loud placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION` (`auto`), `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX`, `BACKUP_CREDENTIALS_SUFFIX` (`-prod`). ### 5. Velero cannot schedule a pod Node pressure, or the node-agent DaemonSet unhealthy. Cross-read `/runbooks/node-io`. ### 6. The metric, not the backup, is missing If `velero_backup_last_successful_timestamp` has no series at all the rule cannot fire — which is the historical state. Confirm the `velero` scrape job is up (`/runbooks/targets-down`) before trusting either its silence or its firing. ## When to escalate or stop - **Escalate: this is a DR alarm and 24h is already a real window of exposure.** - **Do not create an ad-hoc backup and consider the alert answered.** The expression filters `{schedule!=""}`, so a manual backup will not clear it, and more importantly it does not fix the schedule. (An operator's ad-hoc run for reassurance is `velero backup create --from-schedule velero-daily`.) - **Do not conclude "no backups exist" from `kubectl get backup`.** See the trap above. - **Remember the databases are separate.** `keycloak-db`, `nextcloud-db` and `semaphore-db` are covered by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice versa — check `/runbooks/cnpg-backup-missed` too. - **Be honest about restore:** `dr-restore` is authored and has **never been executed**, so the R2 restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding. --- Source of truth: `runbooks/backup-missed.md`. Index: `/runbooks/` Records: commit `2d5171e` (Velero had no ServiceMonitor, so these alerts could never fire); `DECISIONS-LOG.md` (Velero found suspended with the suspension never in git; `dr-restore` never executed); commit `32d6766` (R2 decision and endpoint knobs).