home-network-k3s-impl/runbooks/backup-missed.md

backup-missed — No successful Velero backup in 24 hours

Alert: VeleroNoRecentBackup (critical, 1h) File: platform/monitoring/alerts.yaml, group platform-health.backups Expression: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)

What this means here

A Velero schedule has not produced a successful backup in over a day. This is the alert that catches the quiet failure — not "a backup broke" (that is /runbooks/backup-failure) but "backups stopped happening and nobody noticed". The {schedule!=""} filter means it only watches scheduled backups, so ad-hoc ones do not mask a dead schedule.

Live shape on prod: two schedules, both 0 3 * * * — schedule.velero.io/velero-daily and schedule.velero.io/velero-daily-resources-only — retention 720h, target Cloudflare R2 (bucket k3s-prod-backup, prefix velero, region auto).

Why this alert is worth its critical: the precedent is that these rules could not fire at all. Velero shipped with three backup alerts and the chart default metrics.serviceMonitor.enabled: false, confirmed live on prod — kubectl -n velero get servicemonitor returned no resources. Recorded verbatim: "A nightly backup could fail — or silently stop running — for weeks with nothing to say so." The ServiceMonitor now exists (velero and velero/node-agent are both live scrape jobs as of 2026-09-13), which is the only reason this alert is real.

READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE

kubectl get backup -A prints "No resources found" on this cluster even when backups exist. Three CRDs register the kind Backup — longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1 — and the bare short name resolves to Longhorn's, which holds zero objects. Verified on prod 2026-09-13: the fully-qualified queries returned 6 Velero backups and 27 CNPG backups while the short name returned nothing. This false negative has already produced two wrong "the backups are missing" conclusions in one night. A destructive recovery action taken on it is how a CLI quirk becomes real data loss. Always:

backups.velero.io           backups.postgresql.cnpg.io           backups.longhorn.io
schedules.velero.io         scheduledbackups.postgresql.cnpg.io  clusters.postgresql.cnpg.io

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Are the schedules enabled, and when did each last run?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get schedules.velero.io \
     -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LASTBACKUP:.status.lastBackup,PHASE:.status.phase'

What backups actually exist (fully-qualified — the short name lies):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.velero.io -A \
     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,SCHEDULE:.metadata.labels.velero\\.io/schedule-name,START:.status.startTimestamp --sort-by=.status.startTimestamp'

Is the storage location usable at all?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io \
     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,LASTVALIDATED:.status.lastValidationTime,DEFAULT:.spec.default'

Is Velero's own Flux tier reconciling, or was it suspended?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
     -o custom-columns=NAME:.metadata.name,SUSPENDED:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status | grep -Ei "velero|NAME"'

And the controller:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get pods; \
   kubectl --context k3s-prod -n velero logs deploy/velero --tail=120 2>&1 | tail -40'

Likely causes, most likely first

1. The Velero Flux tier was suspended and nobody resumed it

This has happened: Velero was found suspended live, with the suspension never recorded in git, and was resumed with flux resume. A suspended backup controller is a silent DR gap and is precisely the class this alert exists for. Remediation (operator): flux resume kustomization velero, and see /runbooks/flux-suspended for why an undocumented live suspension is itself a problem.

2. The schedule is paused

schedules.velero.io has spec.paused. Note from the live listing that velero-daily-resources-only shows an empty PAUSED column while velero-daily shows false — read the field, not the column width. A paused schedule is a git change to unpause.

3. Every run is failing rather than not running

Then backups.velero.io will show recent Failed entries and this alert is a downstream symptom. Go to /runbooks/backup-failure — and in particular check the plugin pin (velero/velero-plugin-for-aws:v1.14.3-rc.2), because the R2 tagging bug produces exactly this: volume data uploads, the backup goes Failed, and no successful backup ever lands.

4. The BackupStorageLocation is not Available

A wrong endpoint, a bad credential, or the placeholder. clusters/prod deliberately carried the loud placeholder https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com rather than inheriting the base (dev) value, because "an omitted op means prod silently inherits a dev address — far worse than a failed apply". Check S3_BACKUP_ENDPOINT, S3_BACKUP_REGION (auto), VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX, BACKUP_CREDENTIALS_SUFFIX (-prod).

5. Velero cannot schedule a pod

Node pressure, or the node-agent DaemonSet unhealthy. Cross-read /runbooks/node-io.

6. The metric, not the backup, is missing

If velero_backup_last_successful_timestamp has no series at all the rule cannot fire — which is the historical state. Confirm the velero scrape job is up (/runbooks/targets-down) before trusting either its silence or its firing.

When to escalate or stop

• Escalate: this is a DR alarm and 24h is already a real window of exposure.
• Do not create an ad-hoc backup and consider the alert answered. The expression filters {schedule!=""}, so a manual backup will not clear it, and more importantly it does not fix the schedule. (An operator's ad-hoc run for reassurance is velero backup create --from-schedule velero-daily.)
• Do not conclude "no backups exist" from kubectl get backup. See the trap above.
• Remember the databases are separate. keycloak-db, nextcloud-db and semaphore-db are covered by CNPG's Barman archive, not by Velero. This alert being clear says nothing about them, and vice versa — check /runbooks/cnpg-backup-missed too.
• Be honest about restore: dr-restore is authored and has never been executed, so the R2 restore path is unrehearsed. Free R2 egress makes a rehearsal cheap; it is still outstanding.

---

Source of truth: runbooks/backup-missed.md. Index: /runbooks/ Records: commit 2d5171e (Velero had no ServiceMonitor, so these alerts could never fire); DECISIONS-LOG.md (Velero found suspended with the suspension never in git; dr-restore never executed); commit 32d6766 (R2 decision and endpoint knobs).