# backup-failure — A Velero backup failed **Alert:** `VeleroBackupFailure` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` **Expression:** `increase(velero_backup_failure_total[1h]) > 0` ## What this means here A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate incidents with separate runbooks (`/runbooks/cnpg-archiving`). ## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE `kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** while the short name returned nothing. This has already caused two false "the backups are missing" conclusions in one night. **A destructive recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use: ``` backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io ``` ## Where the backups go Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain. Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region **`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`. `backupstoragelocation.velero.io/default` should read **Available**. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which backup failed, and why — fully-qualified, always: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.velero.io -A \ -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' ``` The storage location and the controller's log (the R2 error text is in the log, not the CR): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \ kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25' ``` Did the **volume data** upload even though the backup failed? (This distinction is the whole point — see cause 1.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone' ``` The plugin pin (cause 1 depends entirely on this): ``` grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml ``` ## Likely causes, most likely first ### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one) `velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally — `Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on **every** PutObject. **R2 does not implement object tagging at all** and answers: ``` NotImplemented: Header 'x-amz-tagging' with value '' not implemented ``` Observed live on prod **twice**, on the write of `velero/backups//velero-backup.json`. The shape of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client), then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.** **No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to **`velero/velero-plugin-for-aws:v1.14.3-rc.2`**. **EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is your answer. Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for MinIO). ### 2. A placeholder or wrong endpoint `clusters/prod` deliberately carried the **loud** placeholder `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value, because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*. Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and `BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A `backupstoragelocation` that is not `Available` points straight here. ### 3. Credentials The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log. ### 4. A volume could not be read Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are **deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`, which is where that belongs. ### 5. The node or storage was unhealthy during the run Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`. ## When to escalate or stop - **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no recent restorable backup. - **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`, `prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting objects in R2 is an operator decision, not a tidy-up. - **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is precisely "data uploaded, metadata missing". - **Be honest about restore**: `dr-restore` is **authored and has never been executed** (`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free, which is the whole reason restore rehearsals are affordable — but nobody has done one yet. - The three CNPG databases are **not** covered by Velero. If the concern is a database, go to `/runbooks/cnpg-archiving`. --- Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/` Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit `2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire).