# backup-partial — A Velero backup completed with partial failures **Alert:** `VeleroBackupPartialFailure` (warning, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups` **Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0` ## What this means here The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a critical because *most of it worked* — and that is exactly what makes it dangerous: a `PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be the thing you need. **The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is intentional and correct. ## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE `kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**. Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups** while the short name returned nothing. This false negative has already produced two wrong "the backups are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk becomes real data loss.** Always: ``` backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io ``` ## The deliberate exclusions (know these before calling anything a failure) **NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and `unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and `radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with 720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real Longhorn PVCs that must be backed up. The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any volume whose source is NFS, and the schedule references it. The per-pod `backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one** PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none. **Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be covered. **The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and `semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see `/runbooks/cnpg-archiving` for the path that does cover them. **`monitoring` and `unifi` are excluded** from file-system backup as well. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git (`git push origin main` **is** the deploy). Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which backup, and what exactly it warned about: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.velero.io -A \ -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get backups.velero.io -o yaml | sed -n "/status:/,\$p"' ``` Per-volume results — this tells you whether a skip was the NFS policy or a real failure: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \ -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone' ``` The volume policy that does the deliberate skipping: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml' ``` The controller's own account of what it skipped: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30' ``` ## Likely causes, most likely first ### 1. The deliberate NFS / CNPG / monitoring exclusions Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB near-miss above. ### 2. A resource Velero could not serialise Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is genuinely not backed up. ### 3. A volume was busy, degraded, or on a node under pressure Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and `/runbooks/node-io`. ### 4. A partial R2 write Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a hard `Failed`, but a partially-written backup is worth checking the plugin pin for (`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2). ### 5. A namespace was created or deleted mid-backup Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace disappearing under Velero produces exactly this warning. ## When to escalate or stop - **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of the documented exclusions, treat it as an uncovered resource. - **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid. - **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing OPEN, and the consequence of forgetting it is 8 TiB per night. - **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and deleting R2 objects is an operator decision. - Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which is an R-STATE-03 matter and not a transient. --- Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/` Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the fail-open annotation); `docs/BULK-NFS-STORAGE.md`.