--- # Velero VOLUME POLICIES — the opt-out escape hatch for the external bulk tier (R-STATE-04). # # WHY THIS EXISTS. platform/velero/helmrelease.yaml schedule (1) runs # `defaultVolumesToFsBackup: true` — the OPT-OUT model, so no volume with real data is ever # forgotten. That model collides with the NFS bulk tier prod grew on 2026-09-11: # # plex-media 9Ti claim -> 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured) # sonarr-media 9Ti claim -> the SAME export # radarr-media 9Ti claim -> the SAME export # nextcloud-legacy-archive 270Gi -> 192.168.2.102:/mnt/Storage/nextcloud # # Left to opt-out, one nightly run reads that library THREE TIMES over NFS and pushes ~8.6 TiB at # Cloudflare R2, every night, held for ${VELERO_RETENTION} (prod: 720h). kopia dedups the STORED # bytes, not the reads. That is not DR; it is a self-inflicted outage plus a recurring bill. # # WHY NOT namespace exclusion. plex / sonarr / radarr / nextcloud each ALSO hold a real Longhorn # PVC (plex-config 8Gi, sonarr-config 5Gi, radarr-config 5Gi, nextcloud-data 20Gi) whose loss is # unrecoverable. Excluding those namespaces — the shape schedule (2) uses for monitoring+unifi — # would silently drop exactly the data that matters. The opt-out has to be per VOLUME. # # WHY NOT the per-pod `backup.velero.io/backup-volumes-excludes` annotation. It works, but it fails # OPEN: the next NFS PV someone mounts is backed up in full until a human remembers to annotate its # pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier — # its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet. # # RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds # no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up # cannot protect anything, and ATTEMPTING it actively breaks the backup. # # HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced # mid-flight the PVB fails with `error to expose PVB: error getting client pod : pods "" # not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not # advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a # backup that protected every real volume. Any Deployment rollout landing in the backup window is # enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is # how a real backup failure gets ignored, and this one guards R-STATE-03. # # OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a # targeted exclusion: # * ns velero — Velero's OWN transient kopia repository-maintenance Jobs # (`-default-kopia-maintain-job-`), emptyDir `plugins` + `scratch`. Four # failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed, # 2 errors, 3079 of 3079 items backed up. # * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs # writable scratch). Four failed PVBs when a runbook edit rolled the Deployment # while the backup was running. # # PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open # question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a # THROWAWAY ConfigMap first, leaving this one untouched until it passed: # control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed # same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors # The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted. # # `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to # narrow it; deliberately not, so a second export is covered too). First matching rule wins. # # PROVEN, not assumed (prod, 2026-09-12, velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud # with defaultVolumesToFsBackup:true and this policy produced exactly ONE PodVolumeBackup — # `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume # produced NO PodVolumeBackup at all. # # Referenced by name from schedule (1)'s template (`resourcePolicy: {kind: configmap, name: ...}`). # Applied to BOTH environments: dev has no NFS PVs today, so the rule is a no-op there, and it is # already correct if dev ever grows one. apiVersion: v1 kind: ConfigMap metadata: name: velero-volume-policies namespace: velero data: policies.yaml: | version: v1 volumePolicies: # emptyDir first: cheapest condition, and the most common match. - conditions: volumeTypes: - emptyDir action: type: skip - conditions: nfs: {} action: type: skip