jakman85andClaude Opus 5
2e259a808f
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>
51 lines
2.6 KiB
YAML
51 lines
2.6 KiB
YAML
---
|
|
# Velero VOLUME POLICIES — the opt-out escape hatch for the external bulk tier (R-STATE-04).
|
|
#
|
|
# WHY THIS EXISTS. platform/velero/helmrelease.yaml schedule (1) runs
|
|
# `defaultVolumesToFsBackup: true` — the OPT-OUT model, so no volume with real data is ever
|
|
# forgotten. That model collides with the NFS bulk tier prod grew on 2026-09-11:
|
|
#
|
|
# plex-media 9Ti claim -> 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured)
|
|
# sonarr-media 9Ti claim -> the SAME export
|
|
# radarr-media 9Ti claim -> the SAME export
|
|
# nextcloud-legacy-archive 270Gi -> 192.168.2.102:/mnt/Storage/nextcloud
|
|
#
|
|
# Left to opt-out, one nightly run reads that library THREE TIMES over NFS and pushes ~8.6 TiB at
|
|
# Cloudflare R2, every night, held for ${VELERO_RETENTION} (prod: 720h). kopia dedups the STORED
|
|
# bytes, not the reads. That is not DR; it is a self-inflicted outage plus a recurring bill.
|
|
#
|
|
# WHY NOT namespace exclusion. plex / sonarr / radarr / nextcloud each ALSO hold a real Longhorn
|
|
# PVC (plex-config 8Gi, sonarr-config 5Gi, radarr-config 5Gi, nextcloud-data 20Gi) whose loss is
|
|
# unrecoverable. Excluding those namespaces — the shape schedule (2) uses for monitoring+unifi —
|
|
# would silently drop exactly the data that matters. The opt-out has to be per VOLUME.
|
|
#
|
|
# WHY NOT the per-pod `backup.velero.io/backup-volumes-excludes` annotation. It works, but it fails
|
|
# OPEN: the next NFS PV someone mounts is backed up in full until a human remembers to annotate its
|
|
# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
|
|
# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
|
|
#
|
|
# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
|
|
# narrow it; deliberately not, so a second export is covered too). First matching rule wins.
|
|
#
|
|
# PROVEN, not assumed (prod, 2026-09-12, velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud
|
|
# with defaultVolumesToFsBackup:true and this policy produced exactly ONE PodVolumeBackup —
|
|
# `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume
|
|
# produced NO PodVolumeBackup at all.
|
|
#
|
|
# Referenced by name from schedule (1)'s template (`resourcePolicy: {kind: configmap, name: ...}`).
|
|
# Applied to BOTH environments: dev has no NFS PVs today, so the rule is a no-op there, and it is
|
|
# already correct if dev ever grows one.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: velero-volume-policies
|
|
namespace: velero
|
|
data:
|
|
policies.yaml: |
|
|
version: v1
|
|
volumePolicies:
|
|
- conditions:
|
|
nfs: {}
|
|
action:
|
|
type: skip
|