Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.
1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
`x-amz-tagging` header on every PUT. R2 does not implement object tagging in
any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
the volume data fine (it uses its own S3 client), then the backup went
phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
`config.tagging` with a real value sends a tag set R2 also rejects. Upstream
PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
with the exit condition (v1.14.3 GA) stated at the pin.
2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
plex-media, sonarr-media and radarr-media all bound to the SAME export
/mnt/Storage/plex (8.32 TiB measured via zfs list), plus
nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
720h. Namespace exclusion cannot express the fix: those four namespaces also
hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
loss is unrecoverable.
So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
(ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
schedule (1) references it. That tests the property that makes a volume
bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
the next PV would slip past. The per-pod backup-volumes-excludes annotation
was rejected for failing OPEN — a new NFS mount would be backed up in full
until someone remembered.
PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
defaultVolumesToFsBackup:true plus this policy produced exactly ONE
PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
Completed). The NFS `legacy-archive` volume produced none.
Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.
Co-Authored-By: Claude Opus 5 <[email protected]>
