fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB

Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.

1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
   It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
   `x-amz-tagging` header on every PUT. R2 does not implement object tagging in
   any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
   it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
   the volume data fine (it uses its own S3 client), then the backup went
   phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
   `config.tagging` with a real value sends a tag set R2 also rejects. Upstream
   PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
   was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
   with the exit condition (v1.14.3 GA) stated at the pin.

2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
   Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
   and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
   plex-media, sonarr-media and radarr-media all bound to the SAME export
   /mnt/Storage/plex (8.32 TiB measured via zfs list), plus
   nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
   library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
   720h. Namespace exclusion cannot express the fix: those four namespaces also
   hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
   loss is unrecoverable.

   So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
   (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
   schedule (1) references it. That tests the property that makes a volume
   bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
   the next PV would slip past. The per-pod backup-volumes-excludes annotation
   was rejected for failing OPEN — a new NFS mount would be backed up in full
   until someone remembered.

   PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
   defaultVolumesToFsBackup:true plus this policy produced exactly ONE
   PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
   Completed). The NFS `legacy-archive` volume produced none.

Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/velero/values.yaml

@@ -9,6 +9,14 @@
 #     sharing one bucket with CNPG/barman under a disjoint prefix.
 #   - config.checksumAlgorithm: "" — MANDATORY for R2, which rejects the AWS SDK's
 #     x-amz-checksum-* headers with HTTP 400 InvalidArgument. Harmless against MinIO.
+#   - velero-plugin-for-aws PINNED TO v1.14.3-rc.2 — also MANDATORY for R2. v1.14.2 sends an empty
+#     `x-amz-tagging` header on every PutObject, which R2 answers with HTTP 501 NotImplemented, so
+#     every Backup lands its kopia volume data and then FAILS writing velero-backup.json. Upstream
+#     PR #309 makes the field conditional; rc.2 is the earliest build carrying it. Move to v1.14.3
+#     when it ships GA. Measured on prod 2026-09-12.
+#   - resourcePolicy on the FSB schedule -> ConfigMap velero-volume-policies (volume-policies.yaml):
+#     skips volumes whose source is NFS, so the 8.6 TiB external bulk tier is not uploaded nightly
+#     while the same namespaces' real Longhorn PVCs still are. Proven on prod 2026-09-12.
 #   - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover).
 #   - snapshotsEnabled:false → no VolumeSnapshotLocation.
 #   - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero).