fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB

Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.

1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
   It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
   `x-amz-tagging` header on every PUT. R2 does not implement object tagging in
   any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
   it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
   the volume data fine (it uses its own S3 client), then the backup went
   phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
   `config.tagging` with a real value sends a tag set R2 also rejects. Upstream
   PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
   was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
   with the exit condition (v1.14.3 GA) stated at the pin.

2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
   Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
   and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
   plex-media, sonarr-media and radarr-media all bound to the SAME export
   /mnt/Storage/plex (8.32 TiB measured via zfs list), plus
   nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
   library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
   720h. Namespace exclusion cannot express the fix: those four namespaces also
   hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
   loss is unrecoverable.

   So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
   (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
   schedule (1) references it. That tests the property that makes a volume
   bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
   the next PV would slip past. The per-pod backup-volumes-excludes annotation
   was rejected for failing OPEN — a new NFS mount would be backed up in full
   until someone remembered.

   PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
   defaultVolumesToFsBackup:true plus this policy produced exactly ONE
   PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
   Completed). The NFS `legacy-archive` volume produced none.

Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/velero/helmrelease.yaml

@@ -65,11 +65,30 @@ spec:
     #     acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
     #     kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
     upgradeCRDs: false
-    # --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container.
-    #     Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). ---
+    # --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init
+    #     container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x).
+    #
+    # *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). ***
+    # v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY:
+    #     Tagging: aws.String(o.tagging)            // o.tagging == "" when config.tagging is unset
+    # so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement
+    # object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus
+    # Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501:
+    #     NotImplemented: Header 'x-amz-tagging' with value '' not implemented
+    # Observed live on prod twice, on the write of `velero/backups/<name>/velero-backup.json`. The
+    # Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then
+    # goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that
+    # avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects.
+    # The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes
+    # the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so
+    # the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet.
+    # EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT
+    # "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.
+    # Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally
+    # rather than becoming a fifth per-environment knob. ---
     initContainers:
       - name: velero-plugin-for-aws
-        image: velero/velero-plugin-for-aws:v1.14.2
+        image: velero/velero-plugin-for-aws:v1.14.3-rc.2
         imagePullPolicy: IfNotPresent
         volumeMounts:
           - mountPath: /target
@@ -128,6 +147,23 @@ spec:
     schedules:
       # (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
       #     defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
+      #
+      #     THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out
+      #     + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs
+      #     (plex-media, sonarr-media, radarr-media — all three bound to the SAME export
+      #     /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and
+      #     their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So
+      #     namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into
+      #     schedule (2) would drop their real config PVCs, and leaving them here uploaded the media
+      #     library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads).
+      #     `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are
+      #     called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so
+      #     the rule tests the property rather than a name list that a future PV would slip past.
+      #     Media on that ZFS pool has its own protection and is reconstructable; it is deliberately
+      #     NOT cluster-DR scope. See platform/velero/volume-policies.yaml.
+      #     RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume
+      #     on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked
+      #     up by opt-out again and needs its own condition added there.
       daily:
         disabled: false
         schedule: "${VELERO_BACKUP_SCHEDULE}"
@@ -137,6 +173,9 @@ spec:
           storageLocation: default
           snapshotVolumes: false
           defaultVolumesToFsBackup: true
+          resourcePolicy:
+            kind: configmap
+            name: velero-volume-policies
           excludedNamespaces:
             - monitoring        # volume DATA excluded here; resources captured by schedule (2)
             - unifi