fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB

Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.

1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
   It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
   `x-amz-tagging` header on every PUT. R2 does not implement object tagging in
   any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
   it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
   the volume data fine (it uses its own S3 client), then the backup went
   phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
   `config.tagging` with a real value sends a tag set R2 also rejects. Upstream
   PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
   was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
   with the exit condition (v1.14.3 GA) stated at the pin.

2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
   Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
   and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
   plex-media, sonarr-media and radarr-media all bound to the SAME export
   /mnt/Storage/plex (8.32 TiB measured via zfs list), plus
   nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
   library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
   720h. Namespace exclusion cannot express the fix: those four namespaces also
   hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
   loss is unrecoverable.

   So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
   (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
   schedule (1) references it. That tests the property that makes a volume
   bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
   the next PV would slip past. The per-pod backup-volumes-excludes annotation
   was rejected for failing OPEN — a new NFS mount would be backed up in full
   until someone remembered.

   PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
   defaultVolumesToFsBackup:true plus this policy produced exactly ONE
   PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
   Completed). The NFS `legacy-archive` volume produced none.

Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/velero/kustomization.yaml

@@ -5,6 +5,9 @@ resources:
   - namespace.yaml
   - helmrepository.yaml
   - helmrelease.yaml
+  # Volume policies for schedule (1)'s opt-out FSB: skip the external NFS bulk tier by SOURCE TYPE.
+  # Must exist before the Schedule references it by name (same Kustomization, so one apply).
+  - volume-policies.yaml
   # --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) -----------------------------------
   # BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via
   # `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "",