jakman85andClaude Opus 5
2e259a808f
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>
27 lines
1.5 KiB
YAML
27 lines
1.5 KiB
YAML
---
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- namespace.yaml
|
|
- helmrepository.yaml
|
|
- helmrelease.yaml
|
|
# Volume policies for schedule (1)'s opt-out FSB: skip the external NFS bulk tier by SOURCE TYPE.
|
|
# Must exist before the Schedule references it by name (same Kustomization, so one apply).
|
|
- volume-policies.yaml
|
|
# --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) -----------------------------------
|
|
# BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via
|
|
# `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "",
|
|
# prod "-prod"). They are separate objects so rotating one environment's key can never touch the
|
|
# other's — the failure mode that made the split necessary (a single shared Secret carrying R2 keys
|
|
# would have pointed dev's Velero at MinIO with credentials MinIO has never issued).
|
|
#
|
|
# dev -> velero-cloud-credentials : the scoped MinIO key for http://192.168.1.9:9000.
|
|
# Kept in sync with the staging source of truth
|
|
# platform/backup/minio-credentials.sops.yaml + the
|
|
# Proxmox host ansible-vault.
|
|
# prod -> velero-cloud-credentials-prod : the Cloudflare R2 token for k3s-prod-backup.
|
|
#
|
|
# Both are decrypted by the `velero` Flux Kustomization's SOPS stanza (clusters/dev + clusters/prod).
|
|
- velero-cloud.sops.yaml
|
|
- velero-cloud-prod.sops.yaml
|