jakman85andClaude Opus 5
2e259a808f
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>
26 lines
2.1 KiB
YAML
26 lines
2.1 KiB
YAML
# Velero Helm values — the AUTHORITATIVE values live inline in helmrelease.yaml (this file is a
|
|
# doc pointer, kept for parity with the other components' layout).
|
|
#
|
|
# LIVE wiring (see helmrelease.yaml):
|
|
# - provider aws + velero-plugin-for-aws (init container) → S3 ${S3_BACKUP_ENDPOINT}, path-style,
|
|
# at s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX} in region ${S3_BACKUP_REGION}. Four
|
|
# cluster-vars knobs: dev = the on-Proxmox MinIO (http://192.168.1.9:9000, bucket `velero`, no
|
|
# prefix, us-east-1); prod = Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, `auto`),
|
|
# sharing one bucket with CNPG/barman under a disjoint prefix.
|
|
# - config.checksumAlgorithm: "" — MANDATORY for R2, which rejects the AWS SDK's
|
|
# x-amz-checksum-* headers with HTTP 400 InvalidArgument. Harmless against MinIO.
|
|
# - velero-plugin-for-aws PINNED TO v1.14.3-rc.2 — also MANDATORY for R2. v1.14.2 sends an empty
|
|
# `x-amz-tagging` header on every PutObject, which R2 answers with HTTP 501 NotImplemented, so
|
|
# every Backup lands its kopia volume data and then FAILS writing velero-backup.json. Upstream
|
|
# PR #309 makes the field conditional; rc.2 is the earliest build carrying it. Move to v1.14.3
|
|
# when it ships GA. Measured on prod 2026-09-12.
|
|
# - resourcePolicy on the FSB schedule -> ConfigMap velero-volume-policies (volume-policies.yaml):
|
|
# skips volumes whose source is NFS, so the 8.6 TiB external bulk tier is not uploaded nightly
|
|
# while the same namespaces' real Longhorn PVCs still are. Proven on prod 2026-09-12.
|
|
# - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover).
|
|
# - snapshotsEnabled:false → no VolumeSnapshotLocation.
|
|
# - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero).
|
|
# - two daily Schedules (${VELERO_BACKUP_SCHEDULE} / ttl ${VELERO_RETENTION}): the whole cluster
|
|
# via FSB, with monitoring+unifi backed up RESOURCES-ONLY (their volume data is reconstructable).
|
|
# DR drill (tested restore) remains a Phase-4 acceptance gate (R-STATE-03) — the PM runs it post-deploy.
|