fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 authored and x committed
1 parent
3ec67bafdd
commit
2e259a808f
4 files changed
+103
-3
No files matched your search
@@ -65,11 +65,30 @@ spec:
|
||||
# acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
|
||||
# kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
|
||||
upgradeCRDs: false
|
||||
# --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container.
|
||||
# Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). ---
|
||||
# --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init
|
||||
# container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x).
|
||||
#
|
||||
# *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). ***
|
||||
# v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY:
|
||||
# Tagging: aws.String(o.tagging) // o.tagging == "" when config.tagging is unset
|
||||
# so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement
|
||||
# object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus
|
||||
# Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501:
|
||||
# NotImplemented: Header 'x-amz-tagging' with value '' not implemented
|
||||
# Observed live on prod twice, on the write of `velero/backups/<name>/velero-backup.json`. The
|
||||
# Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then
|
||||
# goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that
|
||||
# avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects.
|
||||
# The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes
|
||||
# the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so
|
||||
# the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet.
|
||||
# EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT
|
||||
# "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.
|
||||
# Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally
|
||||
# rather than becoming a fifth per-environment knob. ---
|
||||
initContainers:
|
||||
- name: velero-plugin-for-aws
|
||||
image: velero/velero-plugin-for-aws:v1.14.2
|
||||
image: velero/velero-plugin-for-aws:v1.14.3-rc.2
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumeMounts:
|
||||
- mountPath: /target
|
||||
@@ -128,6 +147,23 @@ spec:
|
||||
schedules:
|
||||
# (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
|
||||
# defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
|
||||
#
|
||||
# THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out
|
||||
# + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs
|
||||
# (plex-media, sonarr-media, radarr-media — all three bound to the SAME export
|
||||
# /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and
|
||||
# their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So
|
||||
# namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into
|
||||
# schedule (2) would drop their real config PVCs, and leaving them here uploaded the media
|
||||
# library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads).
|
||||
# `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are
|
||||
# called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so
|
||||
# the rule tests the property rather than a name list that a future PV would slip past.
|
||||
# Media on that ZFS pool has its own protection and is reconstructable; it is deliberately
|
||||
# NOT cluster-DR scope. See platform/velero/volume-policies.yaml.
|
||||
# RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume
|
||||
# on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked
|
||||
# up by opt-out again and needs its own condition added there.
|
||||
daily:
|
||||
disabled: false
|
||||
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
||||
@@ -137,6 +173,9 @@ spec:
|
||||
storageLocation: default
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
resourcePolicy:
|
||||
kind: configmap
|
||||
name: velero-volume-policies
|
||||
excludedNamespaces:
|
||||
- monitoring # volume DATA excluded here; resources captured by schedule (2)
|
||||
- unifi
|
||||
|
||||
@@ -5,6 +5,9 @@ resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml
|
||||
# Volume policies for schedule (1)'s opt-out FSB: skip the external NFS bulk tier by SOURCE TYPE.
|
||||
# Must exist before the Schedule references it by name (same Kustomization, so one apply).
|
||||
- volume-policies.yaml
|
||||
# --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) -----------------------------------
|
||||
# BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via
|
||||
# `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "",
|
||||
|
||||
@@ -9,6 +9,14 @@
|
||||
# sharing one bucket with CNPG/barman under a disjoint prefix.
|
||||
# - config.checksumAlgorithm: "" — MANDATORY for R2, which rejects the AWS SDK's
|
||||
# x-amz-checksum-* headers with HTTP 400 InvalidArgument. Harmless against MinIO.
|
||||
# - velero-plugin-for-aws PINNED TO v1.14.3-rc.2 — also MANDATORY for R2. v1.14.2 sends an empty
|
||||
# `x-amz-tagging` header on every PutObject, which R2 answers with HTTP 501 NotImplemented, so
|
||||
# every Backup lands its kopia volume data and then FAILS writing velero-backup.json. Upstream
|
||||
# PR #309 makes the field conditional; rc.2 is the earliest build carrying it. Move to v1.14.3
|
||||
# when it ships GA. Measured on prod 2026-09-12.
|
||||
# - resourcePolicy on the FSB schedule -> ConfigMap velero-volume-policies (volume-policies.yaml):
|
||||
# skips volumes whose source is NFS, so the 8.6 TiB external bulk tier is not uploaded nightly
|
||||
# while the same namespaces' real Longhorn PVCs still are. Proven on prod 2026-09-12.
|
||||
# - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover).
|
||||
# - snapshotsEnabled:false → no VolumeSnapshotLocation.
|
||||
# - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero).
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
# Velero VOLUME POLICIES — the opt-out escape hatch for the external bulk tier (R-STATE-04).
|
||||
#
|
||||
# WHY THIS EXISTS. platform/velero/helmrelease.yaml schedule (1) runs
|
||||
# `defaultVolumesToFsBackup: true` — the OPT-OUT model, so no volume with real data is ever
|
||||
# forgotten. That model collides with the NFS bulk tier prod grew on 2026-09-11:
|
||||
#
|
||||
# plex-media 9Ti claim -> 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured)
|
||||
# sonarr-media 9Ti claim -> the SAME export
|
||||
# radarr-media 9Ti claim -> the SAME export
|
||||
# nextcloud-legacy-archive 270Gi -> 192.168.2.102:/mnt/Storage/nextcloud
|
||||
#
|
||||
# Left to opt-out, one nightly run reads that library THREE TIMES over NFS and pushes ~8.6 TiB at
|
||||
# Cloudflare R2, every night, held for ${VELERO_RETENTION} (prod: 720h). kopia dedups the STORED
|
||||
# bytes, not the reads. That is not DR; it is a self-inflicted outage plus a recurring bill.
|
||||
#
|
||||
# WHY NOT namespace exclusion. plex / sonarr / radarr / nextcloud each ALSO hold a real Longhorn
|
||||
# PVC (plex-config 8Gi, sonarr-config 5Gi, radarr-config 5Gi, nextcloud-data 20Gi) whose loss is
|
||||
# unrecoverable. Excluding those namespaces — the shape schedule (2) uses for monitoring+unifi —
|
||||
# would silently drop exactly the data that matters. The opt-out has to be per VOLUME.
|
||||
#
|
||||
# WHY NOT the per-pod `backup.velero.io/backup-volumes-excludes` annotation. It works, but it fails
|
||||
# OPEN: the next NFS PV someone mounts is backed up in full until a human remembers to annotate its
|
||||
# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
|
||||
# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
|
||||
#
|
||||
# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
|
||||
# narrow it; deliberately not, so a second export is covered too). First matching rule wins.
|
||||
#
|
||||
# PROVEN, not assumed (prod, 2026-09-12, velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud
|
||||
# with defaultVolumesToFsBackup:true and this policy produced exactly ONE PodVolumeBackup —
|
||||
# `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume
|
||||
# produced NO PodVolumeBackup at all.
|
||||
#
|
||||
# Referenced by name from schedule (1)'s template (`resourcePolicy: {kind: configmap, name: ...}`).
|
||||
# Applied to BOTH environments: dev has no NFS PVs today, so the rule is a no-op there, and it is
|
||||
# already correct if dev ever grows one.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: velero-volume-policies
|
||||
namespace: velero
|
||||
data:
|
||||
policies.yaml: |
|
||||
version: v1
|
||||
volumePolicies:
|
||||
- conditions:
|
||||
nfs: {}
|
||||
action:
|
||||
type: skip
|
||||
Reference in new issue
Block a user