jakman85andClaude Opus 5
2e259a808f
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>
215 lines
14 KiB
YAML
215 lines
14 KiB
YAML
---
|
|
# Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
|
|
#
|
|
# TARGET: the S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars) at
|
|
# s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX}, in region ${S3_BACKUP_REGION}. Four knobs,
|
|
# because the two environments' object stores differ in shape as well as address:
|
|
# dev http://192.168.1.9:9000 bucket `velero`, NO prefix, region us-east-1 (on-Proxmox MinIO,
|
|
# plain HTTP, path-style; MinIO ignores the region)
|
|
# prod https://<acct>.r2.cloudflarestorage.com bucket `k3s-prod-backup`, prefix `velero`,
|
|
# region `auto` (Cloudflare R2; prod cannot route to that MinIO,
|
|
# and R2 demands exactly `auto`)
|
|
# Prod's bucket is SHARED with CNPG/barman (the operator created one bucket), so the `velero` prefix
|
|
# is what keeps this controller's retention sweep away from the Postgres WAL stream. See the
|
|
# "OBJECT-STORE TENANCY" block in clusters/base/cluster-vars.yaml.
|
|
#
|
|
# WHY THIS SHAPE:
|
|
# - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
|
|
# - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
|
|
# NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
|
|
# Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
|
|
# - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero). ONE
|
|
# Secret serves both environments, and it now carries the Cloudflare R2 API token (written by
|
|
# platform/backup/set-r2-credentials.sh, 2026-09-07) rather than the scoped MinIO key. That is
|
|
# safe only because the R2 token is bucket-scoped to k3s-prod-backup, which dev never names:
|
|
# dev's Velero would fail loudly against its MinIO endpoint with an R2 key, it could not
|
|
# silently write somewhere unexpected. Rotating either target means re-running that script.
|
|
#
|
|
# VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
|
|
# cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
|
|
#
|
|
# NON-GATING TIER: reconciled by clusters/base/backup.yaml (dependsOn platform-config, wait:false).
|
|
# A backup controller must NEVER gate the platform graph (mirror headlamp/oauth2-proxy).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: velero
|
|
namespace: velero
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: velero
|
|
version: ">=12.0.0 <13.0.0" # Velero v1.18.x line (k8s-1.36 catch-up). CRDs pre-applied server-side (upgradeCRDs:false). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: vmware-tanzu
|
|
namespace: velero
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
# --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs
|
|
# docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) —
|
|
# `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero
|
|
# never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are
|
|
# first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/
|
|
# schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies
|
|
# UNCONDITIONALLY on install — before hooks — regardless of this flag.
|
|
# Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while
|
|
# the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the
|
|
# initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step —
|
|
# acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
|
|
# kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
|
|
upgradeCRDs: false
|
|
# --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init
|
|
# container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x).
|
|
#
|
|
# *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). ***
|
|
# v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY:
|
|
# Tagging: aws.String(o.tagging) // o.tagging == "" when config.tagging is unset
|
|
# so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement
|
|
# object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus
|
|
# Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501:
|
|
# NotImplemented: Header 'x-amz-tagging' with value '' not implemented
|
|
# Observed live on prod twice, on the write of `velero/backups/<name>/velero-backup.json`. The
|
|
# Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then
|
|
# goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that
|
|
# avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects.
|
|
# The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes
|
|
# the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so
|
|
# the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet.
|
|
# EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT
|
|
# "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.
|
|
# Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally
|
|
# rather than becoming a fifth per-environment knob. ---
|
|
initContainers:
|
|
- name: velero-plugin-for-aws
|
|
image: velero/velero-plugin-for-aws:v1.14.3-rc.2
|
|
imagePullPolicy: IfNotPresent
|
|
volumeMounts:
|
|
- mountPath: /target
|
|
name: plugins
|
|
# --- File System Backup (kopia): the node-agent DaemonSet mounts each pod's volumes and copies
|
|
# the data into S3. Storage-agnostic; no CSI-snapshot / data-mover wiring. ---
|
|
deployNodeAgent: true
|
|
snapshotsEnabled: false # no VolumeSnapshotLocation — kopia FSB is the volume-data path
|
|
configuration:
|
|
uploaderType: kopia
|
|
backupStorageLocation:
|
|
- name: default
|
|
provider: aws
|
|
# Bucket and prefix are BOTH knobs (cluster-vars), because the two environments' object
|
|
# stores are shaped differently. Dev's MinIO has a dedicated `velero` bucket and dev's
|
|
# backups sit at its ROOT, so dev renders bucket=velero with an EMPTY prefix — and the
|
|
# chart omits `objectStorage.prefix` entirely when it is empty (verified, chart 12.1.0),
|
|
# leaving dev's live BackupStorageLocation byte-unchanged. Prod's operator created ONE R2
|
|
# bucket shared with CNPG/barman, so prod renders bucket=k3s-prod-backup with prefix
|
|
# `velero` — a disjoint top-level prefix from barman's. Getting this wrong is not a
|
|
# cosmetic error: Velero ADOPTS every backup it finds under its location and TTL-DELETES
|
|
# it on retention, so a prefix that overlaps another tenant deletes that tenant's data.
|
|
bucket: ${VELERO_BACKUP_BUCKET}
|
|
prefix: "${VELERO_BACKUP_PREFIX}"
|
|
default: true
|
|
config:
|
|
# Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores
|
|
# whatever it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1`
|
|
# was correct-by-accident and broke the moment the target stopped being MinIO.
|
|
region: ${S3_BACKUP_REGION}
|
|
s3Url: ${S3_BACKUP_ENDPOINT} # cluster-vars; dev MinIO (http) or prod R2 (https)
|
|
s3ForcePathStyle: "true" # MinIO is path-style; R2 accepts path-style too
|
|
# *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
|
|
# velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums ON by
|
|
# default, so it sends `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32` on every
|
|
# PUT. R2 REJECTS those headers with HTTP 400 InvalidArgument, which would fail every
|
|
# backup AND every kopia blob upload. The empty string is the documented opt-out, and
|
|
# it must be PRESENT-but-empty rather than absent — the plugin branches on the key
|
|
# existing, and the chart does emit an empty value (verified, chart 12.1.0). Harmless
|
|
# against MinIO, which ignores it, so it is stated unconditionally rather than as a
|
|
# fourth knob. If backups start failing with InvalidArgument after a plugin bump, look
|
|
# here first.
|
|
checksumAlgorithm: ""
|
|
credentials:
|
|
# PER-ENVIRONMENT credential (2026-09-07). dev renders velero-cloud-credentials (the scoped
|
|
# MinIO key); prod renders velero-cloud-credentials-prod (the Cloudflare R2 token). BOTH
|
|
# Secrets are applied to BOTH clusters by platform/velero/kustomization.yaml — the suffix
|
|
# only chooses which one this Deployment mounts. cluster-vars: BACKUP_CREDENTIALS_SUFFIX,
|
|
# dev "" | prod "-prod". A name that resolves to a MISSING Secret does not degrade the
|
|
# backup, it wedges the pod in ContainerCreating (the volume never mounts), which is why
|
|
# the base default is the empty suffix — the unsuffixed Secret always exists.
|
|
existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX} # SOPS Secret carrying the `cloud` INI key (ns velero)
|
|
# --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
|
|
# by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
|
|
# capturing their Kubernetes resources. ---
|
|
schedules:
|
|
# (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
|
|
# defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
|
|
#
|
|
# THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out
|
|
# + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs
|
|
# (plex-media, sonarr-media, radarr-media — all three bound to the SAME export
|
|
# /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and
|
|
# their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So
|
|
# namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into
|
|
# schedule (2) would drop their real config PVCs, and leaving them here uploaded the media
|
|
# library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads).
|
|
# `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are
|
|
# called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so
|
|
# the rule tests the property rather than a name list that a future PV would slip past.
|
|
# Media on that ZFS pool has its own protection and is reconstructable; it is deliberately
|
|
# NOT cluster-DR scope. See platform/velero/volume-policies.yaml.
|
|
# RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume
|
|
# on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked
|
|
# up by opt-out again and needs its own condition added there.
|
|
daily:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: true
|
|
resourcePolicy:
|
|
kind: configmap
|
|
name: velero-volume-policies
|
|
excludedNamespaces:
|
|
- monitoring # volume DATA excluded here; resources captured by schedule (2)
|
|
- unifi
|
|
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
|
|
# PVCs are reconstructable and excluded from data backup per the sizing decision:
|
|
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
|
|
# (Grafana/Alertmanager are emptyDir — no PVC). Grafana dashboards are GitOps-provisioned.
|
|
# - unifi: the ~7.4 GiB KubeVirt VM disk (huge + reconstructable; UniFi is protected by its
|
|
# own config export, not this disk).
|
|
# Backing up their K8s objects (VM/DataVolume/Services/HelmRelease-owned CRs) still aids DR.
|
|
daily-resources-only:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: false
|
|
includedNamespaces:
|
|
- monitoring
|
|
- unifi
|
|
# Modest home-dev footprint (mirror the other platform components).
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|
|
nodeAgent:
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|