jakman85andClaude Opus 5
a1cc0a5556
Supersedes 05a8094, which fixed the right bug in the wrong layer.
That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."
It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with
error to expose PVB: error getting client pod runbooks-...: pods "..." not found
because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.
So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.
The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:
control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.
The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
235 lines
15 KiB
YAML
235 lines
15 KiB
YAML
---
|
|
# Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
|
|
#
|
|
# TARGET: the S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars) at
|
|
# s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX}, in region ${S3_BACKUP_REGION}. Four knobs,
|
|
# because the two environments' object stores differ in shape as well as address:
|
|
# dev http://192.168.1.9:9000 bucket `velero`, NO prefix, region us-east-1 (on-Proxmox MinIO,
|
|
# plain HTTP, path-style; MinIO ignores the region)
|
|
# prod https://<acct>.r2.cloudflarestorage.com bucket `k3s-prod-backup`, prefix `velero`,
|
|
# region `auto` (Cloudflare R2; prod cannot route to that MinIO,
|
|
# and R2 demands exactly `auto`)
|
|
# Prod's bucket is SHARED with CNPG/barman (the operator created one bucket), so the `velero` prefix
|
|
# is what keeps this controller's retention sweep away from the Postgres WAL stream. See the
|
|
# "OBJECT-STORE TENANCY" block in clusters/base/cluster-vars.yaml.
|
|
#
|
|
# WHY THIS SHAPE:
|
|
# - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
|
|
# - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
|
|
# NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
|
|
# Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
|
|
# - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero). ONE
|
|
# Secret serves both environments, and it now carries the Cloudflare R2 API token (written by
|
|
# platform/backup/set-r2-credentials.sh, 2026-09-07) rather than the scoped MinIO key. That is
|
|
# safe only because the R2 token is bucket-scoped to k3s-prod-backup, which dev never names:
|
|
# dev's Velero would fail loudly against its MinIO endpoint with an R2 key, it could not
|
|
# silently write somewhere unexpected. Rotating either target means re-running that script.
|
|
#
|
|
# VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
|
|
# cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
|
|
#
|
|
# NON-GATING TIER: reconciled by clusters/base/backup.yaml (dependsOn platform-config, wait:false).
|
|
# A backup controller must NEVER gate the platform graph (mirror headlamp/oauth2-proxy).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: velero
|
|
namespace: velero
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: velero
|
|
version: ">=12.0.0 <13.0.0" # Velero v1.18.x line (k8s-1.36 catch-up). CRDs pre-applied server-side (upgradeCRDs:false). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: vmware-tanzu
|
|
namespace: velero
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
# --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs
|
|
# docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) —
|
|
# `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero
|
|
# never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are
|
|
# first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/
|
|
# schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies
|
|
# UNCONDITIONALLY on install — before hooks — regardless of this flag.
|
|
# Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while
|
|
# the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the
|
|
# initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step —
|
|
# acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
|
|
# kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
|
|
upgradeCRDs: false
|
|
# --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init
|
|
# container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x).
|
|
#
|
|
# *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). ***
|
|
# v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY:
|
|
# Tagging: aws.String(o.tagging) // o.tagging == "" when config.tagging is unset
|
|
# so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement
|
|
# object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus
|
|
# Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501:
|
|
# NotImplemented: Header 'x-amz-tagging' with value '' not implemented
|
|
# Observed live on prod twice, on the write of `velero/backups/<name>/velero-backup.json`. The
|
|
# Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then
|
|
# goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that
|
|
# avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects.
|
|
# The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes
|
|
# the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so
|
|
# the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet.
|
|
# EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT
|
|
# "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.
|
|
# Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally
|
|
# rather than becoming a fifth per-environment knob. ---
|
|
initContainers:
|
|
- name: velero-plugin-for-aws
|
|
image: velero/velero-plugin-for-aws:v1.14.3-rc.2
|
|
imagePullPolicy: IfNotPresent
|
|
volumeMounts:
|
|
- mountPath: /target
|
|
name: plugins
|
|
# --- File System Backup (kopia): the node-agent DaemonSet mounts each pod's volumes and copies
|
|
# the data into S3. Storage-agnostic; no CSI-snapshot / data-mover wiring. ---
|
|
deployNodeAgent: true
|
|
snapshotsEnabled: false # no VolumeSnapshotLocation — kopia FSB is the volume-data path
|
|
configuration:
|
|
uploaderType: kopia
|
|
backupStorageLocation:
|
|
- name: default
|
|
provider: aws
|
|
# Bucket and prefix are BOTH knobs (cluster-vars), because the two environments' object
|
|
# stores are shaped differently. Dev's MinIO has a dedicated `velero` bucket and dev's
|
|
# backups sit at its ROOT, so dev renders bucket=velero with an EMPTY prefix — and the
|
|
# chart omits `objectStorage.prefix` entirely when it is empty (verified, chart 12.1.0),
|
|
# leaving dev's live BackupStorageLocation byte-unchanged. Prod's operator created ONE R2
|
|
# bucket shared with CNPG/barman, so prod renders bucket=k3s-prod-backup with prefix
|
|
# `velero` — a disjoint top-level prefix from barman's. Getting this wrong is not a
|
|
# cosmetic error: Velero ADOPTS every backup it finds under its location and TTL-DELETES
|
|
# it on retention, so a prefix that overlaps another tenant deletes that tenant's data.
|
|
bucket: ${VELERO_BACKUP_BUCKET}
|
|
prefix: "${VELERO_BACKUP_PREFIX}"
|
|
default: true
|
|
config:
|
|
# Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores
|
|
# whatever it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1`
|
|
# was correct-by-accident and broke the moment the target stopped being MinIO.
|
|
region: ${S3_BACKUP_REGION}
|
|
s3Url: ${S3_BACKUP_ENDPOINT} # cluster-vars; dev MinIO (http) or prod R2 (https)
|
|
s3ForcePathStyle: "true" # MinIO is path-style; R2 accepts path-style too
|
|
# *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
|
|
# velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums ON by
|
|
# default, so it sends `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32` on every
|
|
# PUT. R2 REJECTS those headers with HTTP 400 InvalidArgument, which would fail every
|
|
# backup AND every kopia blob upload. The empty string is the documented opt-out, and
|
|
# it must be PRESENT-but-empty rather than absent — the plugin branches on the key
|
|
# existing, and the chart does emit an empty value (verified, chart 12.1.0). Harmless
|
|
# against MinIO, which ignores it, so it is stated unconditionally rather than as a
|
|
# fourth knob. If backups start failing with InvalidArgument after a plugin bump, look
|
|
# here first.
|
|
checksumAlgorithm: ""
|
|
credentials:
|
|
# PER-ENVIRONMENT credential (2026-09-07). dev renders velero-cloud-credentials (the scoped
|
|
# MinIO key); prod renders velero-cloud-credentials-prod (the Cloudflare R2 token). BOTH
|
|
# Secrets are applied to BOTH clusters by platform/velero/kustomization.yaml — the suffix
|
|
# only chooses which one this Deployment mounts. cluster-vars: BACKUP_CREDENTIALS_SUFFIX,
|
|
# dev "" | prod "-prod". A name that resolves to a MISSING Secret does not degrade the
|
|
# backup, it wedges the pod in ContainerCreating (the volume never mounts), which is why
|
|
# the base default is the empty suffix — the unsuffixed Secret always exists.
|
|
existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX} # SOPS Secret carrying the `cloud` INI key (ns velero)
|
|
# --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
|
|
# by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
|
|
# capturing their Kubernetes resources. ---
|
|
schedules:
|
|
# (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
|
|
# defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
|
|
#
|
|
# THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out
|
|
# + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs
|
|
# (plex-media, sonarr-media, radarr-media — all three bound to the SAME export
|
|
# /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and
|
|
# their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So
|
|
# namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into
|
|
# schedule (2) would drop their real config PVCs, and leaving them here uploaded the media
|
|
# library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads).
|
|
# `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are
|
|
# called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so
|
|
# the rule tests the property rather than a name list that a future PV would slip past.
|
|
# Media on that ZFS pool has its own protection and is reconstructable; it is deliberately
|
|
# NOT cluster-DR scope. See platform/velero/volume-policies.yaml.
|
|
# RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume
|
|
# on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked
|
|
# up by opt-out again and needs its own condition added there.
|
|
daily:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: true
|
|
resourcePolicy:
|
|
kind: configmap
|
|
name: velero-volume-policies
|
|
excludedNamespaces:
|
|
- monitoring # volume DATA excluded here; resources captured by schedule (2)
|
|
- unifi
|
|
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
|
|
# PVCs are reconstructable and excluded from data backup per the sizing decision:
|
|
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
|
|
# (Grafana/Alertmanager are emptyDir — no PVC). Grafana dashboards are GitOps-provisioned.
|
|
# - unifi: the ~7.4 GiB KubeVirt VM disk (huge + reconstructable; UniFi is protected by its
|
|
# own config export, not this disk).
|
|
# Backing up their K8s objects (VM/DataVolume/Services/HelmRelease-owned CRs) still aids DR.
|
|
daily-resources-only:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: false
|
|
includedNamespaces:
|
|
- monitoring
|
|
- unifi
|
|
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
|
|
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
|
|
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,
|
|
# because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor
|
|
# existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` →
|
|
# no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp
|
|
# were never scraped, so a nightly backup could fail — or stop running — indefinitely with no
|
|
# alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of
|
|
# the backup path, not an observability nice-to-have.
|
|
# The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1
|
|
# (verified: `helm template` emits neither object without --api-versions, and emits both with
|
|
# it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities
|
|
# from the API server. No additionalLabels needed: platform/monitoring sets
|
|
# serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so
|
|
# Prometheus selects every ServiceMonitor/PodMonitor regardless of labels.
|
|
metrics:
|
|
serviceMonitor:
|
|
enabled: true # velero server: backup/restore success+failure counters
|
|
nodeAgentPodMonitor:
|
|
enabled: true # node-agent: kopia FSB (PodVolumeBackup) progress + failures
|
|
# Modest home-dev footprint (mirror the other platform components).
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|
|
nodeAgent:
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|