Files
home-network-k3s-impl/platform/velero/helmrelease.yaml
T
jakman85andClaude Opus 5 32d67660a6 feat(backup): move off-cluster DR to Cloudflare R2 (operator decision 2026-09-05)
"That's where backups will go for the time being until a better on-prem backup
solution is created." Two reasons beyond the (negligible) cost:

  1. Prod CANNOT reach the on-Proxmox MinIO. Prod is on the SOHO network
     192.168.2.0/24 — pfSense's WAN — while that MinIO sits inside pfSense's LAN.
     Measured from a pod inside prod: connection timed out after 15s. R2 needs no
     firewall change.
  2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's
     "off-cluster" backup shared a physical failure domain with dev — one host
     loss took the cluster AND its backups. R2 is genuinely off-site.

COST, measured rather than estimated (read straight out of the buckets):
  velero  4.0 GiB / 6,264 objects  (7d retention; 8.65 GB live -> kopia dedup)
  cnpg    211 MiB / 1,755 objects  (keycloak 66Mi/331, nextcloud 87Mi/1329,
                                    semaphore 58Mi/95; ~7d of a 30d window)
Both clusters project to ~13-15 GiB and ~30k writes/month => pennies on R2, and
free egress means restore REHEARSALS cost nothing, so R-STATE-03 has no financial
disincentive to stay proven. Wasabi/Hetzner are 50-200x worse here purely because
of 1 TB minimums; Glacier tiers are a trap (90-day minimum billing against a
7-day Velero retention plus early-delete fees on 30-day WAL pruning).

*** THE R2 GOTCHA, WIRED UP FRONT: checksumAlgorithm: "" ***
velero-plugin-for-aws v1.14.2 bundles an AWS SDK that sends
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT. R2 REJECTS
those with HTTP 400 InvalidArgument, so every backup and every kopia blob upload
would fail. Set unconditionally — MinIO ignores it, so it needs no fork.

`region` was hardcoded us-east-1 inside the Velero HelmRelease: correct-by-
accident for MinIO (which ignores it) and WRONG for R2 (which demands exactly
`auto`). Now the S3_BACKUP_REGION knob.

PREFIXES ARE NOW SYMMETRIC (dev/prod), reversing the reasoning from two commits
ago. While the target was MinIO, dev HAD to keep the bucket root or it would
orphan the backups its proven restore depends on. R2 starts BOTH clusters on an
empty bucket, so there is no history to orphan — and leaving the bucket root
unused means a misrendered prefix lands somewhere obviously wrong instead of
silently colliding with a real environment.

NEITHER OVERLAY PINS THE ENDPOINT ANY MORE — a deliberate reversal of this
repo's "prod must never inherit base" rule. That rule exists because base
defaults are dev's LAN addresses; R2 is a single environment-INDEPENDENT endpoint,
so under R-ENV-02 it belongs in base, and two copies of one URL is a rotation
waiting to drift. Isolation comes from the path knobs, not the URL. Caught while
verifying: dev's overlay DID still pin MinIO, so dev would have taken the new
`dev` prefix while writing to the old endpoint — orphaning its MinIO backups for
no benefit. That is why the endpoint and the prefixes must move in ONE commit.

*** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts empty:
Velero writes a fresh kopia repo, CNPG takes new base backups, the 30-day Barman
window rebuilds from zero. The MinIO copy is NOT deleted and stays restorable by
pointing the endpoint back. Do not merge while dev's proven restore is the only
recovery path you rely on.

STILL BLOCKED ON THE OPERATOR: the endpoint is the loud placeholder
https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com. Creating the R2 buckets and
an API token needs Cloudflare account access I do not have (the token in this
repo is Zone:DNS scoped). Velero+R2 and CNPG-barman+R2 are also UNPROVEN here and
must be verified by an actual backup AND restore, not just an upload.

VERIFIED: full CI gate PASS. Rendered both overlays — dev dev/s3://cnpg/dev,
prod prod/s3://cnpg/prod, region auto, one shared endpoint.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:29:22 -05:00

151 lines
8.1 KiB
YAML

---
# Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
#
# TARGET: the MinIO S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars; dev http://192.168.1.9:9000 on
# Proxmox — plain HTTP, path-style), bucket
# `velero`. region us-east-1 is a placeholder MinIO ignores (S3 clients still require one).
#
# WHY THIS SHAPE:
# - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
# - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
# NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
# Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
# - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero) — the
# SCOPED MinIO key limited to the velero+cnpg buckets. Kept in sync with the staging source of
# truth platform/backup/minio-credentials.sops.yaml and the Proxmox host ansible-vault.
#
# VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
# cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
#
# NON-GATING TIER: reconciled by clusters/base/backup.yaml (dependsOn platform-config, wait:false).
# A backup controller must NEVER gate the platform graph (mirror headlamp/oauth2-proxy).
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: velero
namespace: velero
spec:
interval: 30m
chart:
spec:
chart: velero
version: ">=12.0.0 <13.0.0" # Velero v1.18.x line (k8s-1.36 catch-up). CRDs pre-applied server-side (upgradeCRDs:false). Renovate maintains
sourceRef:
kind: HelmRepository
name: vmware-tanzu
namespace: velero
install:
remediation:
retries: 3
upgrade:
remediation:
retries: 3
values:
# --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs
# docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) —
# `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero
# never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are
# first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/
# schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies
# UNCONDITIONALLY on install — before hooks — regardless of this flag.
# Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while
# the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the
# initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step —
# acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
# kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
upgradeCRDs: false
# --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container.
# Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). ---
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:v1.14.2
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /target
name: plugins
# --- File System Backup (kopia): the node-agent DaemonSet mounts each pod's volumes and copies
# the data into S3. Storage-agnostic; no CSI-snapshot / data-mover wiring. ---
deployNodeAgent: true
snapshotsEnabled: false # no VolumeSnapshotLocation — kopia FSB is the volume-data path
configuration:
uploaderType: kopia
backupStorageLocation:
- name: default
provider: aws
bucket: velero
# PER-ENVIRONMENT PREFIX (cluster-vars). Empty on dev = the bucket root it already uses;
# `prod` on prod. Without it both clusters share one BackupStorageLocation — and Velero
# SYNCS backups it finds there into itself as Backup CRs and GARBAGE-COLLECTS them when
# their TTL expires, so prod would adopt and then delete dev's backups. Both clusters also
# run schedules named `daily`/`daily-resources-only`, so even the object keys would collide.
prefix: ${VELERO_BACKUP_PREFIX}
default: true
config:
# Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores whatever
# it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1` silently broke
# the moment the target stopped being MinIO.
region: ${S3_BACKUP_REGION}
s3Url: ${S3_BACKUP_ENDPOINT} # cluster-vars; R2 (https) or a MinIO/versitygw (http)
s3ForcePathStyle: "true" # MinIO is path-style; R2 accepts path-style too
# *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
# velero-plugin-for-aws v1.14+ bundles an AWS SDK that sends `x-amz-sdk-checksum-algorithm`
# / `x-amz-checksum-crc32` on every PUT. R2 REJECTS those headers (HTTP 400
# "InvalidArgument"), so every backup and every kopia blob upload fails. Setting this to the
# empty string disables the SDK's flexible-checksum behaviour. It is harmless against MinIO,
# which simply ignores it — so it is stated unconditionally rather than per-environment.
# If backups start failing with InvalidArgument after a plugin bump, look here first.
checksumAlgorithm: ""
credentials:
existingSecret: velero-cloud-credentials # SOPS Secret carrying the `cloud` INI key (ns velero)
# --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
# by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
# capturing their Kubernetes resources. ---
schedules:
# (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
# defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
daily:
disabled: false
schedule: "${VELERO_BACKUP_SCHEDULE}"
useOwnerReferencesInBackup: false
template:
ttl: "${VELERO_RETENTION}"
storageLocation: default
snapshotVolumes: false
defaultVolumesToFsBackup: true
excludedNamespaces:
- monitoring # volume DATA excluded here; resources captured by schedule (2)
- unifi
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
# PVCs are reconstructable and excluded from data backup per the sizing decision:
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
# (Grafana/Alertmanager are emptyDir — no PVC). Grafana dashboards are GitOps-provisioned.
# - unifi: the ~7.4 GiB KubeVirt VM disk (huge + reconstructable; UniFi is protected by its
# own config export, not this disk).
# Backing up their K8s objects (VM/DataVolume/Services/HelmRelease-owned CRs) still aids DR.
daily-resources-only:
disabled: false
schedule: "${VELERO_BACKUP_SCHEDULE}"
useOwnerReferencesInBackup: false
template:
ttl: "${VELERO_RETENTION}"
storageLocation: default
snapshotVolumes: false
defaultVolumesToFsBackup: false
includedNamespaces:
- monitoring
- unifi
# Modest home-dev footprint (mirror the other platform components).
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 512Mi
nodeAgent:
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 512Mi