jakman85andClaude Opus 5
161430d915
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
176 lines
10 KiB
YAML
176 lines
10 KiB
YAML
---
|
|
# Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
|
|
#
|
|
# TARGET: the S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars) at
|
|
# s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX}, in region ${S3_BACKUP_REGION}. Four knobs,
|
|
# because the two environments' object stores differ in shape as well as address:
|
|
# dev http://192.168.1.9:9000 bucket `velero`, NO prefix, region us-east-1 (on-Proxmox MinIO,
|
|
# plain HTTP, path-style; MinIO ignores the region)
|
|
# prod https://<acct>.r2.cloudflarestorage.com bucket `k3s-prod-backup`, prefix `velero`,
|
|
# region `auto` (Cloudflare R2; prod cannot route to that MinIO,
|
|
# and R2 demands exactly `auto`)
|
|
# Prod's bucket is SHARED with CNPG/barman (the operator created one bucket), so the `velero` prefix
|
|
# is what keeps this controller's retention sweep away from the Postgres WAL stream. See the
|
|
# "OBJECT-STORE TENANCY" block in clusters/base/cluster-vars.yaml.
|
|
#
|
|
# WHY THIS SHAPE:
|
|
# - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
|
|
# - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
|
|
# NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
|
|
# Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
|
|
# - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero). ONE
|
|
# Secret serves both environments, and it now carries the Cloudflare R2 API token (written by
|
|
# platform/backup/set-r2-credentials.sh, 2026-09-07) rather than the scoped MinIO key. That is
|
|
# safe only because the R2 token is bucket-scoped to k3s-prod-backup, which dev never names:
|
|
# dev's Velero would fail loudly against its MinIO endpoint with an R2 key, it could not
|
|
# silently write somewhere unexpected. Rotating either target means re-running that script.
|
|
#
|
|
# VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
|
|
# cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
|
|
#
|
|
# NON-GATING TIER: reconciled by clusters/base/backup.yaml (dependsOn platform-config, wait:false).
|
|
# A backup controller must NEVER gate the platform graph (mirror headlamp/oauth2-proxy).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: velero
|
|
namespace: velero
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: velero
|
|
version: ">=12.0.0 <13.0.0" # Velero v1.18.x line (k8s-1.36 catch-up). CRDs pre-applied server-side (upgradeCRDs:false). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: vmware-tanzu
|
|
namespace: velero
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
# --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs
|
|
# docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) —
|
|
# `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero
|
|
# never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are
|
|
# first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/
|
|
# schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies
|
|
# UNCONDITIONALLY on install — before hooks — regardless of this flag.
|
|
# Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while
|
|
# the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the
|
|
# initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step —
|
|
# acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
|
|
# kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
|
|
upgradeCRDs: false
|
|
# --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container.
|
|
# Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). ---
|
|
initContainers:
|
|
- name: velero-plugin-for-aws
|
|
image: velero/velero-plugin-for-aws:v1.14.2
|
|
imagePullPolicy: IfNotPresent
|
|
volumeMounts:
|
|
- mountPath: /target
|
|
name: plugins
|
|
# --- File System Backup (kopia): the node-agent DaemonSet mounts each pod's volumes and copies
|
|
# the data into S3. Storage-agnostic; no CSI-snapshot / data-mover wiring. ---
|
|
deployNodeAgent: true
|
|
snapshotsEnabled: false # no VolumeSnapshotLocation — kopia FSB is the volume-data path
|
|
configuration:
|
|
uploaderType: kopia
|
|
backupStorageLocation:
|
|
- name: default
|
|
provider: aws
|
|
# Bucket and prefix are BOTH knobs (cluster-vars), because the two environments' object
|
|
# stores are shaped differently. Dev's MinIO has a dedicated `velero` bucket and dev's
|
|
# backups sit at its ROOT, so dev renders bucket=velero with an EMPTY prefix — and the
|
|
# chart omits `objectStorage.prefix` entirely when it is empty (verified, chart 12.1.0),
|
|
# leaving dev's live BackupStorageLocation byte-unchanged. Prod's operator created ONE R2
|
|
# bucket shared with CNPG/barman, so prod renders bucket=k3s-prod-backup with prefix
|
|
# `velero` — a disjoint top-level prefix from barman's. Getting this wrong is not a
|
|
# cosmetic error: Velero ADOPTS every backup it finds under its location and TTL-DELETES
|
|
# it on retention, so a prefix that overlaps another tenant deletes that tenant's data.
|
|
bucket: ${VELERO_BACKUP_BUCKET}
|
|
prefix: "${VELERO_BACKUP_PREFIX}"
|
|
default: true
|
|
config:
|
|
# Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores
|
|
# whatever it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1`
|
|
# was correct-by-accident and broke the moment the target stopped being MinIO.
|
|
region: ${S3_BACKUP_REGION}
|
|
s3Url: ${S3_BACKUP_ENDPOINT} # cluster-vars; dev MinIO (http) or prod R2 (https)
|
|
s3ForcePathStyle: "true" # MinIO is path-style; R2 accepts path-style too
|
|
# *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
|
|
# velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums ON by
|
|
# default, so it sends `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32` on every
|
|
# PUT. R2 REJECTS those headers with HTTP 400 InvalidArgument, which would fail every
|
|
# backup AND every kopia blob upload. The empty string is the documented opt-out, and
|
|
# it must be PRESENT-but-empty rather than absent — the plugin branches on the key
|
|
# existing, and the chart does emit an empty value (verified, chart 12.1.0). Harmless
|
|
# against MinIO, which ignores it, so it is stated unconditionally rather than as a
|
|
# fourth knob. If backups start failing with InvalidArgument after a plugin bump, look
|
|
# here first.
|
|
checksumAlgorithm: ""
|
|
credentials:
|
|
# PER-ENVIRONMENT credential (2026-09-07). dev renders velero-cloud-credentials (the scoped
|
|
# MinIO key); prod renders velero-cloud-credentials-prod (the Cloudflare R2 token). BOTH
|
|
# Secrets are applied to BOTH clusters by platform/velero/kustomization.yaml — the suffix
|
|
# only chooses which one this Deployment mounts. cluster-vars: BACKUP_CREDENTIALS_SUFFIX,
|
|
# dev "" | prod "-prod". A name that resolves to a MISSING Secret does not degrade the
|
|
# backup, it wedges the pod in ContainerCreating (the volume never mounts), which is why
|
|
# the base default is the empty suffix — the unsuffixed Secret always exists.
|
|
existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX} # SOPS Secret carrying the `cloud` INI key (ns velero)
|
|
# --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
|
|
# by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
|
|
# capturing their Kubernetes resources. ---
|
|
schedules:
|
|
# (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
|
|
# defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
|
|
daily:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: true
|
|
excludedNamespaces:
|
|
- monitoring # volume DATA excluded here; resources captured by schedule (2)
|
|
- unifi
|
|
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
|
|
# PVCs are reconstructable and excluded from data backup per the sizing decision:
|
|
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
|
|
# (Grafana/Alertmanager are emptyDir — no PVC). Grafana dashboards are GitOps-provisioned.
|
|
# - unifi: the ~7.4 GiB KubeVirt VM disk (huge + reconstructable; UniFi is protected by its
|
|
# own config export, not this disk).
|
|
# Backing up their K8s objects (VM/DataVolume/Services/HelmRelease-owned CRs) still aids DR.
|
|
daily-resources-only:
|
|
disabled: false
|
|
schedule: "${VELERO_BACKUP_SCHEDULE}"
|
|
useOwnerReferencesInBackup: false
|
|
template:
|
|
ttl: "${VELERO_RETENTION}"
|
|
storageLocation: default
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: false
|
|
includedNamespaces:
|
|
- monitoring
|
|
- unifi
|
|
# Modest home-dev footprint (mirror the other platform components).
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|
|
nodeAgent:
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
memory: 512Mi
|