jakman85andClaude Opus 5
32d67660a6
"That's where backups will go for the time being until a better on-prem backup
solution is created." Two reasons beyond the (negligible) cost:
1. Prod CANNOT reach the on-Proxmox MinIO. Prod is on the SOHO network
192.168.2.0/24 — pfSense's WAN — while that MinIO sits inside pfSense's LAN.
Measured from a pod inside prod: connection timed out after 15s. R2 needs no
firewall change.
2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's
"off-cluster" backup shared a physical failure domain with dev — one host
loss took the cluster AND its backups. R2 is genuinely off-site.
COST, measured rather than estimated (read straight out of the buckets):
velero 4.0 GiB / 6,264 objects (7d retention; 8.65 GB live -> kopia dedup)
cnpg 211 MiB / 1,755 objects (keycloak 66Mi/331, nextcloud 87Mi/1329,
semaphore 58Mi/95; ~7d of a 30d window)
Both clusters project to ~13-15 GiB and ~30k writes/month => pennies on R2, and
free egress means restore REHEARSALS cost nothing, so R-STATE-03 has no financial
disincentive to stay proven. Wasabi/Hetzner are 50-200x worse here purely because
of 1 TB minimums; Glacier tiers are a trap (90-day minimum billing against a
7-day Velero retention plus early-delete fees on 30-day WAL pruning).
*** THE R2 GOTCHA, WIRED UP FRONT: checksumAlgorithm: "" ***
velero-plugin-for-aws v1.14.2 bundles an AWS SDK that sends
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT. R2 REJECTS
those with HTTP 400 InvalidArgument, so every backup and every kopia blob upload
would fail. Set unconditionally — MinIO ignores it, so it needs no fork.
`region` was hardcoded us-east-1 inside the Velero HelmRelease: correct-by-
accident for MinIO (which ignores it) and WRONG for R2 (which demands exactly
`auto`). Now the S3_BACKUP_REGION knob.
PREFIXES ARE NOW SYMMETRIC (dev/prod), reversing the reasoning from two commits
ago. While the target was MinIO, dev HAD to keep the bucket root or it would
orphan the backups its proven restore depends on. R2 starts BOTH clusters on an
empty bucket, so there is no history to orphan — and leaving the bucket root
unused means a misrendered prefix lands somewhere obviously wrong instead of
silently colliding with a real environment.
NEITHER OVERLAY PINS THE ENDPOINT ANY MORE — a deliberate reversal of this
repo's "prod must never inherit base" rule. That rule exists because base
defaults are dev's LAN addresses; R2 is a single environment-INDEPENDENT endpoint,
so under R-ENV-02 it belongs in base, and two copies of one URL is a rotation
waiting to drift. Isolation comes from the path knobs, not the URL. Caught while
verifying: dev's overlay DID still pin MinIO, so dev would have taken the new
`dev` prefix while writing to the old endpoint — orphaning its MinIO backups for
no benefit. That is why the endpoint and the prefixes must move in ONE commit.
*** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts empty:
Velero writes a fresh kopia repo, CNPG takes new base backups, the 30-day Barman
window rebuilds from zero. The MinIO copy is NOT deleted and stays restorable by
pointing the endpoint back. Do not merge while dev's proven restore is the only
recovery path you rely on.
STILL BLOCKED ON THE OPERATOR: the endpoint is the loud placeholder
https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com. Creating the R2 buckets and
an API token needs Cloudflare account access I do not have (the token in this
repo is Zone:DNS scoped). Velero+R2 and CNPG-barman+R2 are also UNPROVEN here and
must be verified by an actual backup AND restore, not just an upload.
VERIFIED: full CI gate PASS. Rendered both overlays — dev dev/s3://cnpg/dev,
prod prod/s3://cnpg/prod, region auto, one shared endpoint.
Refs R-STATE-03, R-ENV-02, R-IDP-05.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/velero
Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.
- Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-03
- Owner: WS-STORAGE
What this is (LIVE)
Flux HelmRelease (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox MinIO S3 target:
- Endpoint:
http://192.168.1.9:9000(plain HTTP, path-style, regionus-east-1placeholder) - BucketStorageLocation
default: provideraws, bucketvelero,s3ForcePathStyle=true. - AWS plugin (
velero/velero-plugin-for-aws) loaded as an init container. - File System Backup (kopia):
deployNodeAgent=true,uploaderType=kopia,snapshotsEnabled=false— storage-agnostic PVC backup, no CSI snapshot / data-mover. - Credentials:
velero-cloud-credentials— a SOPS Secret (velero-cloud.sops.yaml) holding thecloudINI key, the scoped MinIO key limited to thevelero+cnpgbuckets. Kept in sync with the staging source of truthplatform/backup/minio-credentials.sops.yamland the Proxmox host ansible-vault.
Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml
(dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.
Backup scope (what's included vs excluded)
Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):
| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
daily |
ALL except monitoring, unifi |
all PVC data (defaultVolumesToFsBackup:true, opt-out) |
daily-resources-only |
monitoring, unifi |
none (defaultVolumesToFsBackup:false) — resources only |
Excluded volume data (reconstructable, per the sizing decision):
monitoring— Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager areemptyDir(no PVC); Grafana dashboards are GitOps-provisioned.unifi— the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.
Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the cnpg bucket — see the DB manifests.)
Post-deploy (PM)
velero backup-location get→defaultAvailable.velero backup create --from-schedule daily→ objects land unders3://velero/.- DR drill (R-STATE-03, Phase-4 gate):
velero restore create --from-backup <name>into a scratch namespace and verify PVC data restores via the node-agent.