feat(backup): move off-cluster DR to Cloudflare R2 (operator decision 2026-09-05)
"That's where backups will go for the time being until a better on-prem backup
solution is created." Two reasons beyond the (negligible) cost:
1. Prod CANNOT reach the on-Proxmox MinIO. Prod is on the SOHO network
192.168.2.0/24 — pfSense's WAN — while that MinIO sits inside pfSense's LAN.
Measured from a pod inside prod: connection timed out after 15s. R2 needs no
firewall change.
2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's
"off-cluster" backup shared a physical failure domain with dev — one host
loss took the cluster AND its backups. R2 is genuinely off-site.
COST, measured rather than estimated (read straight out of the buckets):
velero 4.0 GiB / 6,264 objects (7d retention; 8.65 GB live -> kopia dedup)
cnpg 211 MiB / 1,755 objects (keycloak 66Mi/331, nextcloud 87Mi/1329,
semaphore 58Mi/95; ~7d of a 30d window)
Both clusters project to ~13-15 GiB and ~30k writes/month => pennies on R2, and
free egress means restore REHEARSALS cost nothing, so R-STATE-03 has no financial
disincentive to stay proven. Wasabi/Hetzner are 50-200x worse here purely because
of 1 TB minimums; Glacier tiers are a trap (90-day minimum billing against a
7-day Velero retention plus early-delete fees on 30-day WAL pruning).
*** THE R2 GOTCHA, WIRED UP FRONT: checksumAlgorithm: "" ***
velero-plugin-for-aws v1.14.2 bundles an AWS SDK that sends
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT. R2 REJECTS
those with HTTP 400 InvalidArgument, so every backup and every kopia blob upload
would fail. Set unconditionally — MinIO ignores it, so it needs no fork.
`region` was hardcoded us-east-1 inside the Velero HelmRelease: correct-by-
accident for MinIO (which ignores it) and WRONG for R2 (which demands exactly
`auto`). Now the S3_BACKUP_REGION knob.
PREFIXES ARE NOW SYMMETRIC (dev/prod), reversing the reasoning from two commits
ago. While the target was MinIO, dev HAD to keep the bucket root or it would
orphan the backups its proven restore depends on. R2 starts BOTH clusters on an
empty bucket, so there is no history to orphan — and leaving the bucket root
unused means a misrendered prefix lands somewhere obviously wrong instead of
silently colliding with a real environment.
NEITHER OVERLAY PINS THE ENDPOINT ANY MORE — a deliberate reversal of this
repo's "prod must never inherit base" rule. That rule exists because base
defaults are dev's LAN addresses; R2 is a single environment-INDEPENDENT endpoint,
so under R-ENV-02 it belongs in base, and two copies of one URL is a rotation
waiting to drift. Isolation comes from the path knobs, not the URL. Caught while
verifying: dev's overlay DID still pin MinIO, so dev would have taken the new
`dev` prefix while writing to the old endpoint — orphaning its MinIO backups for
no benefit. That is why the endpoint and the prefixes must move in ONE commit.
*** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts empty:
Velero writes a fresh kopia repo, CNPG takes new base backups, the 30-day Barman
window rebuilds from zero. The MinIO copy is NOT deleted and stays restorable by
pointing the endpoint back. Do not merge while dev's proven restore is the only
recovery path you rely on.
STILL BLOCKED ON THE OPERATOR: the endpoint is the loud placeholder
https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com. Creating the R2 buckets and
an API token needs Cloudflare account access I do not have (the token in this
repo is Zone:DNS scoped). Velero+R2 and CNPG-barman+R2 are also UNPROVEN here and
must be verified by an actual backup AND restore, not just an upload.
VERIFIED: full CI gate PASS. Rendered both overlays — dev dev/s3://cnpg/dev,
prod prod/s3://cnpg/prod, region auto, one shared endpoint.
Refs R-STATE-03, R-ENV-02, R-IDP-05.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
9063e53ce2
commit
32d67660a6
4 files changed
+94
-47
No files matched your search
@@ -154,11 +154,29 @@ data:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
||||
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
||||
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
||||
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
|
||||
# Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
|
||||
# backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
|
||||
#
|
||||
# OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
|
||||
# being until a better on-prem backup solution is created". Two reasons beyond cost:
|
||||
# 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
|
||||
# pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
|
||||
# 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
|
||||
# physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
|
||||
# genuinely off-site.
|
||||
# Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
|
||||
# 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
|
||||
# stay free and R-STATE-03 has no cost disincentive to stay proven.
|
||||
#
|
||||
# The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
|
||||
# means pointing this knob back at http://192.168.1.9:9000 temporarily.
|
||||
S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
|
||||
|
||||
# S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
|
||||
# This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
|
||||
# MinIO and wrong for R2, which is why it is a knob now.
|
||||
S3_BACKUP_REGION: "auto"
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
@@ -227,12 +245,23 @@ data:
|
||||
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
|
||||
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
|
||||
#
|
||||
# THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE, and must stay that way: dev's existing
|
||||
# backups live at the bucket root / at `s3://cnpg`. Giving dev a prefix would ORPHAN the very
|
||||
# backups its proven restore depends on — the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays
|
||||
# `default`. Prod, which has no history to preserve, takes the namespaced paths.
|
||||
VELERO_BACKUP_PREFIX: "" # dev: bucket root (existing layout) | prod: "prod"
|
||||
CNPG_BACKUP_PATH: "s3://cnpg" # dev: bucket root (existing layout) | prod: "s3://cnpg/prod"
|
||||
# Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
|
||||
# worth explaining, because the earlier reasoning was the opposite:
|
||||
#
|
||||
# While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
|
||||
# backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
|
||||
# restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
|
||||
#
|
||||
# Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
|
||||
# reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
|
||||
# free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
|
||||
# colliding with a real environment's backups.
|
||||
#
|
||||
# *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
|
||||
# the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
|
||||
# one change; do not split the commit.
|
||||
VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod"
|
||||
CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod
|
||||
|
||||
# --- placement ---
|
||||
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
||||
|
||||
Reference in new issue
Block a user