feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock

Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap.
Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False,
"unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because
archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK.
Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata
PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills,
the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana,
Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain
the whole accumulated backlog in one pass.

THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way.

The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup
tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix`
as separate fields, while CNPG's barmanObjectStore.destinationPath is one full
s3:// URL whose first path segment IS the bucket. So:

  S3_BACKUP_ENDPOINT     dev http://192.168.1.9:9000   prod https://<acct>.r2...
  S3_BACKUP_REGION       dev us-east-1                 prod auto
  VELERO_BACKUP_BUCKET   dev velero                    prod k3s-prod-backup
  VELERO_BACKUP_PREFIX   dev ""                        prod velero
  CNPG_BACKUP_PATH       dev s3://cnpg                 prod s3://k3s-prod-backup/barman

There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives
inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a
rotation waiting to drift. Three knobs, each shaped for its own consumer, and no
two of them can contradict each other.

TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL
prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/
— with the bucket ROOT left empty so a misrendered prefix lands somewhere
obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS
every backup under its location and TTL-DELETES it on retention, so an overlapping
prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL
into one prefix corrupts the chain outright. Dev↔prod isolation is doubly
independent — different endpoint AND different bucket names — so neither cluster
can reach the other's objects even with a mis-substituted prefix.

DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base
defaults are dev's LIVE values (verified against the running cluster: BSL
bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters
ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now
restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated —
a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on.
The empty-prefix case was the one real risk, and it is settled empirically: with
prefix "" the chart omits objectStorage.prefix from the rendered
BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is
unchanged, not merely equivalent.

R2's TWO GOTCHAS, both wired up front:
  - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"},
    which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded
    us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong
    for anything else — so it is a knob now.
  - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with
    flexible checksums on by default and sends x-amz-sdk-checksum-algorithm /
    x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400
    InvalidArgument, failing every backup and every kopia blob upload. It must be
    PRESENT-but-empty (the plugin branches on the key existing), and the chart does
    emit an empty value — verified by rendering it. Harmless against MinIO, so it
    is unconditional rather than a fourth knob.

WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is
owned by field manager `flux` (verified: kustomize-controller's managedFields do
not include f:suspend), so server-side apply will not strip it, and nothing here
declares suspension — which would also trip R-ENV-03, since a prod overlay may
patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is
the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip.

NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every
EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign
was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by
knob, and merges nothing.

VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards,
33 knobs defined / 28 referenced, every new token resolved in BOTH overlays);
render-golden, lint, schema, spec-immutability green; touched requirement
R-STATE-03 covered.

Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 <[email protected]>

clusters/dev/kustomization.yaml

@@ -224,9 +224,38 @@ patches:
         value: "192.168.1.55"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
+      #
+      # DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
+      # Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
+      # operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
+      # a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
+      # would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
+      # (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
       - op: replace
         path: /data/S3_BACKUP_ENDPOINT
         value: "http://192.168.1.9:9000"
+      # DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
+      # reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
+      # defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
+      # its proven restore depends on — without any manifest looking wrong.
+      #
+      # `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
+      # ignores the value, but changing it is a needless risk against a live, working backup path.
+      # An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
+      # are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
+      # against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
+      - op: replace
+        path: /data/S3_BACKUP_REGION
+        value: "us-east-1"
+      - op: replace
+        path: /data/VELERO_BACKUP_BUCKET
+        value: "velero"
+      - op: replace
+        path: /data/VELERO_BACKUP_PREFIX
+        value: ""
+      - op: replace
+        path: /data/CNPG_BACKUP_PATH
+        value: "s3://cnpg"
       # dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT