feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX

The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).

    cluster-vars  BACKUP_CREDENTIALS_SUFFIX   base ""  |  dev ""  |  prod "-prod"

    velero HelmRelease  credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
    3x CNPG Cluster     s3Credentials.*.name:      <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}

ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.

THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.

Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).

VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:

  strict envsubst              PASSES for all 8 (env x tier) combinations
  dev   velero existingSecret  velero-cloud-credentials
  dev   keycloak/nextcloud/semaphore-db  <db>-backup-credentials         (both key refs each)
  prod  velero existingSecret  velero-cloud-credentials-prod
  prod  keycloak/nextcloud/semaphore-db  <db>-backup-credentials-prod    (both key refs each)
  both  all EIGHT Secret objects present in the rendered output (dev and prod alike)
  no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
    escapes in apps/wave-0, which are pre-existing and unrelated.

Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
  dev   velero s3://velero (root) + cnpg s3://cnpg      @ http://192.168.1.9:9000  region us-east-1
  prod  velero k3s-prod-backup/velero + barman prefix   @ https://<acct>.r2.cloudflarestorage.com  region auto

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 01:41:56 -05:00
1 parent e7c4a95c2d
commit 161430d915
8 files changed
+142 -16

No files matched your search

+23 -2
View File
@@ -106,8 +106,11 @@ patches:
secretRef:
name: sops-age
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
# It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
# MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
# stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
# BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
# out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -256,6 +259,24 @@ patches:
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://cnpg"
# WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
# (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
# its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
# renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
#
# RESTATED EXPLICITLY even though it equals the base default, for the same reason as
# EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
# base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
# signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
# turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
# restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
#
# DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
# 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
# backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: ""
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT