jakman85andClaude Opus 5
161430d915
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
97 lines
4.9 KiB
YAML
97 lines
4.9 KiB
YAML
---
|
|
# Keycloak's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs installed by platform/cnpg).
|
|
# CARRIED BY platform-config (dependsOn platform) so the postgresql.cnpg.io CRDs exist first (F-MON1).
|
|
#
|
|
# DR-critical (R-IDP-05): storage on the 3-replica `longhorn` StorageClass (ADR-0003) — the data
|
|
# survives node loss. CNPG creates the keycloak-db-app Secret (keys username/password) that the
|
|
# Keycloak CR consumes, plus the keycloak-db-rw/-ro/-r Services. Keycloak is stateless and rebuilds
|
|
# from this DB + the git-managed realm import (docs/15).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: keycloak-db
|
|
namespace: keycloak
|
|
spec:
|
|
# Single Postgres instance on 3-replica Longhorn is acceptable for the home scale (ADR-0014:
|
|
# "single replica with DB on Longhorn + backups is acceptable"). Raise to 3 for Postgres-level HA
|
|
# as a future hardening (prod) — the storage is already 3-way replicated by Longhorn.
|
|
instances: 1
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
|
storage:
|
|
storageClass: longhorn
|
|
size: 5Gi
|
|
bootstrap:
|
|
initdb:
|
|
database: keycloak
|
|
owner: keycloak
|
|
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
|
|
# PodMonitor at runtime (CRD already present from Phase-4 monitoring) — no kustomize CRD-ordering hazard.
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
# --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → object storage ---
|
|
# Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
|
|
# PVC snapshot for a live DB). CNPG appends `serverName` (defaults to the Cluster name) to
|
|
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/keycloak-db/ — dev
|
|
# s3://cnpg/keycloak-db/ on the on-Proxmox MinIO, prod s3://k3s-prod-backup/barman/keycloak-db/ on
|
|
# Cloudflare R2. Both endpointURL and destinationPath are cluster-vars knobs (R-ENV-02),
|
|
# substituted by the `platform-config` Kustomization (this file sits under platform/keycloak/ but
|
|
# platform-config is the only tier that lists it as a resource — see its kustomization.yaml).
|
|
# Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
|
|
# (a Pod only reads same-ns Secrets).
|
|
#
|
|
# *** archive_mode IS ON, SO A FAILING ARCHIVE COMMAND IS A DISK-FULL CLOCK, NOT A DEGRADED
|
|
# BACKUP. *** Postgres will not recycle a WAL segment it has not successfully archived, so
|
|
# whenever these two values are wrong the pgdata PVC fills monotonically until the primary
|
|
# PANICs. This is not hypothetical: prod ran from bootstrap to 2026-09-07 with
|
|
# `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT` (barman-cloud-wal-archive exit 4 on every segment)
|
|
# and had burned ~579 MiB of a 5 GiB PVC on the way to taking SSO — and with it kubectl OIDC,
|
|
# Grafana, Headlamp, Gitea and forward-auth — down with it. Never leave a placeholder here, and
|
|
# treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one.
|
|
backup:
|
|
retentionPolicy: "30d" # keep 30 days of base backups + WAL (PITR window); prunes older
|
|
barmanObjectStore:
|
|
destinationPath: ${CNPG_BACKUP_PATH}
|
|
endpointURL: ${S3_BACKUP_ENDPOINT}
|
|
# PER-ENVIRONMENT credential (2026-09-07): dev reads keycloak-db-backup-credentials (MinIO),
|
|
# prod reads keycloak-db-backup-credentials-prod (Cloudflare R2). Both Secrets are applied to
|
|
# both clusters by platform/keycloak/kustomization.yaml; ${BACKUP_CREDENTIALS_SUFFIX} (dev ""
|
|
# | prod "-prod") picks one. Before the split these were ONE shared Secret, so writing prod's
|
|
# R2 token into it would have left dev signing MinIO requests with an R2 key — a 403 on every
|
|
# archive command, which under archive_mode=on is the disk-full clock described above, not a
|
|
# missing backup.
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 2
|
|
data:
|
|
compression: gzip
|
|
---
|
|
# Daily base backup of keycloak-db (WAL is archived continuously by the barmanObjectStore above).
|
|
# CNPG cron is 6-field (sec min hour dom mon dow). immediate:true takes a first base backup as soon
|
|
# as the schedule is created so the PITR chain has a starting point. Staggered 01:00 (nextcloud 02:00,
|
|
# semaphore 04:00, Velero 03:00) to avoid concurrent base backups hammering MinIO.
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: keycloak-db-daily
|
|
namespace: keycloak
|
|
spec:
|
|
immediate: true
|
|
schedule: "0 0 1 * * *"
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: keycloak-db
|
|
method: barmanObjectStore
|