jakman85
f338e91770
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
122 lines
6.7 KiB
YAML
122 lines
6.7 KiB
YAML
---
|
|
# Keycloak's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs installed by platform/cnpg).
|
|
# CARRIED BY platform-config (dependsOn platform) so the postgresql.cnpg.io CRDs exist first (F-MON1).
|
|
#
|
|
# DR-critical (R-IDP-05): storage on the 3-replica `longhorn` StorageClass (ADR-0003) — the data
|
|
# survives node loss. CNPG creates the keycloak-db-app Secret (keys username/password) that the
|
|
# Keycloak CR consumes, plus the keycloak-db-rw/-ro/-r Services. Keycloak is stateless and rebuilds
|
|
# from this DB + the git-managed realm import (docs/15).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: keycloak-db
|
|
namespace: keycloak
|
|
spec:
|
|
# THREE instances — one primary + two hot standbys, one per node (2026-09-07).
|
|
#
|
|
# WAS `instances: 1`, on ADR-0014's "single replica with DB on Longhorn + backups is acceptable".
|
|
# Longhorn's 3-way replication saves the DATA when a node dies; it does NOT save the SERVICE. With
|
|
# one instance, losing the node means rescheduling the pod, re-attaching the volume and replaying
|
|
# WAL before Postgres will accept a connection again — minutes, and only if scheduling cooperates.
|
|
#
|
|
# WHY NOW (the 2026-09-07 incident): all three CNPG primaries — keycloak-db, nextcloud-db and
|
|
# semaphore-db — were co-resident on k3s-prod-2, because with `instances: 1` nothing in the spec
|
|
# ever said they must not be. One node took out SSO, Nextcloud and Semaphore together, and with
|
|
# SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give this
|
|
# Cluster something to fail OVER to: CNPG promotes a streaming standby in seconds and repoints the
|
|
# -rw Service, with no volume re-attach and no restore. The `affinity.podAntiAffinityType:
|
|
# required` below is what makes the three land on three DIFFERENT nodes.
|
|
#
|
|
# UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Replicas copy corruption and deletions
|
|
# faithfully; only the Barman archive below is a backup. Adding standbys while archiving was
|
|
# broken would have multiplied the disk-full clock described further down, so this change waited
|
|
# for all three Clusters to report ContinuousArchiving=True with a real firstRecoverabilityPoint.
|
|
#
|
|
# COST: two more 5Gi PVCs here (Longhorn thin-provisions; the live pgdata is ~600 MiB), ~30 GiB
|
|
# PROVISIONED across all three DBs and well under 3 GiB actually consumed. Longhorn allows 200%
|
|
# over-provisioning on both clusters and both have headroom.
|
|
#
|
|
# APPLYING THIS IS NON-DISRUPTIVE: CNPG scales out by creating -2 then -3 sequentially, each
|
|
# seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are
|
|
# not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES
|
|
# cause a switchover — see it for the blast radius.)
|
|
instances: 3
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
|
storage:
|
|
storageClass: longhorn
|
|
size: 5Gi
|
|
bootstrap:
|
|
initdb:
|
|
database: keycloak
|
|
owner: keycloak
|
|
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
|
|
# PodMonitor at runtime (CRD already present from Phase-4 monitoring) — no kustomize CRD-ordering hazard.
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
# --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → object storage ---
|
|
# Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
|
|
# PVC snapshot for a live DB). CNPG appends `serverName` (defaults to the Cluster name) to
|
|
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/keycloak-db/ — dev
|
|
# s3://cnpg/keycloak-db/ on the on-Proxmox MinIO, prod s3://k3s-prod-backup/barman/keycloak-db/ on
|
|
# Cloudflare R2. Both endpointURL and destinationPath are cluster-vars knobs (R-ENV-02),
|
|
# substituted by the `platform-config` Kustomization (this file sits under platform/keycloak/ but
|
|
# platform-config is the only tier that lists it as a resource — see its kustomization.yaml).
|
|
# Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
|
|
# (a Pod only reads same-ns Secrets).
|
|
#
|
|
# *** archive_mode IS ON, SO A FAILING ARCHIVE COMMAND IS A DISK-FULL CLOCK, NOT A DEGRADED
|
|
# BACKUP. *** Postgres will not recycle a WAL segment it has not successfully archived, so
|
|
# whenever these two values are wrong the pgdata PVC fills monotonically until the primary
|
|
# PANICs. This is not hypothetical: prod ran from bootstrap to 2026-09-07 with
|
|
# `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT` (barman-cloud-wal-archive exit 4 on every segment)
|
|
# and had burned ~579 MiB of a 5 GiB PVC on the way to taking SSO — and with it kubectl OIDC,
|
|
# Grafana, Headlamp, Gitea and forward-auth — down with it. Never leave a placeholder here, and
|
|
# treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one.
|
|
backup:
|
|
retentionPolicy: "30d" # keep 30 days of base backups + WAL (PITR window); prunes older
|
|
barmanObjectStore:
|
|
destinationPath: ${CNPG_BACKUP_PATH}
|
|
endpointURL: ${S3_BACKUP_ENDPOINT}
|
|
# PER-ENVIRONMENT credential (2026-09-07): dev reads keycloak-db-backup-credentials (MinIO),
|
|
# prod reads keycloak-db-backup-credentials-prod (Cloudflare R2). Both Secrets are applied to
|
|
# both clusters by platform/keycloak/kustomization.yaml; ${BACKUP_CREDENTIALS_SUFFIX} (dev ""
|
|
# | prod "-prod") picks one. Before the split these were ONE shared Secret, so writing prod's
|
|
# R2 token into it would have left dev signing MinIO requests with an R2 key — a 403 on every
|
|
# archive command, which under archive_mode=on is the disk-full clock described above, not a
|
|
# missing backup.
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 2
|
|
data:
|
|
compression: gzip
|
|
---
|
|
# Daily base backup of keycloak-db (WAL is archived continuously by the barmanObjectStore above).
|
|
# CNPG cron is 6-field (sec min hour dom mon dow). immediate:true takes a first base backup as soon
|
|
# as the schedule is created so the PITR chain has a starting point. Staggered 01:00 (nextcloud 02:00,
|
|
# semaphore 04:00, Velero 03:00) to avoid concurrent base backups hammering MinIO.
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: keycloak-db-daily
|
|
namespace: keycloak
|
|
spec:
|
|
immediate: true
|
|
schedule: "0 0 1 * * *"
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: keycloak-db
|
|
method: barmanObjectStore
|