188 lines
11 KiB
YAML
188 lines
11 KiB
YAML
---
|
|
# Keycloak's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs installed by platform/cnpg).
|
|
# CARRIED BY platform-config (dependsOn platform) so the postgresql.cnpg.io CRDs exist first (F-MON1).
|
|
#
|
|
# DR-critical (R-IDP-05): storage on the 3-replica `longhorn` StorageClass (ADR-0003) — the data
|
|
# survives node loss. CNPG creates the keycloak-db-app Secret (keys username/password) that the
|
|
# Keycloak CR consumes, plus the keycloak-db-rw/-ro/-r Services. Keycloak is stateless and rebuilds
|
|
# from this DB + the git-managed realm import (docs/15).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: keycloak-db
|
|
namespace: keycloak
|
|
spec:
|
|
# THREE instances — one primary + two hot standbys, one per node (2026-09-07).
|
|
#
|
|
# WAS `instances: 1`, on ADR-0014's "single replica with DB on Longhorn + backups is acceptable".
|
|
# Longhorn's 3-way replication saves the DATA when a node dies; it does NOT save the SERVICE. With
|
|
# one instance, losing the node means rescheduling the pod, re-attaching the volume and replaying
|
|
# WAL before Postgres will accept a connection again — minutes, and only if scheduling cooperates.
|
|
#
|
|
# WHY NOW (the 2026-09-07 incident): all three CNPG primaries — keycloak-db, nextcloud-db and
|
|
# semaphore-db — were co-resident on k3s-prod-2, because with `instances: 1` nothing in the spec
|
|
# ever said they must not be. One node took out SSO, Nextcloud and Semaphore together, and with
|
|
# SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give this
|
|
# Cluster something to fail OVER to: CNPG promotes a streaming standby in seconds and repoints the
|
|
# -rw Service, with no volume re-attach and no restore. The `affinity.podAntiAffinityType:
|
|
# required` below is what makes the three land on three DIFFERENT nodes.
|
|
#
|
|
# UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Replicas copy corruption and deletions
|
|
# faithfully; only the Barman archive below is a backup. Adding standbys while archiving was
|
|
# broken would have multiplied the disk-full clock described further down, so this change waited
|
|
# for all three Clusters to report ContinuousArchiving=True with a real firstRecoverabilityPoint.
|
|
#
|
|
# COST: two more 5Gi PVCs here (Longhorn thin-provisions; the live pgdata is ~600 MiB), ~30 GiB
|
|
# PROVISIONED across all three DBs and well under 3 GiB actually consumed. Longhorn allows 200%
|
|
# over-provisioning on both clusters and both have headroom.
|
|
#
|
|
# APPLYING THIS IS NON-DISRUPTIVE: CNPG scales out by creating -2 then -3 sequentially, each
|
|
# seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are
|
|
# not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES
|
|
# cause a switchover — see it for the blast radius.)
|
|
instances: 3
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.15
|
|
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes (2026-09-07).
|
|
#
|
|
# CNPG's default is `preferred`, which is only a scheduler HINT: the scheduler may put two
|
|
# instances on one node whenever that scores better, and it WILL do so on the path that matters —
|
|
# when CNPG recreates an instance after losing a node, the surviving nodes are the only candidates
|
|
# and one of them already runs an instance. So `preferred` is precisely how `instances: 3` decays
|
|
# back into the single-node failure domain this whole change exists to remove; nothing pins the
|
|
# good layout in place. `required` makes it a scheduling CONSTRAINT instead of a preference.
|
|
#
|
|
# THE TRADE, STATED PLAINLY: on a 3-node cluster with no spare node, an instance whose node is
|
|
# gone stays Pending until a node is available. That is the CORRECT behaviour here. A replacement
|
|
# squeezed onto a node that already hosts an instance buys nothing — both die with that node —
|
|
# while making the cluster LOOK three-way replicated. Availability is unaffected either way:
|
|
# 2 of 3 instances still serve and CNPG promotes one of them.
|
|
#
|
|
# Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes (checked
|
|
# read-only before merge). topologyKey defaults to kubernetes.io/hostname — the failure domain
|
|
# that matters here. enablePodAntiAffinity defaults to true; stated explicitly so this block can
|
|
# be read without knowing the default.
|
|
affinity:
|
|
enablePodAntiAffinity: true
|
|
podAntiAffinityType: required
|
|
# Roll the PRIMARY by SWITCHOVER, not by restarting it in place. CNPG's default for
|
|
# primaryUpdateMethod is `restart`, which deletes and recreates the primary pod — tens of seconds
|
|
# with no Postgres serving, and on THIS database that is an SSO outage. With standbys present
|
|
# (instances: 3 above) a switchover promotes an already-updated, already-streaming replica and
|
|
# demotes the old primary, so the interruption is a connection reset rather than a cold start.
|
|
# This is the field that makes the anti-affinity roll below a switchover rather than an outage.
|
|
#
|
|
# KNOWN CONSTRAINT: under `switchover` the operator REJECTS an update that changes `imageName`
|
|
# AND a postgresql parameter in the same reconcile. Renovate bumping the image alone is fine;
|
|
# never combine an image bump with a postgresql.parameters change in one commit.
|
|
primaryUpdateMethod: switchover
|
|
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07).
|
|
#
|
|
# Standbys are asynchronous by default: COMMIT returns as soon as the PRIMARY has flushed locally,
|
|
# so a primary that dies before its standbys catch up takes the last few transactions with it and
|
|
# the promoted standby comes up quietly missing them. On the IdP database that is a just-created
|
|
# user, a rotated credential or a consent silently disappearing at exactly the moment of a
|
|
# failover — the hardest class of bug to notice and the worst one to have. `method: any` +
|
|
# `number: 1` makes COMMIT wait for ONE of the two standbys to confirm, which is the entire point
|
|
# of having them.
|
|
#
|
|
# dataDurability: preferred — AND NEVER `required`. `required` is a STRICT quorum: if fewer than
|
|
# `number` healthy standbys are available, writes BLOCK. Postgres here shares three nodes with
|
|
# etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium
|
|
# restart — would turn a partial degradation into a TOTAL WRITE OUTAGE on SSO, and SSO is the
|
|
# dependency of kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. `preferred` keeps
|
|
# synchronous commit whenever a healthy standby is there and relaxes the required count
|
|
# dynamically when one is not: durability when we can have it, availability when we cannot. That
|
|
# is the right trade for a 3-node home cluster; a 5-node cluster with a spare could revisit it.
|
|
# (dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — they are.)
|
|
#
|
|
# NON-DISRUPTIVE TO APPLY: this compiles to `synchronous_standby_names`, which PostgreSQL treats
|
|
# as a RELOAD-only parameter. No pod restart, no switchover, no client disconnect — CNPG reloads
|
|
# the config in place. Landing it AFTER the anti-affinity roll is deliberate: it wants standbys
|
|
# that already exist and are already on their final nodes, and it must not be folded into a roll.
|
|
#
|
|
# Field shape verified against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical
|
|
# on prod and dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number`
|
|
# (>0); `dataDurability` is enum required|preferred.
|
|
postgresql:
|
|
synchronous:
|
|
method: any
|
|
number: 1
|
|
dataDurability: preferred
|
|
storage:
|
|
storageClass: longhorn
|
|
size: 5Gi
|
|
bootstrap:
|
|
initdb:
|
|
database: keycloak
|
|
owner: keycloak
|
|
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
|
|
# PodMonitor at runtime (CRD already present from Phase-4 monitoring) — no kustomize CRD-ordering hazard.
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
# --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → object storage ---
|
|
# Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
|
|
# PVC snapshot for a live DB). CNPG appends `serverName` (defaults to the Cluster name) to
|
|
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/keycloak-db/ — dev
|
|
# s3://cnpg/keycloak-db/ on the on-Proxmox MinIO, prod s3://k3s-prod-backup/barman/keycloak-db/ on
|
|
# Cloudflare R2. Both endpointURL and destinationPath are cluster-vars knobs (R-ENV-02),
|
|
# substituted by the `platform-config` Kustomization (this file sits under platform/keycloak/ but
|
|
# platform-config is the only tier that lists it as a resource — see its kustomization.yaml).
|
|
# Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
|
|
# (a Pod only reads same-ns Secrets).
|
|
#
|
|
# *** archive_mode IS ON, SO A FAILING ARCHIVE COMMAND IS A DISK-FULL CLOCK, NOT A DEGRADED
|
|
# BACKUP. *** Postgres will not recycle a WAL segment it has not successfully archived, so
|
|
# whenever these two values are wrong the pgdata PVC fills monotonically until the primary
|
|
# PANICs. This is not hypothetical: prod ran from bootstrap to 2026-09-07 with
|
|
# `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT` (barman-cloud-wal-archive exit 4 on every segment)
|
|
# and had burned ~579 MiB of a 5 GiB PVC on the way to taking SSO — and with it kubectl OIDC,
|
|
# Grafana, Headlamp, Gitea and forward-auth — down with it. Never leave a placeholder here, and
|
|
# treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one.
|
|
backup:
|
|
retentionPolicy: "30d" # keep 30 days of base backups + WAL (PITR window); prunes older
|
|
barmanObjectStore:
|
|
destinationPath: ${CNPG_BACKUP_PATH}
|
|
endpointURL: ${S3_BACKUP_ENDPOINT}
|
|
# PER-ENVIRONMENT credential (2026-09-07): dev reads keycloak-db-backup-credentials (MinIO),
|
|
# prod reads keycloak-db-backup-credentials-prod (Cloudflare R2). Both Secrets are applied to
|
|
# both clusters by platform/keycloak/kustomization.yaml; ${BACKUP_CREDENTIALS_SUFFIX} (dev ""
|
|
# | prod "-prod") picks one. Before the split these were ONE shared Secret, so writing prod's
|
|
# R2 token into it would have left dev signing MinIO requests with an R2 key — a 403 on every
|
|
# archive command, which under archive_mode=on is the disk-full clock described above, not a
|
|
# missing backup.
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 2
|
|
data:
|
|
compression: gzip
|
|
---
|
|
# Daily base backup of keycloak-db (WAL is archived continuously by the barmanObjectStore above).
|
|
# CNPG cron is 6-field (sec min hour dom mon dow). immediate:true takes a first base backup as soon
|
|
# as the schedule is created so the PITR chain has a starting point. Staggered 01:00 (nextcloud 02:00,
|
|
# semaphore 04:00, Velero 03:00) to avoid concurrent base backups hammering MinIO.
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: keycloak-db-daily
|
|
namespace: keycloak
|
|
spec:
|
|
immediate: true
|
|
schedule: "0 0 1 * * *"
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: keycloak-db
|
|
method: barmanObjectStore
|