home-network-k3s-impl/platform/keycloak/postgres-cluster.yaml

---
# Keycloak's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs installed by platform/cnpg).
# CARRIED BY platform-config (dependsOn platform) so the postgresql.cnpg.io CRDs exist first (F-MON1).
#
# DR-critical (R-IDP-05): storage on the 3-replica `longhorn` StorageClass (ADR-0003) — the data
# survives node loss. CNPG creates the keycloak-db-app Secret (keys username/password) that the
# Keycloak CR consumes, plus the keycloak-db-rw/-ro/-r Services. Keycloak is stateless and rebuilds
# from this DB + the git-managed realm import (docs/15).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: keycloak-db
  namespace: keycloak
spec:
  # THREE instances — one primary + two hot standbys, one per node (2026-09-07).
  #
  # WAS `instances: 1`, on ADR-0014's "single replica with DB on Longhorn + backups is acceptable".
  # Longhorn's 3-way replication saves the DATA when a node dies; it does NOT save the SERVICE. With
  # one instance, losing the node means rescheduling the pod, re-attaching the volume and replaying
  # WAL before Postgres will accept a connection again — minutes, and only if scheduling cooperates.
  #
  # WHY NOW (the 2026-09-07 incident): all three CNPG primaries — keycloak-db, nextcloud-db and
  # semaphore-db — were co-resident on k3s-prod-2, because with `instances: 1` nothing in the spec
  # ever said they must not be. One node took out SSO, Nextcloud and Semaphore together, and with
  # SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give this
  # Cluster something to fail OVER to: CNPG promotes a streaming standby in seconds and repoints the
  # -rw Service, with no volume re-attach and no restore. The `affinity.podAntiAffinityType:
  # required` below is what makes the three land on three DIFFERENT nodes.
  #
  # UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Replicas copy corruption and deletions
  # faithfully; only the Barman archive below is a backup. Adding standbys while archiving was
  # broken would have multiplied the disk-full clock described further down, so this change waited
  # for all three Clusters to report ContinuousArchiving=True with a real firstRecoverabilityPoint.
  #
  # COST: two more 5Gi PVCs here (Longhorn thin-provisions; the live pgdata is ~600 MiB), ~30 GiB
  # PROVISIONED across all three DBs and well under 3 GiB actually consumed. Longhorn allows 200%
  # over-provisioning on both clusters and both have headroom.
  #
  # APPLYING THIS IS NON-DISRUPTIVE: CNPG scales out by creating -2 then -3 sequentially, each
  # seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are
  # not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES
  # cause a switchover — see it for the blast radius.)
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes (2026-09-07).
  #
  # CNPG's default is `preferred`, which is only a scheduler HINT: the scheduler may put two
  # instances on one node whenever that scores better, and it WILL do so on the path that matters —
  # when CNPG recreates an instance after losing a node, the surviving nodes are the only candidates
  # and one of them already runs an instance. So `preferred` is precisely how `instances: 3` decays
  # back into the single-node failure domain this whole change exists to remove; nothing pins the
  # good layout in place. `required` makes it a scheduling CONSTRAINT instead of a preference.
  #
  # THE TRADE, STATED PLAINLY: on a 3-node cluster with no spare node, an instance whose node is
  # gone stays Pending until a node is available. That is the CORRECT behaviour here. A replacement
  # squeezed onto a node that already hosts an instance buys nothing — both die with that node —
  # while making the cluster LOOK three-way replicated. Availability is unaffected either way:
  # 2 of 3 instances still serve and CNPG promotes one of them.
  #
  # Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes (checked
  # read-only before merge). topologyKey defaults to kubernetes.io/hostname — the failure domain
  # that matters here. enablePodAntiAffinity defaults to true; stated explicitly so this block can
  # be read without knowing the default.
  affinity:
    enablePodAntiAffinity: true
    podAntiAffinityType: required
  # Roll the PRIMARY by SWITCHOVER, not by restarting it in place. CNPG's default for
  # primaryUpdateMethod is `restart`, which deletes and recreates the primary pod — tens of seconds
  # with no Postgres serving, and on THIS database that is an SSO outage. With standbys present
  # (instances: 3 above) a switchover promotes an already-updated, already-streaming replica and
  # demotes the old primary, so the interruption is a connection reset rather than a cold start.
  # This is the field that makes the anti-affinity roll below a switchover rather than an outage.
  #
  # KNOWN CONSTRAINT: under `switchover` the operator REJECTS an update that changes `imageName`
  # AND a postgresql parameter in the same reconcile. Renovate bumping the image alone is fine;
  # never combine an image bump with a postgresql.parameters change in one commit.
  primaryUpdateMethod: switchover
  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07).
  #
  # Standbys are asynchronous by default: COMMIT returns as soon as the PRIMARY has flushed locally,
  # so a primary that dies before its standbys catch up takes the last few transactions with it and
  # the promoted standby comes up quietly missing them. On the IdP database that is a just-created
  # user, a rotated credential or a consent silently disappearing at exactly the moment of a
  # failover — the hardest class of bug to notice and the worst one to have. `method: any` +
  # `number: 1` makes COMMIT wait for ONE of the two standbys to confirm, which is the entire point
  # of having them.
  #
  # dataDurability: preferred — AND NEVER `required`. `required` is a STRICT quorum: if fewer than
  # `number` healthy standbys are available, writes BLOCK. Postgres here shares three nodes with
  # etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium
  # restart — would turn a partial degradation into a TOTAL WRITE OUTAGE on SSO, and SSO is the
  # dependency of kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. `preferred` keeps
  # synchronous commit whenever a healthy standby is there and relaxes the required count
  # dynamically when one is not: durability when we can have it, availability when we cannot. That
  # is the right trade for a 3-node home cluster; a 5-node cluster with a spare could revisit it.
  # (dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — they are.)
  #
  # NON-DISRUPTIVE TO APPLY: this compiles to `synchronous_standby_names`, which PostgreSQL treats
  # as a RELOAD-only parameter. No pod restart, no switchover, no client disconnect — CNPG reloads
  # the config in place. Landing it AFTER the anti-affinity roll is deliberate: it wants standbys
  # that already exist and are already on their final nodes, and it must not be folded into a roll.
  #
  # Field shape verified against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical
  # on prod and dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number`
  # (>0); `dataDurability` is enum required|preferred.
  postgresql:
    synchronous:
      method: any
      number: 1
      dataDurability: preferred
  storage:
    storageClass: longhorn
    size: 5Gi
  bootstrap:
    initdb:
      database: keycloak
      owner: keycloak
  # Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
  # PodMonitor at runtime (CRD already present from Phase-4 monitoring) — no kustomize CRD-ordering hazard.
  monitoring:
    enablePodMonitor: true
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 512Mi
  # --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → object storage ---
  # Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
  # PVC snapshot for a live DB). CNPG appends `serverName` (defaults to the Cluster name) to
  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/keycloak-db/ — dev
  # s3://cnpg/keycloak-db/ on the on-Proxmox MinIO, prod s3://k3s-prod-backup/barman/keycloak-db/ on
  # Cloudflare R2. Both endpointURL and destinationPath are cluster-vars knobs (R-ENV-02),
  # substituted by the `platform-config` Kustomization (this file sits under platform/keycloak/ but
  # platform-config is the only tier that lists it as a resource — see its kustomization.yaml).
  # Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
  # (a Pod only reads same-ns Secrets).
  #
  # *** archive_mode IS ON, SO A FAILING ARCHIVE COMMAND IS A DISK-FULL CLOCK, NOT A DEGRADED
  # BACKUP. *** Postgres will not recycle a WAL segment it has not successfully archived, so
  # whenever these two values are wrong the pgdata PVC fills monotonically until the primary
  # PANICs. This is not hypothetical: prod ran from bootstrap to 2026-09-07 with
  # `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT` (barman-cloud-wal-archive exit 4 on every segment)
  # and had burned ~579 MiB of a 5 GiB PVC on the way to taking SSO — and with it kubectl OIDC,
  # Grafana, Headlamp, Gitea and forward-auth — down with it. Never leave a placeholder here, and
  # treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one.
  backup:
    retentionPolicy: "30d"          # keep 30 days of base backups + WAL (PITR window); prunes older
    barmanObjectStore:
      destinationPath: ${CNPG_BACKUP_PATH}
      endpointURL: ${S3_BACKUP_ENDPOINT}
      # PER-ENVIRONMENT credential (2026-09-07): dev reads keycloak-db-backup-credentials (MinIO),
      # prod reads keycloak-db-backup-credentials-prod (Cloudflare R2). Both Secrets are applied to
      # both clusters by platform/keycloak/kustomization.yaml; ${BACKUP_CREDENTIALS_SUFFIX} (dev ""
      # | prod "-prod") picks one. Before the split these were ONE shared Secret, so writing prod's
      # R2 token into it would have left dev signing MinIO requests with an R2 key — a 403 on every
      # archive command, which under archive_mode=on is the disk-full clock described above, not a
      # missing backup.
      s3Credentials:
        accessKeyId:
          name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        maxParallel: 2
      data:
        compression: gzip
---
# Daily base backup of keycloak-db (WAL is archived continuously by the barmanObjectStore above).
# CNPG cron is 6-field (sec min hour dom mon dow). immediate:true takes a first base backup as soon
# as the schedule is created so the PITR chain has a starting point. Staggered 01:00 (nextcloud 02:00,
# semaphore 04:00, Velero 03:00) to avoid concurrent base backups hammering MinIO.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: keycloak-db-daily
  namespace: keycloak
spec:
  immediate: true
  schedule: "0 0 1 * * *"
  backupOwnerReference: self
  cluster:
    name: keycloak-db
  method: barmanObjectStore