# keycloak-db — Keycloak's PostgreSQL is unavailable **Alert:** `KeycloakDatabaseDown` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` **Expression:** ``` max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0 or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"}) ``` ## What this means here No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with **3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The `pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by those. **Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and still works. This page is deliberately served **without** forward-auth so it stays readable during exactly this outage. ## The HA properties you are relying on, and why they are set the way they are - **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.** Scale-out to 3 is what gives you a standby to fail over to. - **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler *hint* and on the path that matters — recreating an instance after node loss — it co-locates. Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node failure domain."* The accepted cost is that an instance stays **Pending** until a node returns. - **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under `switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in the same reconcile**. - **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.** `required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset. **And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup."* Three ready replicas are not a recovery plan. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' ``` Where the instances are, and whether they are all on one node (the 2026-09-07 defect): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide' ``` Why a pod is not ready: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \ kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40' ``` The conditions that matter for recoverability — check these **before** any recovery action: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \ -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak' ``` `kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide anything from the short name. Storage underneath: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak get pvc; \ kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy' ``` ## Likely causes, most likely first ### 1. A node is down or was rebooted The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With `instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost instance stays **Pending** until a node returns, which is expected and accepted. Go to `/runbooks/node-down`. ### 2. The data PVC filled up The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure. With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL; `pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments / ~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to `/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.** ### 3. A Longhorn volume is degraded or faulted `/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`. ### 4. An upgrade is stuck mid-switchover Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together. Split the change in git. See `/runbooks/update-rollback`. ### 5. The database is up and Keycloak still fails Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the 2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green cluster. ## When to escalate or stop - **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous. - **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts one degraded standby into a total write outage on the SSO database. - **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle; deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the archive does not have. - **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on all three Clusters and nothing said so. Know what you actually have before you destroy what you have. - **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation. --- Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/` Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e` (sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06 archiving incident and the measured WAL fill rate).