home-network-k3s-impl/runbooks/keycloak-db.md

keycloak-db — Keycloak's PostgreSQL is unavailable

Alert: KeycloakDatabaseDown (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.identity Expression:

max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})

What this means here

No ready keycloak-db-* pod for five minutes. keycloak-db is a CloudNativePG Cluster with 3 instances on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The pod=~"keycloak-db-.*" scoping matters for the same reason as in /runbooks/keycloak-down: the namespace also holds the Keycloak server and operator pods, and an unscoped max() would be masked by those.

Blast radius is the whole platform's identity: Keycloak fails → kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and still works. This page is deliberately served without forward-auth so it stays readable during exactly this outage.

The HA properties you are relying on, and why they are set the way they are

• instances: 3. All three CNPG Clusters used to run instances: 1, and all three primaries had drifted onto one node — on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together. Scale-out to 3 is what gives you a standby to fail over to.
• podAntiAffinityType: required, not CNPG's default preferred. preferred is a scheduler hint and on the path that matters — recreating an instance after node loss — it co-locates. Recorded conclusion: "preferred is how a 3-instance cluster decays back into the single-node failure domain." The accepted cost is that an instance stays Pending until a node returns.
• primaryUpdateMethod: switchover. The CRD default is restart, which is a delete-and-recreate of the primary — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under switchover the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile.
• Synchronous replication method: any, number: 1, dataDurability: preferred — never required. required is a strict quorum: with fewer than number healthy standbys, writes BLOCK. On this database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a total write outage. dataDurability is only honoured while standbyNamesPre/Post are unset.

And the thing that is easy to get wrong: "Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup." Three ready replicas are not a recovery plan.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

The Cluster's own verdict first — fully-qualified, because cluster is an ambiguous short name:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'

Where the instances are, and whether they are all on one node (the 2026-09-07 defect):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide'

Why a pod is not ready:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \
   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40'

The conditions that matter for recoverability — check these before any recovery action:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \
     -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak'

kubectl get backup -A prints "No resources found" here even with 27 CNPG backups present — three CRDs register the kind Backup and the short name resolves to Longhorn's, which has none. Never decide anything from the short name.

Storage underneath:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pvc; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'

Likely causes, most likely first

1. A node is down or was rebooted

The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With instances: 3 the cluster should survive it and promote; with podAntiAffinityType: required the lost instance stays Pending until a node returns, which is expected and accepted. Go to /runbooks/node-down.

2. The data PVC filled up

The most dangerous cause, because the mechanism is a backup fault presenting as a storage failure. With archive_mode=on and a failing archive_command, Postgres refuses to recycle unarchived WAL; pg_wal shares the data PVC on every Cluster here (no walStorage: configured) and grows until the volume is full and the postmaster PANICs. Measured on this very database: 17 unarchived segments / ~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to /runbooks/cnpg-archiving and /runbooks/pvc-capacity. Never delete WAL by hand.

3. A Longhorn volume is degraded or faulted

/runbooks/longhorn-degraded, /runbooks/longhorn-faulted.

4. An upgrade is stuck mid-switchover

Signature: the operator refusing an update that changes imageName and a postgresql parameter together. Split the change in git. See /runbooks/update-rollback.

5. The database is up and Keycloak still fails

Then read /runbooks/keycloak-down and, more likely, /runbooks/keycloak-client-secrets — the 2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green cluster.

When to escalate or stop

• Escalate immediately. This is SSO's foundation; a 5m for on a critical is already generous.
• Never change dataDurability to required while trying to "make replication safer" — it converts one degraded standby into a total write outage on the SSO database.
• Do not delete a CNPG pod or PVC to force a rebuild. The operator manages instance lifecycle; deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the archive does not have.
• Do not attempt a restore before checking firstRecoverabilityPoint and ContinuousArchiving. Prod ran 38+ hours from bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint: null on all three Clusters and nothing said so. Know what you actually have before you destroy what you have.
• Standbys are not backups. Recovery is from the Barman archive in R2 (CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman, 30d retention), which is an operator operation.

---

Source of truth: runbooks/keycloak-db.md. Index: /runbooks/ Records: commits f338e91 (instances 3), e8d8601 (required anti-affinity + switchover), cac413e (sync replication, dataDurability: preferred); platform/cnpg/alerts.yaml header (the 2026-09-06 archiving incident and the measured WAL fill rate).