home-network-k3s-impl/runbooks/keycloak-db.md

# keycloak-db — Keycloak's PostgreSQL is unavailable

**Alert:** `KeycloakDatabaseDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
```

## What this means here

No ready `keycloak-db-*` pod for five minutes. `keycloak-db` is a **CloudNativePG Cluster** with
**3 instances** on Longhorn (3 replicas each), and it is the database the IdP cannot live without. The
`pod=~"keycloak-db-.*"` scoping matters for the same reason as in `/runbooks/keycloak-down`: the
namespace also holds the Keycloak server and operator pods, and an unscoped `max()` would be masked by
those.

**Blast radius is the whole platform's identity:** Keycloak fails → kubectl OIDC, Grafana, Headlamp,
Gitea and forward-auth all fail. The certificate-based kubeconfig is an independent break-glass path and
still works. This page is deliberately served **without** forward-auth so it stays readable during
exactly this outage.

## The HA properties you are relying on, and why they are set the way they are

- **`instances: 3`.** All three CNPG Clusters used to run `instances: 1`, and all three primaries had
  drifted onto one node — **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together.**
  Scale-out to 3 is what gives you a standby to fail over to.
- **`podAntiAffinityType: required`**, not CNPG's default `preferred`. `preferred` is a scheduler
  *hint* and on the path that matters — recreating an instance after node loss — it co-locates.
  Recorded conclusion: *"`preferred` is how a 3-instance cluster decays back into the single-node
  failure domain."* The accepted cost is that an instance stays **Pending** until a node returns.
- **`primaryUpdateMethod: switchover`.** The CRD default is `restart`, which is a delete-and-recreate of
  the **primary** — tens of seconds with no Postgres, i.e. an SSO outage. Constraint to remember: under
  `switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in
  the same reconcile**.
- **Synchronous replication `method: any`, `number: 1`, `dataDurability: preferred` — never `required`.**
  `required` is a strict quorum: with fewer than `number` healthy standbys, **writes BLOCK**. On this
  database one degraded standby (a node reboot, a Longhorn rebuild, a Cilium restart) would become a
  total write outage. `dataDurability` is only honoured while `standbyNamesPre/Post` are unset.

**And the thing that is easy to get wrong:** *"Standbys replicate corruption and deletions faithfully;
only the Barman archive is a backup."* Three ready replicas are not a recovery plan.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

The Cluster's own verdict first — fully-qualified, because `cluster` is an ambiguous short name:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```

Where the instances are, and whether they are all on one node (the 2026-09-07 defect):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods -l cnpg.io/cluster=keycloak-db -o wide'
```

Why a pod is not ready:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak describe pod keycloak-db-2 | sed -n "/Events:/,\$p"; echo ====; \
   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=80 2>&1 | tail -40'
```

The conditions that matter for recoverability — check these **before** any recovery action:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db \
     -o custom-columns=ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -n keycloak'
```

`kubectl get backup -A` prints "No resources found" here even with 27 CNPG backups present — three CRDs
register the kind `Backup` and the short name resolves to Longhorn's, which has none. Never decide
anything from the short name.

Storage underneath:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pvc; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'
```

## Likely causes, most likely first

### 1. A node is down or was rebooted
The instances live on Longhorn with hard anti-affinity, so a node loss takes one instance. With
`instances: 3` the cluster should survive it and promote; with `podAntiAffinityType: required` the lost
instance stays **Pending** until a node returns, which is expected and accepted. Go to
`/runbooks/node-down`.

### 2. The data PVC filled up
The most dangerous cause, because the mechanism is a *backup* fault presenting as a *storage* failure.
With `archive_mode=on` and a failing `archive_command`, Postgres **refuses to recycle** unarchived WAL;
`pg_wal` shares the data PVC on every Cluster here (no `walStorage:` configured) and grows until the
volume is full and the postmaster **PANICs**. Measured on this very database: 17 unarchived segments /
~268 MiB on a 5 GiB PVC, growing ~6.8 MiB/h — about 29 days from 93% free to dead. Go to
`/runbooks/cnpg-archiving` and `/runbooks/pvc-capacity`. **Never delete WAL by hand.**

### 3. A Longhorn volume is degraded or faulted
`/runbooks/longhorn-degraded`, `/runbooks/longhorn-faulted`.

### 4. An upgrade is stuck mid-switchover
Signature: the operator refusing an update that changes `imageName` and a postgresql parameter together.
Split the change in git. See `/runbooks/update-rollback`.

### 5. The database is up and Keycloak still fails
Then read `/runbooks/keycloak-down` and, more likely, `/runbooks/keycloak-client-secrets` — the
2026-09-06 lockout had all 12 OIDC clients holding literal placeholder secrets behind an entirely green
cluster.

## When to escalate or stop

- **Escalate immediately.** This is SSO's foundation; a 5m `for` on a `critical` is already generous.
- **Never change `dataDurability` to `required`** while trying to "make replication safer" — it converts
  one degraded standby into a total write outage on the SSO database.
- **Do not delete a CNPG pod or PVC to force a rebuild.** The operator manages instance lifecycle;
  deleting a PVC destroys that instance's data and, if it is the primary, may lose transactions that the
  archive does not have.
- **Do not attempt a restore before checking `firstRecoverabilityPoint` and `ContinuousArchiving`.** Prod
  ran 38+ hours from bootstrap with `ContinuousArchiving=False` and `firstRecoverabilityPoint: null` on
  all three Clusters and nothing said so. Know what you actually have before you destroy what you have.
- **Standbys are not backups.** Recovery is from the Barman archive in R2 (`CNPG_BACKUP_PATH =
  s3://k3s-prod-backup/barman`, 30d retention), which is an operator operation.

---
Source of truth: `runbooks/keycloak-db.md`. Index: `/runbooks/`
Records: commits `f338e91` (instances 3), `e8d8601` (required anti-affinity + switchover), `cac413e`
(sync replication, `dataDurability: preferred`); `platform/cnpg/alerts.yaml` header (the 2026-09-06
archiving incident and the measured WAL fill rate).