# keycloak-down — The Keycloak IdP is down (SSO is unavailable) **Alert:** `KeycloakDown` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity` **Expression:** ``` max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0 or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"}) ``` ## What this means here No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to `keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the original matched `.*postgres.*`. ## Blast radius — read this before triaging anything else Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**: - **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC) - **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea** - **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett, qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there **You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a fully independent break-glass path — that is a deliberate design property, not an accident. From this workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc` context will hang on a browser prompt, which is not the cluster being down. **And you can still read this page.** The runbook route is deliberately **public, with no forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that ever changes, this runbook becomes unreachable exactly when it is needed. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Everything in the namespace, in one view — server, operator and database together: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \ kubectl --context k3s-prod -n keycloak get statefulsets,deployments' ``` Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \ -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase' ``` The server's own complaint: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \ kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"' ``` The operator's view of the CR: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"' ``` Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are misleading here): ``` curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \ https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration ``` ## Likely causes, most likely first ### 1. The CNPG database is not available Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook: `/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with `primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` — `preferred` specifically so that one degraded standby does **not** block writes, which on this database would be a total SSO write outage. ### 2. A node that held the Keycloak pods went away Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together**. The fixes were `instances: 3` on every Cluster and `podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`. ### 3. A HelmRelease / operator upgrade went bad Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile with that message is this. ### 4. TLS to the issuer is broken Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or `/runbooks/cert-expiry` is also firing, fix that first. ### 5. Logins fail but Keycloak is UP Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly. On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with **all 12 clients holding the literal `(env:...)` placeholder as their secret**, while `KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read `Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`. Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook. And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`. ## When to escalate or stop - **Escalate immediately.** SSO down is a platform-wide outage; treat it as such. - **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down. - **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside `keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present, `curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how drift is created. - **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a restart. - **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and `/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions faithfully. - Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native group→admin mapping**, so platform-admins there are promoted by hand. --- Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).