home-network-k3s-impl/runbooks/keycloak-down.md

keycloak-down — The Keycloak IdP is down (SSO is unavailable)

Alert: KeycloakDown (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.identity Expression:

max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})

What this means here

No ready Keycloak server pod for five minutes. The regex is deliberately scoped to keycloak-[0-9]+ (the StatefulSet pods keycloak-0, keycloak-1) because the keycloak namespace also holds the operator (keycloak-operator-*) and the database (keycloak-db-*); an unscoped max() would be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the original matched .*postgres.*.

Blast radius — read this before triaging anything else

Keycloak is the root of trust. With SSO down, all of these are down or degraded at once:

• kubectl OIDC login (the public kubernetes client with PKCE, groups → RBAC)
• Grafana (OIDC), Headlamp (forwards the user's id_token to the apiserver), Gitea
• forward-auth via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett, qbittorrent, radarr, sonarr) — these fail closed, 302ing to a Keycloak that is not there

You can still get in. The certificate-based kubeconfig is untouched by the OIDC work and remains a fully independent break-glass path — that is a deliberate design property, not an accident. From this workstation the working non-interactive path is the dev-container kubeconfig; the host k3s-oidc context will hang on a browser prompt, which is not the cluster being down.

And you can still read this page. The runbook route is deliberately public, with no forward-auth, precisely so that keycloak-down is readable by an operator who cannot log in. If that ever changes, this runbook becomes unreachable exactly when it is needed.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Everything in the namespace, in one view — server, operator and database together:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
   kubectl --context k3s-prod -n keycloak get statefulsets,deployments'

Is the database the real cause? (It usually is. Fully-qualified: clusters.postgresql.cnpg.io.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
     -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'

The server's own complaint:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
   kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'

The operator's view of the CR:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'

Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are misleading here):

curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration

Likely causes, most likely first

1. The CNPG database is not available

Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook: /runbooks/keycloak-db. Note that keycloak-db is a 3-instance CNPG Cluster with primaryUpdateMethod: switchover and synchronous replication at dataDurability: preferred — preferred specifically so that one degraded standby does not block writes, which on this database would be a total SSO write outage.

2. A node that held the Keycloak pods went away

Both Keycloak pods and all three keycloak-db instances previously ran on one node — all three CNPG primaries had drifted onto k3s-prod-2 — and on 2026-09-07 that node took SSO, Nextcloud and Semaphore down together. The fixes were instances: 3 on every Cluster and podAntiAffinityType: required (CNPG's default preferred is only a scheduler hint and co-locates on the path that matters — recreating an instance after node loss). Check -o wide above: if everything is on one node, that is a live re-occurrence of the same defect. Go to /runbooks/node-down.

3. A HelmRelease / operator upgrade went bad

Read /runbooks/update-rollback. Under primaryUpdateMethod: switchover the CNPG operator rejects an update that changes imageName and a postgresql parameter in the same reconcile — a stuck reconcile with that message is this.

4. TLS to the issuer is broken

Keycloak's clients validate the issuer's certificate. While the wildcard was LE-staging, in-cluster clients rejected it with x509: certificate signed by unknown authority. If /runbooks/cert-issuance or /runbooks/cert-expiry is also firing, fix that first.

5. Logins fail but Keycloak is UP

Then this is the wrong alert and the wrong runbook, and the distinction has cost this cluster dearly. On 2026-09-06 every OIDC login on prod failed — "Invalid client or Invalid client credentials" — with all 12 clients holding the literal (env:...) placeholder as their secret, while KeycloakDown/KeycloakDatabaseDown were quiet, the import Job reported success, the CR read Done=True, and the conformance test that asserts Done=True passed throughout. The only observable was a human failing to log in. Go to /runbooks/keycloak-client-secrets. Similarly, a redirect_uri mismatch after a hostname change is not an outage — same runbook. And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config key, giving "Invalid role name: null" → 403, fixed to condUserRole: platform-admin.

When to escalate or stop

• Escalate immediately. SSO down is a platform-wide outage; treat it as such.
• Use the certificate-based kubeconfig (the dev-container path) rather than trying to fix OIDC login first. Do not burn the outage on getting kubectl to authenticate through the thing that is down.
• Do not edit the realm to "reset" anything. The realm import is a ONE-SHOT that fails open: the operator logs "Job already executed - not recreating", so a git change to keycloak-realm-import.yaml does not re-import. The documented path is kcadm.sh inside keycloak-0 (admin credentials in the Secret keycloak-admin-credentials; kcadm is present, curl/python3 are not) and fixing git for reproducibility. Doing only one of the two is how drift is created.
• Do not delete the Keycloak CR or the StatefulSet. Recreating the IdP is a restore operation, not a restart.
• Do not touch keycloak-db volumes. If the database is the cause, read /runbooks/keycloak-db and /runbooks/cnpg-archiving; only the Barman archive is a backup, standbys replicate deletions faithfully.
• Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has no native group→admin mapping, so platform-admins there are promoted by hand.

---

Source of truth: runbooks/keycloak-down.md. Index: /runbooks/ Records: DECISIONS-LOG.md (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO outage); commits f338e91, e8d8601, cac413e (CNPG instances/anti-affinity/switchover/sync).