home-network-k3s-impl/runbooks/keycloak-down.md

# keycloak-down — The Keycloak IdP is down (SSO is unavailable)

**Alert:** `KeycloakDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
```

## What this means here

No ready Keycloak **server** pod for five minutes. The regex is deliberately scoped to
`keycloak-[0-9]+` (the StatefulSet pods `keycloak-0`, `keycloak-1`) because the `keycloak` namespace also
holds the operator (`keycloak-operator-*`) and the database (`keycloak-db-*`); an unscoped `max()` would
be satisfied by those being ready and would never fire for the IdP itself. That scoping was a fix — the
original matched `.*postgres.*`.

## Blast radius — read this before triaging anything else

Keycloak is the root of trust. With SSO down, all of these are down or degraded **at once**:

- **kubectl OIDC login** (the public `kubernetes` client with PKCE, groups → RBAC)
- **Grafana** (OIDC), **Headlamp** (forwards the user's id_token to the apiserver), **Gitea**
- **forward-auth** via oauth2-proxy, and the five per-app oauth2-proxy sidecars (pihole, jackett,
  qbittorrent, radarr, sonarr) — these **fail closed**, 302ing to a Keycloak that is not there

**You can still get in.** The certificate-based kubeconfig is untouched by the OIDC work and remains a
fully independent break-glass path — that is a deliberate design property, not an accident. From this
workstation the working non-interactive path is the dev-container kubeconfig; the host `k3s-oidc`
context will hang on a browser prompt, which is not the cluster being down.

**And you can still read this page.** The runbook route is deliberately **public, with no
forward-auth**, precisely so that `keycloak-down` is readable by an operator who cannot log in. If that
ever changes, this runbook becomes unreachable exactly when it is needed.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Everything in the namespace, in one view — server, operator and database together:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods -o wide; echo; \
   kubectl --context k3s-prod -n keycloak get statefulsets,deployments'
```

Is the database the real cause? (It usually is. Fully-qualified: `clusters.postgresql.cnpg.io`.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak \
     -o custom-columns=NAME:.metadata.name,INSTANCES:.status.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,PHASE:.status.phase'
```

The server's own complaint:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs keycloak-0 --tail=120 2>&1 | tail -50; echo ====; \
   kubectl --context k3s-prod -n keycloak describe pod keycloak-0 | sed -n "/Events:/,\$p"'
```

The operator's view of the CR:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get keycloaks.k8s.keycloak.org -o yaml | grep -A20 "status:"'
```

Is it reachable at all, from outside, with an explicit SNI (in-pod HTTPS tests to these hostnames are
misleading here):

```
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
```

## Likely causes, most likely first

### 1. The CNPG database is not available
Keycloak cannot start without Postgres. This is the highest-probability cause and it has its own runbook:
`/runbooks/keycloak-db`. Note that `keycloak-db` is a 3-instance CNPG Cluster with
`primaryUpdateMethod: switchover` and synchronous replication at `dataDurability: preferred` —
`preferred` specifically so that one degraded standby does **not** block writes, which on this database
would be a total SSO write outage.

### 2. A node that held the Keycloak pods went away
Both Keycloak pods and all three `keycloak-db` instances previously ran on **one node** — all three CNPG
primaries had drifted onto `k3s-prod-2` — and **on 2026-09-07 that node took SSO, Nextcloud and
Semaphore down together**. The fixes were `instances: 3` on every Cluster and
`podAntiAffinityType: required` (CNPG's default `preferred` is only a scheduler hint and co-locates on
the path that matters — recreating an instance after node loss). Check `-o wide` above: if everything is
on one node, that is a live re-occurrence of the same defect. Go to `/runbooks/node-down`.

### 3. A HelmRelease / operator upgrade went bad
Read `/runbooks/update-rollback`. Under `primaryUpdateMethod: switchover` the CNPG operator **rejects an
update that changes `imageName` and a postgresql parameter in the same reconcile** — a stuck reconcile
with that message is this.

### 4. TLS to the issuer is broken
Keycloak's clients validate the issuer's certificate. While the wildcard was LE-**staging**, in-cluster
clients rejected it with `x509: certificate signed by unknown authority`. If `/runbooks/cert-issuance` or
`/runbooks/cert-expiry` is also firing, fix that first.

### 5. Logins fail but Keycloak is UP
Then this is the **wrong alert and the wrong runbook**, and the distinction has cost this cluster dearly.
On 2026-09-06 every OIDC login on prod failed — *"Invalid client or Invalid client credentials"* — with
**all 12 clients holding the literal `(env:...)` placeholder as their secret**, while
`KeycloakDown`/`KeycloakDatabaseDown` were quiet, the import Job reported success, the CR read
`Done=True`, and the conformance test that asserts `Done=True` **passed throughout**. The only observable
was a human failing to log in. Go to `/runbooks/keycloak-client-secrets`.
Similarly, a `redirect_uri` mismatch after a hostname change is not an outage — same runbook.
And once, MFA broke every platform-admin login on its own: conditional-OTP used a non-existent config
key, giving `"Invalid role name: null"` → 403, fixed to `condUserRole: platform-admin`.

## When to escalate or stop

- **Escalate immediately.** SSO down is a platform-wide outage; treat it as such.
- **Use the certificate-based kubeconfig** (the dev-container path) rather than trying to fix OIDC login
  first. Do not burn the outage on getting `kubectl` to authenticate through the thing that is down.
- **Do not edit the realm to "reset" anything.** The realm import is a **ONE-SHOT that fails open**: the
  operator logs *"Job already executed - not recreating"*, so a git change to
  `keycloak-realm-import.yaml` does **not** re-import. The documented path is `kcadm.sh` inside
  `keycloak-0` (admin credentials in the Secret `keycloak-admin-credentials`; `kcadm` is present,
  `curl`/`python3` are **not**) **and** fixing git for reproducibility. Doing only one of the two is how
  drift is created.
- **Do not delete the Keycloak CR or the StatefulSet.** Recreating the IdP is a restore operation, not a
  restart.
- **Do not touch `keycloak-db` volumes.** If the database is the cause, read `/runbooks/keycloak-db` and
  `/runbooks/cnpg-archiving`; only the Barman archive is a backup, standbys replicate deletions
  faithfully.
- Break-glass local admins exist for gitea, nextcloud and semaphore. Nextcloud has **no native
  group→admin mapping**, so platform-admins there are promoted by hand.

---
Source of truth: `runbooks/keycloak-down.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the 2026-09-06 client-secret lockout; the 2026-09-07 single-node SSO
outage); commits `f338e91`, `e8d8601`, `cac413e` (CNPG instances/anti-affinity/switchover/sync).