home-network-k3s-impl/runbooks/keycloak-client-secrets.md

# keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent

**Alerts:** `KeycloakClientSecretReconcilerFailing` (warning, 15m) and
`KeycloakClientSecretReconcilerStale` (warning, 2h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expressions:**
```
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
```

## What this means here

These two rules watch the CronJob (every 15 minutes) that repairs OIDC client **secrets** and
environment-derived client **URLs** in the Keycloak realm. They exist because of the single worst
green-looking outage this cluster has had, and they are best understood through it.

### The 2026-09-06 lockout — why "everything is green" means nothing here

The Keycloak Operator resolves each OIDC client's secret from an `(env:NAME)` placeholder in the realm
import. **When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the
import Job still reports SUCCESS.** On prod, **all 12 clients** ended up holding the literal placeholder
and **every OIDC login on the cluster failed** with *"Invalid client or Invalid client credentials"*.
Meanwhile: the import Job succeeded, the CR rested at `Done=True / Started=False`,
`KeycloakDown` and `KeycloakDatabaseDown` were quiet, and **the conformance test that asserts
`Done=True` passed throughout the outage.** The only observable was a human failing to log in. Dev had
exactly one broken client (oauth2-proxy — its key was added *after* dev's import completed, and
**a completed import is never re-run**).

The realm import is a **ONE-SHOT that fails open**: the operator logs *"Job already executed - not
recreating"*, so a git change to `keycloak-realm-import.yaml` does **not** re-import.

### The second half: redirect URIs, same one-shot, one field over

The realm import carries **26 `${BASE_DOMAIN}` references** (39 files under `platform/` and `apps/`
reference the knob). Substitution happened **once**, at the old hostname, and then froze. After the apex
flip, *"every OIDC login fails with a `redirect_uri` mismatch while the CR still reads `Done=True` and
nothing alerts."* The reconciler was extended with a second pass over `redirectUris`, `webOrigins`,
`rootUrl`, `baseUrl` and `adminUrl`, with desired state **generated** by
`ci/render-keycloak-client-urls.py` (gated by an L0 `--check`).

Semantics worth knowing before you read a Job log:
- URLs are copied **verbatim, including `${BASE_DOMAIN}`**, and rendered by the *same* Flux
  Kustomization as the import.
- **List fields compare order-insensitively (`sorted`, not `set`)** — measured, not defensive: Keycloak
  genuinely reorders lists, and an order-sensitive compare would have rewritten `kubernetes` (8 URIs) and
  `oauth2-proxy` (2) **every 15 minutes forever while reporting success**.
- A field the desired state omits is **UNMANAGED, never forced empty**.
- **Public clients' URLs are managed even though their secrets are not** (`kubernetes` is public, with
  PKCE, so no kubeconfig holds a secret).
- The two passes **fail closed independently** — demonstrated live: pass 1 failed on an empty secrets
  directory while pass 2 completed.

### Why there are exactly two rules, and what each catches

`...Failing` sees a run that *ran and failed* — unrepairable drift, or the admin API unreachable. It
fires alongside `KeycloakDown` during an IdP outage, deliberately: the reconciler genuinely is not
running, so treat `KeycloakDown` as the cause when both are lit. Self-clears when failed Jobs age out
(TTL 1h).

`...Stale` sees the **healer being quietly absent** — the CronJob suspended, never applied by Flux, or
unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists
to end. It is also the rule that would catch the healer missing during a domain migration.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

Does the CronJob exist, is it suspended, and when did it last succeed?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get cronjobs.batch \
     -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime'
```

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \
     -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15'
```

**Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never
prints a secret** (that is a design property, asserted by the L0 check):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \
     --sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200'
```

Is the IdP itself up? (If not, that is the cause; come back here after.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods; echo; \
   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak'
```

Confirm the declaration is self-consistent **offline**, before touching the cluster — this is the same
check the merge gate runs unconditionally (Stage 3d):

```
bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh
python3 ci/render-keycloak-client-urls.py --check
```

Confirm what a user actually experiences:

```
curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
```

```
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami'
```

(Expected on a working OIDC path: `oidc:<user>` with `[oidc:platform-admins]`. From this workstation the
OIDC context hangs on a browser prompt — that is not evidence of breakage.)

## Likely causes, most likely first

### 1. Keycloak or its database is down, so the reconciler cannot reach the admin API
`...Failing` fires alongside `KeycloakDown` by design. Fix the IdP first: `/runbooks/keycloak-down`,
`/runbooks/keycloak-db`. Nothing here is actionable until the admin API answers.

### 2. Unresolvable drift on one client
The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client
in the realm that git does not describe. The contract: keys are **`<clientId>-client-secret`**, and the
public `kubernetes` client declares **no** secret. Remediation is a git change to the declaration, then
let the CronJob heal it — the reconciler is compare-then-write and idempotent.

### 3. A hostname migration left the realm behind
The `redirect_uri` mismatch case. Regenerate desired state with `ci/render-keycloak-client-urls.py` and
push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex
produced **24 drifted fields across 12 clients**, with `kubernetes` correctly `ok` because its localhost
callbacks are not domain-derived.

### 4. The CronJob is suspended or was never applied (`...Stale`)
Check `spec.suspend` and the Flux tier. The ordering requirement is a **real Flux `dependsOn` edge with
`wait: true`**, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee.

### 5. The pod cannot read its own secrets
A specific, recorded trap: the reconciler needs **`fsGroup: 65532`** with the client-secrets volume at
**`defaultMode: 0440`**. With `0400` and no fsGroup the container dies with `PermissionError` on **every**
run. If the log shows that, this is it.

### 6. Everything is green and logins still fail
Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm
directly with `kcadm.sh` inside `keycloak-0`.

## When to escalate or stop

- **Do not "fix" the realm by re-applying the import.** It is a one-shot: the operator logs *"Job already
  executed - not recreating"* and nothing happens. The documented path is `kcadm.sh` inside `keycloak-0`
  (admin credentials in Secret `keycloak-admin-credentials`; `kcadm` is present, `curl` and `python3` are
  **not**) **and** fixing git so the change is reproducible. Doing only the live half creates drift; doing
  only the git half changes nothing.
- **Never print or paste a client secret.** The reconciler deliberately logs URL values in full (they are
  public hostnames) and never logs secrets. Keep that property.
- **Do not widen the reconciler's endpoint allow-list.** It is hard-scoped so that `/users`, `/groups`,
  `/roles` and `/authentication` are unreachable — a blast-radius bound, not an oversight.
- **Do not disable or suspend the CronJob to stop the alert.** That is the healer, and `...Stale` exists
  precisely because a suspended healer is invisible.
- Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a
  conditional-OTP misconfiguration once 403'd every platform-admin login with
  `"Invalid role name: null"`, fixed to `condUserRole: platform-admin`).

---
Source of truth: `runbooks/keycloak-client-secrets.md`. Index: `/runbooks/`
Records: `ci/l0.sh` Stage 3d and
`conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh` (the 2026-09-06 incident and the
reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed);
`DECISIONS-LOG.md` "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over".