home-network-k3s-impl/runbooks/keycloak-client-secrets.md

keycloak-client-secrets — The OIDC client reconciler is failing or has gone silent

Alerts: KeycloakClientSecretReconcilerFailing (warning, 15m) and KeycloakClientSecretReconcilerStale (warning, 2h) File: platform/monitoring/alerts.yaml, group platform-health.identity Expressions:

max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)

What this means here

These two rules watch the CronJob (every 15 minutes) that repairs OIDC client secrets and environment-derived client URLs in the Keycloak realm. They exist because of the single worst green-looking outage this cluster has had, and they are best understood through it.

The 2026-09-06 lockout — why "everything is green" means nothing here

The Keycloak Operator resolves each OIDC client's secret from an (env:NAME) placeholder in the realm import. When it cannot resolve one, it stores the LITERAL placeholder text as the client secret and the import Job still reports SUCCESS. On prod, all 12 clients ended up holding the literal placeholder and every OIDC login on the cluster failed with "Invalid client or Invalid client credentials". Meanwhile: the import Job succeeded, the CR rested at Done=True / Started=False, KeycloakDown and KeycloakDatabaseDown were quiet, and the conformance test that asserts Done=True passed throughout the outage. The only observable was a human failing to log in. Dev had exactly one broken client (oauth2-proxy — its key was added after dev's import completed, and a completed import is never re-run).

The realm import is a ONE-SHOT that fails open: the operator logs "Job already executed - not recreating", so a git change to keycloak-realm-import.yaml does not re-import.

The second half: redirect URIs, same one-shot, one field over

The realm import carries 26 ${BASE_DOMAIN} references (39 files under platform/ and apps/ reference the knob). Substitution happened once, at the old hostname, and then froze. After the apex flip, "every OIDC login fails with a redirect_uri mismatch while the CR still reads Done=True and nothing alerts." The reconciler was extended with a second pass over redirectUris, webOrigins, rootUrl, baseUrl and adminUrl, with desired state generated by ci/render-keycloak-client-urls.py (gated by an L0 --check).

Semantics worth knowing before you read a Job log:

• URLs are copied verbatim, including ${BASE_DOMAIN}, and rendered by the same Flux Kustomization as the import.
• List fields compare order-insensitively (sorted, not set) — measured, not defensive: Keycloak genuinely reorders lists, and an order-sensitive compare would have rewritten kubernetes (8 URIs) and oauth2-proxy (2) every 15 minutes forever while reporting success.
• A field the desired state omits is UNMANAGED, never forced empty.
• Public clients' URLs are managed even though their secrets are not (kubernetes is public, with PKCE, so no kubeconfig holds a secret).
• The two passes fail closed independently — demonstrated live: pass 1 failed on an empty secrets directory while pass 2 completed.

Why there are exactly two rules, and what each catches

...Failing sees a run that ran and failed — unrepairable drift, or the admin API unreachable. It fires alongside KeycloakDown during an IdP outage, deliberately: the reconciler genuinely is not running, so treat KeycloakDown as the cause when both are lit. Self-clears when failed Jobs age out (TTL 1h).

...Stale sees the healer being quietly absent — the CronJob suspended, never applied by Flux, or unable to schedule a pod at all. That is the "looks applied and isn't" failure this whole change exists to end. It is also the rule that would catch the healer missing during a domain migration.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and defaults its current-context to k3s-dev, so a command without --context silently answers about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.) Do not run two dev.sh invocations at once — they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Does the CronJob exist, is it suspended, and when did it last succeed?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get cronjobs.batch \
     -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime,LASTSUCCESS:.status.lastSuccessfulTime'

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get jobs.batch --sort-by=.metadata.creationTimestamp \
     -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed,START:.status.startTime | tail -15'

Read the Job log — it names the pass, the client, and for a URL the exact before/after, and it never prints a secret (that is a design property, asserted by the L0 check):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs job/$(kubectl --context k3s-prod -n keycloak get jobs.batch \
     --sort-by=.metadata.creationTimestamp -o jsonpath="{.items[-1:].metadata.name}") --tail=200'

Is the IdP itself up? (If not, that is the cause; come back here after.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak get pods; echo; \
   kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak'

Confirm the declaration is self-consistent offline, before touching the cluster — this is the same check the merge gate runs unconditionally (Stage 3d):

bash conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh
python3 ci/render-keycloak-client-urls.py --check

Confirm what a user actually experiences:

curl -s -o /dev/null -w '%{http_code}\n' --resolve keycloak.home.keithjackson.dev:443:192.168.2.20 \
  https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod auth whoami'

(Expected on a working OIDC path: oidc:<user> with [oidc:platform-admins]. From this workstation the OIDC context hangs on a browser prompt — that is not evidence of breakage.)

Likely causes, most likely first

1. Keycloak or its database is down, so the reconciler cannot reach the admin API

...Failing fires alongside KeycloakDown by design. Fix the IdP first: /runbooks/keycloak-down, /runbooks/keycloak-db. Nothing here is actionable until the admin API answers.

2. Unresolvable drift on one client

The log names the client and the pass. Most often a secret key missing from the SOPS Secret, or a client in the realm that git does not describe. The contract: keys are <clientId>-client-secret, and the public kubernetes client declares no secret. Remediation is a git change to the declaration, then let the CronJob heal it — the reconciler is compare-then-write and idempotent.

3. A hostname migration left the realm behind

The redirect_uri mismatch case. Regenerate desired state with ci/render-keycloak-client-urls.py and push; the reconciler's second pass applies it. Historical scale: re-rendering at the post-flip apex produced 24 drifted fields across 12 clients, with kubernetes correctly ok because its localhost callbacks are not domain-derived.

4. The CronJob is suspended or was never applied (...Stale)

Check spec.suspend and the Flux tier. The ordering requirement is a real Flux dependsOn edge with wait: true, not a comment — the L0 check asserts that, because a comment is not an ordering guarantee.

5. The pod cannot read its own secrets

A specific, recorded trap: the reconciler needs fsGroup: 65532 with the client-secrets volume at defaultMode: 0440. With 0400 and no fsGroup the container dies with PermissionError on every run. If the log shows that, this is it.

6. Everything is green and logins still fail

Then trust the human, not the dashboard — that is the entire lesson of 2026-09-06. Audit the live realm directly with kcadm.sh inside keycloak-0.

When to escalate or stop

• Do not "fix" the realm by re-applying the import. It is a one-shot: the operator logs "Job already executed - not recreating" and nothing happens. The documented path is kcadm.sh inside keycloak-0 (admin credentials in Secret keycloak-admin-credentials; kcadm is present, curl and python3 are not) and fixing git so the change is reproducible. Doing only the live half creates drift; doing only the git half changes nothing.
• Never print or paste a client secret. The reconciler deliberately logs URL values in full (they are public hostnames) and never logs secrets. Keep that property.
• Do not widen the reconciler's endpoint allow-list. It is hard-scoped so that /users, /groups, /roles and /authentication are unreachable — a blast-radius bound, not an oversight.
• Do not disable or suspend the CronJob to stop the alert. That is the healer, and ...Stale exists precisely because a suspended healer is invisible.
• Escalate for: rotating client secrets, editing the live realm, or anything touching MFA (a conditional-OTP misconfiguration once 403'd every platform-admin login with "Invalid role name: null", fixed to condUserRole: platform-admin).

---

Source of truth: runbooks/keycloak-client-secrets.md. Index: /runbooks/ Records: ci/l0.sh Stage 3d and conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh (the 2026-09-06 incident and the reconciler's contract — 212 L0 assertions, an 84-assertion selftest, 12/12 injected regressions killed); DECISIONS-LOG.md "Keycloak client REDIRECT URIs reconcile too — the same one-shot, one field over".