home-network-k3s-impl/platform/keycloak/README.md

platform/keycloak

Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06
• Owner: WS-SEC

Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:

• Keycloak: the official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak, v26.0.6) driving a Keycloak CR + a KeycloakRealmImport CR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
• Database: CloudNativePG (ghcr.io/cloudnative-pg, see ../cnpg) — a Cluster on the 3-replica longhorn StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern). Renovate tracks the version via # renovate: markers on those URLs (since 2026-10-30 — before that it could not see them) and never auto-merges it; see Upgrading Keycloak.

Layout (controllers / configs split — F-MON1)

File	Applied by	What
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard)	platform	controllers
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS})	platform-config	config
postgres-cluster.yaml (CNPG Cluster on longhorn)	platform-config	config
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA)	platform-config	config
client-secret-reconciler.yaml (CronJob: client secrets and client URLs ← git, every 15m)	platform-config	config
client-urls.yaml (GENERATED ConfigMap: the client-URL desired state)	platform-config	config
passkey-reconciler.yaml (CronJob: WebAuthn passwordless policy + enrolment action + the browser flow ← git every 15m; the flow blue/green, never mutating the bound one, bound only when git says so)	platform-config	config
passkey-desired-state.yaml (GENERATED ConfigMap: the passkey desired state — plus the one hand-edited line, bind-browser-flow)	platform-config	config
httproute.yaml (keycloak.home.keithjackson.dev)	platform-config	config
servicemonitor.yaml (Prometheus scrape)	platform-config	config

Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy, brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04) reconciles purely from git — no console clicking (R-IDP-03).

Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native bootstrapAdmin wants username/password keys, which would require editing the encrypted file). The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

Operator hand-offs (fill before / during deploy)

1. keycloak-realm-clients.sops.yaml (keycloak ns) — from .sops.yaml.example. One <clientId>-client-secret key per confidential client, openssl rand -hex 32 each. The realm import injects them into the Keycloak clients via spec.placeholders. Add to kustomization.yaml once filled. The key name MUST be <clientId>-client-secret — the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)
2. ../oauth2-proxy/oauth2-proxy-credentials.sops.yaml (oauth2-proxy ns) — client-id: oauth2-proxy, client-secret = the SAME oauth2-proxy-client-secret above, cookie-secret = openssl rand -base64 32.
3. ../monitoring/grafana-oidc-credentials.sops.yaml (monitoring ns) — client-secret = the SAME grafana-client-secret above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the exact keys + steps. No plaintext secret is ever committed.

Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

The realm import is a one-shot that fails open. Measured on both clusters:

• When the operator cannot resolve a $(env:...) placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, and HasErrors stays False.
• Once complete the CR rests at Done=True / Started=False and never re-runs, so a Secret key added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; every login rejected with "Invalid client or Invalid client credentials") and a single broken client on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster signal was green throughout, including the conformance test that asserts Done=True.

Three layers now cover it:

Layer	Where	What it does
Prevent	conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh)	Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable.
Heal	client-secret-reconciler.yaml — CronJob, every 15m, two passes	Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ; a second, independent pass does the same for the client URLs against the generated client-urls.yaml (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false).
Detect	conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale	Runs the same script in read-only --audit mode against the live realm and fails on any mismatch, literal placeholder or drifted client URL; the alerts make the reconciler's silence observable in prod.

The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm), no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

Manual repair / one-off run (the CronJob normally does this):

kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1

Client URLs: the same one-shot failure, one field over

Moving an environment's BASE_DOMAIN re-renders every declared redirectUris / webOrigins / rootUrl / baseUrl / adminUrl in git — and reaches the realm never, because a completed import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname while git declared the new one. Every OIDC login then fails with a redirect_uri mismatch, the KeycloakRealmImport still reads Done=True, and nothing alerts.

Pass 2 of the same CronJob closes it. It was made a second pass of the existing reconciler rather than a sibling CronJob for one decisive reason: a URL write needs no new endpoint. It is the same minimal PUT /admin/realms/<realm>/clients/<uuid>, and the current URLs already come back in the allow-listed client lookup — so the hard allow-list that makes /users, /groups, /roles and /authentication unreachable is widened by nothing, and is preserved rather than re-declared in a second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)

The desired state is generated, not authored. client-urls.yaml is rendered from keycloak-realm-import.yaml by ci/render-keycloak-client-urls.py, and the L0 gate runs that renderer with --check on every merge, so the committed file cannot disagree with the import. URL strings are copied verbatim — a ${BASE_DOMAIN} token passes through unchanged, and both files are rendered by the same Flux Kustomization, so both are substituted in one pass with one value. The renderer never parses a hostname and never reads cluster-vars: there is no second place the domain can be decided.

# regenerate after editing a client URL in the realm import (the gate will demand it)
python3 ci/render-keycloak-client-urls.py --write

Properties, each proved by --selftest and each shown to be falsifiable by conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py:

• Converged ⇒ zero writes. List fields compare order-insensitively (sorted, not set — duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris; an order-sensitive compare would rewrite every client forever while reporting success.
• Gaining and losing a URI both work. The declared list is exact, so a URI removed from git is removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect to.
• Undeclared ⇒ unmanaged. A field the desired state omits is left exactly as it is. Absence never means "force to empty".
• Public clients' URLs are managed (their secrets are not): kubernetes is PKCE-public and its localhost callbacks are what kubectl logins land on.
• The two passes are independent. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-secret repair, and an unmounted Secret cannot stop URL repair.
• URLs are logged in full (they are public hostnames, and the before/after is what makes a migration auditable); secrets are still never logged.

What the reconciler does NOT fix: anything outside client secrets and those five URL fields. A NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run (it will not on its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit landed.

Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)

keycloak-realm-import.yaml now declares passkey login: the realm's WebAuthn Passwordless policy (webAuthnPolicyPasswordless*), and a browser-mfa flow that offers a passkey as an ALTERNATIVE beside the password form. Design decisions and their reasoning are in the manifest comments, not repeated here. The headline choices:

Decision	Value	One-line reason
Additive or replacing?	Additive — passkey or password	A misfiring mandatory flow locks the single operator out of every SSO app and kubectl
Enrolment	Optional, never a default action	Same reason; webauthn-register-passwordless is already enabled-not-forced by default
Relying-party ID	${BASE_DOMAIN} (not keycloak.${BASE_DOMAIN})	An unre-keyable credential must not be bound to a hostname that can move
User verification	required	Without it, mere possession of an unlocked device is a complete login
Resident key	Yes	A non-discoverable credential is a second factor, not a passkey
OTP after a passkey?	No — conditional OTP stays on the password branch	A UV-required passkey is already two factors, and a phishing-resistant pair at that

The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)

Same one-shot failure this file already documents twice above, third field over. Measured on prod 2026-10-17:

$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
               {"status":"False","type":"HasErrors"}]}

Done=True / Started=False — the import has completed and will not re-run, so a policy block and three new sub-flows added to this file change nothing about how anyone logs in. The client-secret/client-URL reconciler does not close this gap and was never meant to: its hard endpoint allow-list makes /authentication unreachable by design (see the table above), and realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run" case named above.

passkey-reconciler.yaml closes that gap the same way the client reconciler closed the secret and URL halves: a CronJob on */15 that authenticates to the admin API and converges declared state. It is a sibling, not a second pass of the client reconciler, precisely so that reconciler's allow-list — and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable — stays literally true.

Its desired state is passkey-desired-state.yaml, generated from the realm import by ci/render-keycloak-passkey-policy.py and gated by conformance/tests/l0-static/keycloak-passkey-reconciler, so the RP ID is declared exactly once. That matters more here than for client URLs: an RP ID is baked into every credential at enrolment and can never be re-keyed, so two declarations free to skew about the domain would be an unrecoverable-credential bug, not a cosmetic one.

Declared thing	Who applies it	Why
webAuthnPolicyPasswordless* (realm policy)	The reconciler. GitOps.	Realm top-level attributes; strict read-modify-write of GET/PUT /admin/realms/homelab
webauthn-register-passwordless required action	The reconciler. GitOps.	Per-alias PUT, one literally-named alias; enabled, defaultAction: false
The browser-mfa flow restructure	The reconciler builds and verifies it. Git decides whether it is bound.	Blue/green: build under a new alias, verify, and flip browserFlow in one atomic PUT only when bind-browser-flow: "true" — see below

The flow converges blue/green — it never mutates the flow you log in through

Converging a five-flow / ten-execution restructure needs create-subflow, create-execution, set-requirement and bind-config as separate admin API calls, with no transaction. Doing that to the flow bound as browserFlow would mean a run dying part-way leaves a half-built login path for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the *arr stack and kubectl. This file already documents a total SSO outage from a realm change, and the conditional-OTP sub-flow — which the declared structure moves one level deeper than it sits today — once 403'd every platform-admin.

That risk is real but it is not inherent: it follows only from mutating the bound flow. So the reconciler does not.

1. Build the declared tree under a new, unused alias — the declared alias plus a hash of the desired shape, applied to every flow in the tree. Every multi-call write lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound flow and zero user-visible effect.
2. Verify by readback — nesting levels, ordering, authenticators, requirements, and the conditional-OTP config contents. Nothing is bound unless this passes, so the regression that once 403'd every platform-admin is caught before it can bite.
3. Bind with one atomic PUT of the realm, setting browserFlow — only if git says so (next section). Otherwise the run stops here and logs WOULD BIND browser-mfa-<hash>.
4. Rollback is a PR plus one write (below), and nothing is ever deleted, so the old flow stays usable.

The bind switch: bind-browser-flow in passkey-desired-state.yaml

That generated file carries exactly one hand-edited line, bind-browser-flow, and it is "false" by default. The renderer carries it through on --write, and --check refuses anything but the quoted strings "true" / "false" (an unquoted true is a YAML boolean, which a ConfigMap rejects at apply time and which would fail the whole platform-config Kustomization). The CronJob reads it from the same mount as the desired state; no flag can widen it.

bind-browser-flow	What the reconciler does to the realm's browserFlow
"false" (default)	Never writes it. It builds and verifies the candidate (or reuses an already verified one), logs WOULD BIND <alias>, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it).
"true"	Binds the verified candidate in one atomic PUT, and keeps asserting it: a binding changed by hand is changed back within 15 minutes, and a hand-edit inside the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is "true".

Merging with "false" changes nothing about how anyone logs in. Passes 1 and 2 do write the realm (the passwordless policy and the enrolment action, neither of which any login step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but browserFlow is untouched.

Going live is a later one-line PR — bind-browser-flow: "true" — made after the candidate has been tested in a browser (see "Testing the candidate flow before binding it").

Rolling back, in this order:

1. A PR setting bind-browser-flow: "false", merged and applied by Flux. First — while it is "true", the next run rebinds the candidate over your rollback.
2. One write: bind the previous flow again. The Job log of the bind printed it (ROLLBACK is one write: set browserFlow back to ...); it is browser-mfa for the first go-live. Admin console: Authentication → Flows → → Action → Bind flow → Browser flow, or PUT /admin/realms/homelab as a read-modify-write of the full realm representation (a partial body drops the passkey policy).

If login is broken right now, do step 2 immediately and step 1 straight after: the worst case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux applies the PR, and you repeat step 2.

The safety property is structural, not intentional. The reconciler is incapable of writing to the bound flow or anything beneath it: the live browserFlow alias and its whole subtree of sub-flow aliases and execution ids are read first and screened against every write before a socket opens. It is fail-closed — until that binding has been read, every flow write is refused, and if the bound flow's executions cannot be read the flow pass writes nothing at all. There is no DELETE anywhere in its allow-list.

Idempotence is load-bearing, not a nicety: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared structurally against the declared shape and, when they match, the run issues zero writes and builds nothing. Sub-flow and config aliases are deliberately not compared — a flow the reconciler built carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged.

Recovery. A crashed run can leave a partial flow under the primary candidate alias. A later run reads it: if it matches desired it is a completed-but-unbound build and is simply bound; if not, it is left alone and the next candidate alias is used. The candidate count is bounded, so a persistently failing build fails loudly naming the flows to clean up rather than creating unbounded sprawl.

# What the reconciler currently says. A normal run prints the full comparison.
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200

# Which candidate it would bind (bind-browser-flow "false"):
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
  | grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'

# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
# override the container command, so --audit / --no-bind need a one-off Job with the flag
# appended (--no-bind can only NARROW git's switch, never widen it).
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
  passkey-run-once
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once

# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
# Browser flow). The old flow is never deleted, so this always works.

# Regenerate the desired state after editing the passkey policy or the flow in the realm
# import (the merge gate will demand it).
python3 ci/render-keycloak-passkey-policy.py --write

Testing the candidate flow before binding it

With bind-browser-flow: "false" the candidate exists but nothing logs in through it. Test it on one client without touching the realm binding, using a per-client flow override:

1. Find the candidate: the Job log line WOULD BIND browser-mfa-<hash>.
2. Admin console → Clients → account-console → Advanced → Authentication flow overrides → Browser Flow = browser-mfa-<hash>. This is a temporary, hand-made client change; remove it in step 7.
3. Password fallback. Private window → https://keycloak.<base domain>/realms/homelab/account. Expect a username screen, then a choice: passkey, or the password form. As a user with no passkey, choose password and sign in.
4. Platform-admin TOTP on the password branch. Repeat step 3 as a platform-admins member: after the password, a TOTP prompt must appear. If it does not — or you get a 403 — the conditional-OTP step is wrong: stop, do not bind.
5. Passkey enrolment. Signed in to the account console → Signing in → Passkey → Set up. A browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and nothing server-side will say so.
6. Passkey login. Sign out; private window again; username, then choose passkey. Expect to be signed in with no TOTP prompt, even as a platform-admin.
7. Remove the override from step 2.
8. Only now: the one-line PR bind-browser-flow: "true". The next run binds the candidate you tested (same alias — the hash is of the declared shape).

Verifying it actually works (nobody has, as of 2026-10-29)

No passkey has been enrolled and no browser has been pointed at this. The checks that would prove it on the BOUND flow, after go-live, in order:

# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
#    is wrong for the origin and NOTHING will say so.
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
#    prompt. A platform-admin must NOT be asked for TOTP on this path.
# 4. regression (the important one): a user with NO passkey must still get the password screen,
#    and a platform-admin must still be asked for TOTP there.

A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.

Upgrading Keycloak: the operator IS the version

The Keycloak CR has no spec.image, so the server image is whatever the operator's RELATED_IMAGE_KEYCLOAK names — the same tag as the operator (prod, 2026-10-30: operator 26.0.6, keycloak-0/1 on quay.io/keycloak/keycloak:26.0.6). Operator, CRDs and server are released in lockstep from keycloak/keycloak-k8s-resources, and the release URLs in kustomization.yaml are the one knob. Renovate raises them as ONE PR (keycloak-operator group, labels needs-review + blast-radius), never auto-merged; majors wait for a dashboard tick.

Renovate bumps the URLs that exist; it cannot add a CRD a release introduced. Before merging a bump, diff kubernetes/ (and its kustomization.yml) at the new tag against the URL list. 26.0.6 shipped 2 CRDs, 26.8.2 ships 4 (keycloakoidcclients, keycloaksamlclients added); PR #121 bumped only the existing URLs and the 26.8.2 operator CrashLooped on the missing keycloaksamlclients informer, stalling Flux platform and every tier below it (2026-11-08, decisions/2026-11-08-keycloak-operator-crds.md). The operator's memory is also patched in kustomization.yaml (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps. catalog/keycloak.yaml's image tag is documentation only (its render deploys nothing) and Renovate is disabled for it — update it by hand to match after an operator bump merges.

An operator bump requires re-checking the passkey reconciler's assumptions. passkey-reconciler.yaml (and ci/render-keycloak-passkey-policy.py, which generates its desired state) is written against Keycloak 26.0.6 Admin API behaviour read from that tag's source — grep it for 26.0.6. The gate cannot catch a break: it is static and never runs the reconciler against a server. Before merging a bump, diff the new tag against 26.0.6 for at least:

• AuthenticationManagementResource.recurseExecutions — does a sub-flow row still carry its name in displayName (not alias)? The reconciler's flow matching depends on it.
• updateExecutions / execution-requirement updates — same request shape and id resolution?
• the WebAuthn Passwordless policy — is it still gated all-or-nothing on a non-empty webAuthnPolicyPasswordlessRpEntityName, and does a realm PUT still REPLACE rather than merge requiredActions? (The strict read-modify-write exists because of both.)
• authenticator-config alias handling for copied flows.

Then, after the merge reconciles (operator actions): watch the Keycloak CR reach Ready and both pods roll; run the reconciler once with --audit (see "Passkeys" above for the one-off Job) and confirm it reports convergence with no write; confirm the client-secret and client-URL reconcilers' next runs succeed; log in to one SSO app and run one kubectl --context k3s-prod OIDC command. Roll back by reverting the PR — but a server minor may migrate the database schema on start, so take/confirm a fresh CNPG backup before merging.

Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).