home-network-k3s-impl/platform/keycloak/README.md

platform/keycloak

Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06
• Owner: WS-SEC

Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:

• Keycloak: the official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak, v26.0.6) driving a Keycloak CR + a KeycloakRealmImport CR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
• Database: CloudNativePG (ghcr.io/cloudnative-pg, see ../cnpg) — a Cluster on the 3-replica longhorn StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.

Layout (controllers / configs split — F-MON1)

File	Applied by	What
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard)	platform	controllers
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS})	platform-config	config
postgres-cluster.yaml (CNPG Cluster on longhorn)	platform-config	config
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA)	platform-config	config
client-secret-reconciler.yaml (CronJob: client secrets and client URLs ← git, every 15m)	platform-config	config
client-urls.yaml (GENERATED ConfigMap: the client-URL desired state)	platform-config	config
passkey-reconciler.yaml (CronJob: WebAuthn passwordless policy + enrolment action + the browser flow ← git every 15m; the flow blue/green, never mutating the bound one)	platform-config	config
passkey-desired-state.yaml (GENERATED ConfigMap: the passkey desired state)	platform-config	config
httproute.yaml (keycloak.home.keithjackson.dev)	platform-config	config
servicemonitor.yaml (Prometheus scrape)	platform-config	config

Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy, brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04) reconciles purely from git — no console clicking (R-IDP-03).

Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native bootstrapAdmin wants username/password keys, which would require editing the encrypted file). The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

Operator hand-offs (fill before / during deploy)

1. keycloak-realm-clients.sops.yaml (keycloak ns) — from .sops.yaml.example. One <clientId>-client-secret key per confidential client, openssl rand -hex 32 each. The realm import injects them into the Keycloak clients via spec.placeholders. Add to kustomization.yaml once filled. The key name MUST be <clientId>-client-secret — the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)
2. ../oauth2-proxy/oauth2-proxy-credentials.sops.yaml (oauth2-proxy ns) — client-id: oauth2-proxy, client-secret = the SAME oauth2-proxy-client-secret above, cookie-secret = openssl rand -base64 32.
3. ../monitoring/grafana-oidc-credentials.sops.yaml (monitoring ns) — client-secret = the SAME grafana-client-secret above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the exact keys + steps. No plaintext secret is ever committed.

Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

The realm import is a one-shot that fails open. Measured on both clusters:

• When the operator cannot resolve a $(env:...) placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, and HasErrors stays False.
• Once complete the CR rests at Done=True / Started=False and never re-runs, so a Secret key added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; every login rejected with "Invalid client or Invalid client credentials") and a single broken client on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster signal was green throughout, including the conformance test that asserts Done=True.

Three layers now cover it:

Layer	Where	What it does
Prevent	conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh)	Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable.
Heal	client-secret-reconciler.yaml — CronJob, every 15m, two passes	Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ; a second, independent pass does the same for the client URLs against the generated client-urls.yaml (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false).
Detect	conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale	Runs the same script in read-only --audit mode against the live realm and fails on any mismatch, literal placeholder or drifted client URL; the alerts make the reconciler's silence observable in prod.

The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm), no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

Manual repair / one-off run (the CronJob normally does this):

kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1

Client URLs: the same one-shot failure, one field over

Moving an environment's BASE_DOMAIN re-renders every declared redirectUris / webOrigins / rootUrl / baseUrl / adminUrl in git — and reaches the realm never, because a completed import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname while git declared the new one. Every OIDC login then fails with a redirect_uri mismatch, the KeycloakRealmImport still reads Done=True, and nothing alerts.

Pass 2 of the same CronJob closes it. It was made a second pass of the existing reconciler rather than a sibling CronJob for one decisive reason: a URL write needs no new endpoint. It is the same minimal PUT /admin/realms/<realm>/clients/<uuid>, and the current URLs already come back in the allow-listed client lookup — so the hard allow-list that makes /users, /groups, /roles and /authentication unreachable is widened by nothing, and is preserved rather than re-declared in a second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)

The desired state is generated, not authored. client-urls.yaml is rendered from keycloak-realm-import.yaml by ci/render-keycloak-client-urls.py, and the L0 gate runs that renderer with --check on every merge, so the committed file cannot disagree with the import. URL strings are copied verbatim — a ${BASE_DOMAIN} token passes through unchanged, and both files are rendered by the same Flux Kustomization, so both are substituted in one pass with one value. The renderer never parses a hostname and never reads cluster-vars: there is no second place the domain can be decided.

# regenerate after editing a client URL in the realm import (the gate will demand it)
python3 ci/render-keycloak-client-urls.py --write

Properties, each proved by --selftest and each shown to be falsifiable by conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py:

• Converged ⇒ zero writes. List fields compare order-insensitively (sorted, not set — duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris; an order-sensitive compare would rewrite every client forever while reporting success.
• Gaining and losing a URI both work. The declared list is exact, so a URI removed from git is removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect to.
• Undeclared ⇒ unmanaged. A field the desired state omits is left exactly as it is. Absence never means "force to empty".
• Public clients' URLs are managed (their secrets are not): kubernetes is PKCE-public and its localhost callbacks are what kubectl logins land on.
• The two passes are independent. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-secret repair, and an unmounted Secret cannot stop URL repair.
• URLs are logged in full (they are public hostnames, and the before/after is what makes a migration auditable); secrets are still never logged.

What the reconciler does NOT fix: anything outside client secrets and those five URL fields. A NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run (it will not on its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit landed.

Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)

keycloak-realm-import.yaml now declares passkey login: the realm's WebAuthn Passwordless policy (webAuthnPolicyPasswordless*), and a browser-mfa flow that offers a passkey as an ALTERNATIVE beside the password form. Design decisions and their reasoning are in the manifest comments, not repeated here. The headline choices:

Decision	Value	One-line reason
Additive or replacing?	Additive — passkey or password	A misfiring mandatory flow locks the single operator out of every SSO app and kubectl
Enrolment	Optional, never a default action	Same reason; webauthn-register-passwordless is already enabled-not-forced by default
Relying-party ID	${BASE_DOMAIN} (not keycloak.${BASE_DOMAIN})	An unre-keyable credential must not be bound to a hostname that can move
User verification	required	Without it, mere possession of an unlocked device is a complete login
Resident key	Yes	A non-discoverable credential is a second factor, not a passkey
OTP after a passkey?	No — conditional OTP stays on the password branch	A UV-required passkey is already two factors, and a phishing-resistant pair at that

The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)

Same one-shot failure this file already documents twice above, third field over. Measured on prod 2026-10-17:

$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
               {"status":"False","type":"HasErrors"}]}

Done=True / Started=False — the import has completed and will not re-run, so a policy block and three new sub-flows added to this file change nothing about how anyone logs in. The client-secret/client-URL reconciler does not close this gap and was never meant to: its hard endpoint allow-list makes /authentication unreachable by design (see the table above), and realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run" case named above.

passkey-reconciler.yaml closes that gap the same way the client reconciler closed the secret and URL halves: a CronJob on */15 that authenticates to the admin API and converges declared state. It is a sibling, not a second pass of the client reconciler, precisely so that reconciler's allow-list — and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable — stays literally true.

Its desired state is passkey-desired-state.yaml, generated from the realm import by ci/render-keycloak-passkey-policy.py and gated by conformance/tests/l0-static/keycloak-passkey-reconciler, so the RP ID is declared exactly once. That matters more here than for client URLs: an RP ID is baked into every credential at enrolment and can never be re-keyed, so two declarations free to skew about the domain would be an unrecoverable-credential bug, not a cosmetic one.

Declared thing	Who applies it	Why
webAuthnPolicyPasswordless* (realm policy)	The reconciler. GitOps.	Realm top-level attributes; strict read-modify-write of GET/PUT /admin/realms/homelab
webauthn-register-passwordless required action	The reconciler. GitOps.	Per-alias PUT, one literally-named alias; enabled, defaultAction: false
The browser-mfa flow restructure	The reconciler. GitOps.	Blue/green: build under a new alias, verify, flip browserFlow in one atomic PUT — see below

The flow converges blue/green — it never mutates the flow you log in through

Converging a five-flow / ten-execution restructure needs create-subflow, create-execution, set-requirement and bind-config as separate admin API calls, with no transaction. Doing that to the flow bound as browserFlow would mean a run dying part-way leaves a half-built login path for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the *arr stack and kubectl. This file already documents a total SSO outage from a realm change, and the conditional-OTP sub-flow — which the declared structure moves one level deeper than it sits today — once 403'd every platform-admin.

That risk is real but it is not inherent: it follows only from mutating the bound flow. So the reconciler does not.

1. Build the declared tree under a new, unused alias — the declared alias plus a hash of the desired shape, applied to every flow in the tree. Every multi-call write lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound flow and zero user-visible effect.
2. Verify by readback — nesting levels, ordering, authenticators, requirements, and the conditional-OTP config contents. Nothing is bound unless this passes, so the regression that once 403'd every platform-admin is caught before it can bite.
3. Bind with one atomic PUT of the realm, setting browserFlow.
4. Rollback is one write, and nothing is ever deleted, so the old flow stays usable.

The safety property is structural, not intentional. The reconciler is incapable of writing to the bound flow or anything beneath it: the live browserFlow alias and its whole subtree of sub-flow aliases and execution ids are read first and screened against every write before a socket opens. It is fail-closed — until that binding has been read, every flow write is refused, and if the bound flow's executions cannot be read the flow pass writes nothing at all. There is no DELETE anywhere in its allow-list.

Idempotence is load-bearing, not a nicety: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared structurally against the declared shape and, when they match, the run issues zero writes and builds nothing. Sub-flow and config aliases are deliberately not compared — a flow the reconciler built carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged.

Recovery. A crashed run can leave a partial flow under the primary candidate alias. A later run reads it: if it matches desired it is a completed-but-unbound build and is simply bound; if not, it is left alone and the next candidate alias is used. The candidate count is bounded, so a persistently failing build fails loudly naming the flows to clean up rather than creating unbounded sprawl.

# What the reconciler currently says. A normal run prints the full comparison.
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200

# Force a run now rather than wait for the tick. NOTE: this runs the DEFAULT mode, which
# converges (including flipping browserFlow once a new flow verifies). `kubectl create job
# --from=cronjob/...` CANNOT override the container command, so there is no way to get
# --audit or --no-bind out of it — for those, copy the CronJob's pod spec into a one-off
# Job and append the flag.
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
  passkey-run-once
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once

# ROLLBACK — one write. The run log names the previous alias ("ROLLBACK is one write:
# set browserFlow back to ..."). The old flow is never deleted, so this always works.
# browserFlow is a realm setting: Realm settings -> ... or
#   PUT /admin/realms/homelab  with browserFlow set to the previous alias
# (read-modify-write the realm representation; a partial body drops the passkey policy).
# The admin console path is Authentication -> Flows -> <old flow> -> Action -> Bind flow
# -> Browser flow. The console does this as one atomic change too.

# Regenerate the desired state after editing the passkey policy or the flow in the realm
# import (the merge gate will demand it).
python3 ci/render-keycloak-passkey-policy.py --write

If you would rather the go-live be a human act, the reconciler takes --no-bind: it builds and verifies the flow and stops, printing the one write needed to bind it. That automates everything except one atomic operator action.

Verifying it actually works (nobody has, as of 2026-10-18)

No passkey has been enrolled and no browser has been pointed at this. The checks that would prove it, in order:

# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
#    is wrong for the origin and NOTHING will say so.
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
#    prompt. A platform-admin must NOT be asked for TOTP on this path.
# 4. regression (the important one): a user with NO passkey must still get the password screen,
#    and a platform-admin must still be asked for TOTP there.

A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.

Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).