home-network-k3s-impl/platform/keycloak

platform/keycloak

Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06
• Owner: WS-SEC

Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:

• Keycloak: the official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak, v26.0.6) driving a Keycloak CR + a KeycloakRealmImport CR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
• Database: CloudNativePG (ghcr.io/cloudnative-pg, see ../cnpg) — a Cluster on the 3-replica longhorn StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.

Layout (controllers / configs split — F-MON1)

File	Applied by	What
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard)	platform	controllers
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS})	platform-config	config
postgres-cluster.yaml (CNPG Cluster on longhorn)	platform-config	config
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA)	platform-config	config
client-secret-reconciler.yaml (CronJob: client secrets and client URLs ← git, every 15m)	platform-config	config
client-urls.yaml (GENERATED ConfigMap: the client-URL desired state)	platform-config	config
httproute.yaml (keycloak.home.keithjackson.dev)	platform-config	config
servicemonitor.yaml (Prometheus scrape)	platform-config	config

Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy, brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04) reconciles purely from git — no console clicking (R-IDP-03).

Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native bootstrapAdmin wants username/password keys, which would require editing the encrypted file). The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

Operator hand-offs (fill before / during deploy)

1. keycloak-realm-clients.sops.yaml (keycloak ns) — from .sops.yaml.example. One <clientId>-client-secret key per confidential client, openssl rand -hex 32 each. The realm import injects them into the Keycloak clients via spec.placeholders. Add to kustomization.yaml once filled. The key name MUST be <clientId>-client-secret — the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)
2. ../oauth2-proxy/oauth2-proxy-credentials.sops.yaml (oauth2-proxy ns) — client-id: oauth2-proxy, client-secret = the SAME oauth2-proxy-client-secret above, cookie-secret = openssl rand -base64 32.
3. ../monitoring/grafana-oidc-credentials.sops.yaml (monitoring ns) — client-secret = the SAME grafana-client-secret above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the exact keys + steps. No plaintext secret is ever committed.

Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

The realm import is a one-shot that fails open. Measured on both clusters:

• When the operator cannot resolve a $(env:...) placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, and HasErrors stays False.
• Once complete the CR rests at Done=True / Started=False and never re-runs, so a Secret key added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; every login rejected with "Invalid client or Invalid client credentials") and a single broken client on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster signal was green throughout, including the conformance test that asserts Done=True.

Three layers now cover it:

Layer	Where	What it does
Prevent	conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh)	Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable.
Heal	client-secret-reconciler.yaml — CronJob, every 15m, two passes	Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ; a second, independent pass does the same for the client URLs against the generated client-urls.yaml (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false).
Detect	conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale	Runs the same script in read-only --audit mode against the live realm and fails on any mismatch, literal placeholder or drifted client URL; the alerts make the reconciler's silence observable in prod.

The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm), no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

Manual repair / one-off run (the CronJob normally does this):

kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1

Client URLs: the same one-shot failure, one field over

Moving an environment's BASE_DOMAIN re-renders every declared redirectUris / webOrigins / rootUrl / baseUrl / adminUrl in git — and reaches the realm never, because a completed import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname while git declared the new one. Every OIDC login then fails with a redirect_uri mismatch, the KeycloakRealmImport still reads Done=True, and nothing alerts.

Pass 2 of the same CronJob closes it. It was made a second pass of the existing reconciler rather than a sibling CronJob for one decisive reason: a URL write needs no new endpoint. It is the same minimal PUT /admin/realms/<realm>/clients/<uuid>, and the current URLs already come back in the allow-listed client lookup — so the hard allow-list that makes /users, /groups, /roles and /authentication unreachable is widened by nothing, and is preserved rather than re-declared in a second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)

The desired state is generated, not authored. client-urls.yaml is rendered from keycloak-realm-import.yaml by ci/render-keycloak-client-urls.py, and the L0 gate runs that renderer with --check on every merge, so the committed file cannot disagree with the import. URL strings are copied verbatim — a ${BASE_DOMAIN} token passes through unchanged, and both files are rendered by the same Flux Kustomization, so both are substituted in one pass with one value. The renderer never parses a hostname and never reads cluster-vars: there is no second place the domain can be decided.

# regenerate after editing a client URL in the realm import (the gate will demand it)
python3 ci/render-keycloak-client-urls.py --write

Properties, each proved by --selftest and each shown to be falsifiable by conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py:

• Converged ⇒ zero writes. List fields compare order-insensitively (sorted, not set — duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris; an order-sensitive compare would rewrite every client forever while reporting success.
• Gaining and losing a URI both work. The declared list is exact, so a URI removed from git is removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect to.
• Undeclared ⇒ unmanaged. A field the desired state omits is left exactly as it is. Absence never means "force to empty".
• Public clients' URLs are managed (their secrets are not): kubernetes is PKCE-public and its localhost callbacks are what kubectl logins land on.
• The two passes are independent. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-secret repair, and an unmounted Secret cannot stop URL repair.
• URLs are logged in full (they are public hostnames, and the before/after is what makes a migration auditable); secrets are still never logged.

What the reconciler does NOT fix: anything outside client secrets and those five URL fields. A NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run (it will not on its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit landed.

Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)

keycloak-realm-import.yaml now declares passkey login: the realm's WebAuthn Passwordless policy (webAuthnPolicyPasswordless*), and a browser-mfa flow that offers a passkey as an ALTERNATIVE beside the password form. Design decisions and their reasoning are in the manifest comments, not repeated here. The headline choices:

Decision	Value	One-line reason
Additive or replacing?	Additive — passkey or password	A misfiring mandatory flow locks the single operator out of every SSO app and kubectl
Enrolment	Optional, never a default action	Same reason; webauthn-register-passwordless is already enabled-not-forced by default
Relying-party ID	${BASE_DOMAIN} (not keycloak.${BASE_DOMAIN})	An unre-keyable credential must not be bound to a hostname that can move
User verification	required	Without it, mere possession of an unlocked device is a complete login
Resident key	Yes	A non-discoverable credential is a second factor, not a passkey
OTP after a passkey?	No — conditional OTP stays on the password branch	A UV-required passkey is already two factors, and a phishing-resistant pair at that

This change does NOT reach the live realm. At all.

Same one-shot failure this file already documents twice above, third field over. Measured on prod 2026-10-17:

$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
               {"status":"False","type":"HasErrors"}]}

Done=True / Started=False — the import has completed and will not re-run, so a policy block and three new sub-flows added to this file change nothing about how anyone logs in. The client-secret/client-URL reconciler does not close this gap and was never meant to: its hard endpoint allow-list makes /authentication unreachable by design (see the table above), and realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run" case named above.

So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until an operator applies it by hand. Two paths, both operator actions:

1. Admin console (expected path). Authentication → Policies → WebAuthn Passwordless Policy for the policy block, then Authentication → Flows → browser-mfa to rebuild the executions to match the manifest. The flow edit is the fiddly half: split the existing Username Password Form into Username Form + Password Form and add the two sub-flows exactly as declared.
2. A deliberate re-import. Deleting and recreating the KeycloakRealmImport re-runs it against the existing realm. Do not reach for this casually — it is the operation whose failure mode is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.

Either way the realm is then configured by hand and git is only the record of intent, which is a live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.

Verifying it actually works (nobody has, as of 2026-10-17)

No passkey has been enrolled and no browser has been pointed at this. The checks that would prove it, in order:

# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
#    is wrong for the origin and NOTHING will say so.
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
#    prompt. A platform-admin must NOT be asked for TOTP on this path.
# 4. regression (the important one): a user with NO passkey must still get the password screen,
#    and a platform-admin must still be asked for TOTP there.

A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.

Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).