home-network-k3s-impl/platform/keycloak/README.md

platform/keycloak

Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06
• Owner: WS-SEC

Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:

• Keycloak: the official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak, v26.0.6) driving a Keycloak CR + a KeycloakRealmImport CR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
• Database: CloudNativePG (ghcr.io/cloudnative-pg, see ../cnpg) — a Cluster on the 3-replica longhorn StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.

Layout (controllers / configs split — F-MON1)

File	Applied by	What
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard)	platform	controllers
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS})	platform-config	config
postgres-cluster.yaml (CNPG Cluster on longhorn)	platform-config	config
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA)	platform-config	config
client-secret-reconciler.yaml (CronJob: client secrets and client URLs ← git, every 15m)	platform-config	config
client-urls.yaml (GENERATED ConfigMap: the client-URL desired state)	platform-config	config
httproute.yaml (keycloak.home.keithjackson.dev)	platform-config	config
servicemonitor.yaml (Prometheus scrape)	platform-config	config

Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy, brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04) reconciles purely from git — no console clicking (R-IDP-03).

Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native bootstrapAdmin wants username/password keys, which would require editing the encrypted file). The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

Operator hand-offs (fill before / during deploy)

1. keycloak-realm-clients.sops.yaml (keycloak ns) — from .sops.yaml.example. One <clientId>-client-secret key per confidential client, openssl rand -hex 32 each. The realm import injects them into the Keycloak clients via spec.placeholders. Add to kustomization.yaml once filled. The key name MUST be <clientId>-client-secret — the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)
2. ../oauth2-proxy/oauth2-proxy-credentials.sops.yaml (oauth2-proxy ns) — client-id: oauth2-proxy, client-secret = the SAME oauth2-proxy-client-secret above, cookie-secret = openssl rand -base64 32.
3. ../monitoring/grafana-oidc-credentials.sops.yaml (monitoring ns) — client-secret = the SAME grafana-client-secret above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the exact keys + steps. No plaintext secret is ever committed.

Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

The realm import is a one-shot that fails open. Measured on both clusters:

• When the operator cannot resolve a $(env:...) placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, and HasErrors stays False.
• Once complete the CR rests at Done=True / Started=False and never re-runs, so a Secret key added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; every login rejected with "Invalid client or Invalid client credentials") and a single broken client on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster signal was green throughout, including the conformance test that asserts Done=True.

Three layers now cover it:

Layer	Where	What it does
Prevent	conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh)	Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable.
Heal	client-secret-reconciler.yaml — CronJob, every 15m, two passes	Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ; a second, independent pass does the same for the client URLs against the generated client-urls.yaml (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false).
Detect	conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale	Runs the same script in read-only --audit mode against the live realm and fails on any mismatch, literal placeholder or drifted client URL; the alerts make the reconciler's silence observable in prod.

The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm), no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

Manual repair / one-off run (the CronJob normally does this):

kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1

Client URLs: the same one-shot failure, one field over

Moving an environment's BASE_DOMAIN re-renders every declared redirectUris / webOrigins / rootUrl / baseUrl / adminUrl in git — and reaches the realm never, because a completed import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname while git declared the new one. Every OIDC login then fails with a redirect_uri mismatch, the KeycloakRealmImport still reads Done=True, and nothing alerts.

Pass 2 of the same CronJob closes it. It was made a second pass of the existing reconciler rather than a sibling CronJob for one decisive reason: a URL write needs no new endpoint. It is the same minimal PUT /admin/realms/<realm>/clients/<uuid>, and the current URLs already come back in the allow-listed client lookup — so the hard allow-list that makes /users, /groups, /roles and /authentication unreachable is widened by nothing, and is preserved rather than re-declared in a second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)

The desired state is generated, not authored. client-urls.yaml is rendered from keycloak-realm-import.yaml by ci/render-keycloak-client-urls.py, and the L0 gate runs that renderer with --check on every merge, so the committed file cannot disagree with the import. URL strings are copied verbatim — a ${BASE_DOMAIN} token passes through unchanged, and both files are rendered by the same Flux Kustomization, so both are substituted in one pass with one value. The renderer never parses a hostname and never reads cluster-vars: there is no second place the domain can be decided.

# regenerate after editing a client URL in the realm import (the gate will demand it)
python3 ci/render-keycloak-client-urls.py --write

Properties, each proved by --selftest and each shown to be falsifiable by conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py:

• Converged ⇒ zero writes. List fields compare order-insensitively (sorted, not set — duplicates are still drift), because Keycloak does not preserve the authored order of redirectUris; an order-sensitive compare would rewrite every client forever while reporting success.
• Gaining and losing a URI both work. The declared list is exact, so a URI removed from git is removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect to.
• Undeclared ⇒ unmanaged. A field the desired state omits is left exactly as it is. Absence never means "force to empty".
• Public clients' URLs are managed (their secrets are not): kubernetes is PKCE-public and its localhost callbacks are what kubectl logins land on.
• The two passes are independent. Each reads its own desired state and fails closed for its own fields only. A malformed URL ConfigMap cannot stop client-secret repair, and an unmounted Secret cannot stop URL repair.
• URLs are logged in full (they are public hostnames, and the before/after is what makes a migration auditable); secrets are still never logged.

What the reconciler does NOT fix: anything outside client secrets and those five URL fields. A NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run (it will not on its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit landed.

Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).