jakman85andClaude Opus 5
16894b7b73
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
228 lines
15 KiB
Markdown
228 lines
15 KiB
Markdown
# platform/keycloak
|
|
|
|
**Keycloak** — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
|
|
|
|
- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
|
|
- **Spec:** [docs/15 — Identity & Access](../../../home-network-k3s/docs/15-identity-and-access.md)
|
|
- **Serves:** R-IDP-01..06
|
|
- **Owner:** WS-SEC
|
|
|
|
## Image / tooling strategy (F-KY1 landmine avoided)
|
|
|
|
The original stub used the **Bitnami Keycloak chart + bundled Bitnami PostgreSQL**. Bitnami's free
|
|
Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullBackOffs. Replaced with:
|
|
|
|
- **Keycloak: the official Keycloak Operator** (`quay.io/keycloak/keycloak-operator` +
|
|
`quay.io/keycloak/keycloak`, v26.0.6) driving a **`Keycloak` CR** + a **`KeycloakRealmImport` CR** for
|
|
config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
|
|
- **Database: CloudNativePG** (`ghcr.io/cloudnative-pg`, see [`../cnpg`](../cnpg)) — a `Cluster` on the
|
|
3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
|
|
|
|
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release
|
|
files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
|
|
|
|
## Layout (controllers / configs split — F-MON1)
|
|
|
|
| File | Applied by | What |
|
|
|---|---|---|
|
|
| `kustomization.yaml` (namespace, operator CRDs+Deployment, admin secret, dashboard) | **platform** | controllers |
|
|
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
|
|
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
|
|
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
|
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
|
|
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
|
|
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
|
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
|
|
|
Realm config (groups `platform-admins`/`media-users`/`viewers`, OIDC clients `grafana`+`oauth2-proxy`,
|
|
brute-force ON, and a **conditional MFA browser flow that REQUIRES OTP for platform-admins**, R-IDP-04)
|
|
reconciles purely from git — no console clicking (R-IDP-03).
|
|
|
|
## Admin credential + break-glass (R-IDP-04 / R-IDP-05)
|
|
|
|
The provisioned `keycloak-admin-credentials.sops.yaml` (keys `admin-user`/`admin-password`) is injected
|
|
as the temporary **bootstrap/break-glass admin** via `KC_BOOTSTRAP_ADMIN_*` env (podTemplate escape
|
|
hatch) — this consumes the secret **as provisioned, without re-keying it** (the operator's native
|
|
`bootstrapAdmin` wants `username`/`password` keys, which would require editing the encrypted file).
|
|
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
|
|
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
|
|
|
|
## Operator hand-offs (fill before / during deploy)
|
|
|
|
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
|
|
`<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
|
|
import injects them into the Keycloak clients via `spec.placeholders`. Add to
|
|
`kustomization.yaml` once filled.
|
|
**The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
|
|
from it, and CI rejects any other name.
|
|
*(This step used to read "the realm import stays PENDING until this exists". It does not — see
|
|
the next section. That false belief is what allowed the 2026-09-06 outage.)*
|
|
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
|
|
`client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
|
|
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
|
|
`grafana-client-secret` above.
|
|
|
|
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
|
|
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
|
|
exact keys + steps. No plaintext secret is ever committed.
|
|
|
|
## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
|
|
|
|
**The realm import is a one-shot that fails open.** Measured on both clusters:
|
|
|
|
* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
|
|
text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
|
|
error condition, and `HasErrors` stays `False`.
|
|
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
|
|
added later is never picked up.
|
|
|
|
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
|
|
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
|
|
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
|
|
signal was green throughout, including the conformance test that asserts `Done=True`.
|
|
|
|
Three layers now cover it:
|
|
|
|
| Layer | Where | What it does |
|
|
|---|---|---|
|
|
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
|
|
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m, two passes | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ; a second, independent pass does the same for the client URLs against the generated `client-urls.yaml` (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
|
|
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch, literal placeholder **or drifted client URL**; the alerts make the reconciler's silence observable in prod. |
|
|
|
|
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
|
|
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
|
|
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
|
|
|
|
**Manual repair / one-off run** (the CronJob normally does this):
|
|
|
|
```sh
|
|
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
|
|
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
|
|
```
|
|
|
|
## Client URLs: the same one-shot failure, one field over
|
|
|
|
Moving an environment's `BASE_DOMAIN` re-renders every declared `redirectUris` / `webOrigins` /
|
|
`rootUrl` / `baseUrl` / `adminUrl` in git — and reaches the realm **never**, because a completed
|
|
import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname
|
|
while git declared the new one. Every OIDC login then fails with a `redirect_uri` mismatch, the
|
|
`KeycloakRealmImport` still reads `Done=True`, and nothing alerts.
|
|
|
|
**Pass 2 of the same CronJob** closes it. It was made a second pass of the existing reconciler rather
|
|
than a sibling CronJob for one decisive reason: **a URL write needs no new endpoint.** It is the same
|
|
minimal `PUT /admin/realms/<realm>/clients/<uuid>`, and the current URLs already come back in the
|
|
allow-listed client lookup — so the hard allow-list that makes `/users`, `/groups`, `/roles` and
|
|
`/authentication` unreachable is widened by *nothing*, and is preserved rather than re-declared in a
|
|
second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would
|
|
have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)
|
|
|
|
**The desired state is generated, not authored.** `client-urls.yaml` is rendered from
|
|
`keycloak-realm-import.yaml` by `ci/render-keycloak-client-urls.py`, and the L0 gate runs that
|
|
renderer with `--check` on every merge, so the committed file cannot disagree with the import. URL
|
|
strings are copied verbatim — a `${BASE_DOMAIN}` token passes through unchanged, and both files are
|
|
rendered by the **same** Flux Kustomization, so both are substituted in one pass with one value. The
|
|
renderer never parses a hostname and never reads `cluster-vars`: there is no second place the domain
|
|
can be decided.
|
|
|
|
```sh
|
|
# regenerate after editing a client URL in the realm import (the gate will demand it)
|
|
python3 ci/render-keycloak-client-urls.py --write
|
|
```
|
|
|
|
Properties, each proved by `--selftest` and each shown to be *falsifiable* by
|
|
`conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`:
|
|
|
|
* **Converged ⇒ zero writes.** List fields compare order-insensitively (`sorted`, not `set` —
|
|
duplicates are still drift), because Keycloak does not preserve the authored order of
|
|
`redirectUris`; an order-sensitive compare would rewrite every client forever while reporting
|
|
success.
|
|
* **Gaining and losing a URI both work.** The declared list is exact, so a URI removed from git is
|
|
removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect
|
|
to.
|
|
* **Undeclared ⇒ unmanaged.** A field the desired state omits is left exactly as it is. Absence never
|
|
means "force to empty".
|
|
* **Public clients' URLs *are* managed** (their secrets are not): `kubernetes` is PKCE-public and its
|
|
localhost callbacks are what `kubectl` logins land on.
|
|
* **The two passes are independent.** Each reads its own desired state and fails closed for its own
|
|
fields only. A malformed URL ConfigMap cannot stop client-*secret* repair, and an unmounted Secret
|
|
cannot stop URL repair.
|
|
* URLs are logged in full (they are public hostnames, and the before/after is what makes a migration
|
|
auditable); secrets are still never logged.
|
|
|
|
**What the reconciler does NOT fix:** anything outside client secrets and those five URL fields. A
|
|
NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run
|
|
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
|
a realm edit landed.
|
|
|
|
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
|
|
|
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
|
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
|
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
|
|
comments, not repeated here. The headline choices:
|
|
|
|
| Decision | Value | One-line reason |
|
|
|---|---|---|
|
|
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
|
|
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
|
|
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
|
|
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
|
|
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
|
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
|
|
|
### This change does NOT reach the live realm. At all.
|
|
|
|
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
|
2026-10-17:
|
|
|
|
```
|
|
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
|
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
|
|
{"status":"False","type":"HasErrors"}]}
|
|
```
|
|
|
|
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
|
|
and three new sub-flows added to this file change nothing about how anyone logs in. The
|
|
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
|
|
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
|
|
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
|
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
|
|
|
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
|
an operator applies it by hand.** Two paths, both operator actions:
|
|
|
|
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
|
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
|
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
|
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
|
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
|
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
|
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
|
|
|
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
|
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
|
|
|
### Verifying it actually works (nobody has, as of 2026-10-17)
|
|
|
|
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
|
it, in order:
|
|
|
|
```sh
|
|
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
|
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
|
|
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
|
|
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
|
|
# is wrong for the origin and NOTHING will say so.
|
|
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
|
|
# prompt. A platform-admin must NOT be asked for TOTP on this path.
|
|
# 4. regression (the important one): a user with NO passkey must still get the password screen,
|
|
# and a platform-admin must still be asked for TOTP there.
|
|
```
|
|
|
|
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
|
|
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
|
|
|
|
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
|
|
|
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|