# platform/keycloak

**Keycloak** — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
- **Spec:** [docs/15 — Identity & Access](../../../home-network-k3s/docs/15-identity-and-access.md)
- **Serves:** R-IDP-01..06
- **Owner:** WS-SEC

## Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the **Bitnami Keycloak chart + bundled Bitnami PostgreSQL**. Bitnami's free
Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullBackOffs. Replaced with:

- **Keycloak: the official Keycloak Operator** (`quay.io/keycloak/keycloak-operator` +
  `quay.io/keycloak/keycloak`, v26.0.6) driving a **`Keycloak` CR** + a **`KeycloakRealmImport` CR** for
  config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
- **Database: CloudNativePG** (`ghcr.io/cloudnative-pg`, see [`../cnpg`](../cnpg)) — a `Cluster` on the
  3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release
files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.

## Layout (controllers / configs split — F-MON1)

| File | Applied by | What |
|---|---|---|
| `kustomization.yaml` (namespace, operator CRDs+Deployment, admin secret, dashboard) | **platform** | controllers |
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |

Realm config (groups `platform-admins`/`media-users`/`viewers`, OIDC clients `grafana`+`oauth2-proxy`,
brute-force ON, and a **conditional MFA browser flow that REQUIRES OTP for platform-admins**, R-IDP-04)
reconciles purely from git — no console clicking (R-IDP-03).

## Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned `keycloak-admin-credentials.sops.yaml` (keys `admin-user`/`admin-password`) is injected
as the temporary **bootstrap/break-glass admin** via `KC_BOOTSTRAP_ADMIN_*` env (podTemplate escape
hatch) — this consumes the secret **as provisioned, without re-keying it** (the operator's native
`bootstrapAdmin` wants `username`/`password` keys, which would require editing the encrypted file).
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

## Operator hand-offs (fill before / during deploy)

1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
   `<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
   import injects them into the Keycloak clients via `spec.placeholders`. Add to
   `kustomization.yaml` once filled.
   **The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
   from it, and CI rejects any other name.
   *(This step used to read "the realm import stays PENDING until this exists". It does not — see
   the next section. That false belief is what allowed the 2026-09-06 outage.)*
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
   `client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
   `grafana-client-secret` above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
exact keys + steps. No plaintext secret is ever committed.

## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

**The realm import is a one-shot that fails open.** Measured on both clusters:

* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
  text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
  error condition, and `HasErrors` stays `False`.
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
  added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
signal was green throughout, including the conformance test that asserts `Done=True`.

Three layers now cover it:

| Layer | Where | What it does |
|---|---|---|
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m, two passes | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ; a second, independent pass does the same for the client URLs against the generated `client-urls.yaml` (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch, literal placeholder **or drifted client URL**; the alerts make the reconciler's silence observable in prod. |

The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

**Manual repair / one-off run** (the CronJob normally does this):

```sh
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
```

## Client URLs: the same one-shot failure, one field over

Moving an environment's `BASE_DOMAIN` re-renders every declared `redirectUris` / `webOrigins` /
`rootUrl` / `baseUrl` / `adminUrl` in git — and reaches the realm **never**, because a completed
import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname
while git declared the new one. Every OIDC login then fails with a `redirect_uri` mismatch, the
`KeycloakRealmImport` still reads `Done=True`, and nothing alerts.

**Pass 2 of the same CronJob** closes it. It was made a second pass of the existing reconciler rather
than a sibling CronJob for one decisive reason: **a URL write needs no new endpoint.** It is the same
minimal `PUT /admin/realms/<realm>/clients/<uuid>`, and the current URLs already come back in the
allow-listed client lookup — so the hard allow-list that makes `/users`, `/groups`, `/roles` and
`/authentication` unreachable is widened by *nothing*, and is preserved rather than re-declared in a
second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would
have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)

**The desired state is generated, not authored.** `client-urls.yaml` is rendered from
`keycloak-realm-import.yaml` by `ci/render-keycloak-client-urls.py`, and the L0 gate runs that
renderer with `--check` on every merge, so the committed file cannot disagree with the import. URL
strings are copied verbatim — a `${BASE_DOMAIN}` token passes through unchanged, and both files are
rendered by the **same** Flux Kustomization, so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads `cluster-vars`: there is no second place the domain
can be decided.

```sh
# regenerate after editing a client URL in the realm import (the gate will demand it)
python3 ci/render-keycloak-client-urls.py --write
```

Properties, each proved by `--selftest` and each shown to be *falsifiable* by
`conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`:

* **Converged ⇒ zero writes.** List fields compare order-insensitively (`sorted`, not `set` —
  duplicates are still drift), because Keycloak does not preserve the authored order of
  `redirectUris`; an order-sensitive compare would rewrite every client forever while reporting
  success.
* **Gaining and losing a URI both work.** The declared list is exact, so a URI removed from git is
  removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect
  to.
* **Undeclared ⇒ unmanaged.** A field the desired state omits is left exactly as it is. Absence never
  means "force to empty".
* **Public clients' URLs *are* managed** (their secrets are not): `kubernetes` is PKCE-public and its
  localhost callbacks are what `kubectl` logins land on.
* **The two passes are independent.** Each reads its own desired state and fails closed for its own
  fields only. A malformed URL ConfigMap cannot stop client-*secret* repair, and an unmounted Secret
  cannot stop URL repair.
* URLs are logged in full (they are public hostnames, and the before/after is what makes a migration
  auditable); secrets are still never logged.

**What the reconciler does NOT fix:** anything outside client secrets and those five URL fields. A
NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
a realm edit landed.

## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)

`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
comments, not repeated here. The headline choices:

| Decision | Value | One-line reason |
|---|---|---|
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |

### This change does NOT reach the live realm. At all.

Same one-shot failure this file already documents twice above, third field over. Measured on prod
2026-10-17:

```
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
               {"status":"False","type":"HasErrors"}]}
```

`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
and three new sub-flows added to this file change nothing about how anyone logs in. The
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
mapper or a changed scope still needs the realm import to actually re-run" case named above.

**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
an operator applies it by hand.** Two paths, both operator actions:

1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
   into Username Form + Password Form and add the two sub-flows exactly as declared.
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.

Either way the realm is then configured by hand and git is only the record of intent, which is a
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.

### Verifying it actually works (nobody has, as of 2026-10-17)

No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
it, in order:

```sh
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
#    is wrong for the origin and NOTHING will say so.
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
#    prompt. A platform-admin must NOT be asked for TOTP on this path.
# 4. regression (the important one): a user with NO passkey must still get the password screen,
#    and a platform-admin must still be asked for TOTP there.
```

A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.

## Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).