feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa flow in the realm import. That declaration could not take effect: a KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so every realm-level field and flow declared after the first import is inert -- in git, applied by Flux, reported Ready, reaching the realm never. Merging it changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the 2026-09-06 client-secret incident, third field over. A SIBLING reconciler converges all three declared things every 15 minutes. It is a sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's allow-list -- and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable -- stays literally true. Its selftest still passes unchanged. pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of GET/PUT /admin/realms/homelab, readback-verified. pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE literally-named alias; enabled, defaultAction false so enrolment stays opt-in. pass 3 the browser flow -- BLUE/GREEN. PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a five-flow, ten-execution tree needs create-subflow, create-execution, set-requirement and bind-config as SEPARATE untransacted calls; doing that to the BOUND flow means a run dying between them leaves a half-built login path for every SSO app and kubectl, and this repo has had a total SSO outage from a realm change. So instead: 1. BUILD under a NEW, unused alias (declared alias + a hash of the desired shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow and zero user-visible effect. 2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the conditional-OTP config CONTENTS -- before anything is bound, so the regression that once 403'd every platform-admin is caught BEFORE it bites. 3. BIND with ONE atomic PUT setting browserFlow. 4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old flow survives as the rollback target. The widened allow-list is made safe structurally. It cannot say "but not the bound flow" -- its write entries are alias-parameterised -- so a second guard screens every write against the live browserFlow's WHOLE subtree (sub-flow aliases and execution ids) before a socket opens, FAIL-CLOSED until that binding has been read. The create-flow endpoint carries no alias in its path, so it is screened on its request BODY; the selftest caught that it otherwise escaped the guard. Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared STRUCTURALLY and a match produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately not compared -- a built flow carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged. Single source of truth: the desired state is GENERATED from the realm import by ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked into every credential at enrolment and never re-keyable -- is declared once. The pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no rbac.yaml is added anywhere. Read from Keycloak 26.0.6 source before being relied on: updateRealm (DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/ getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read getBrowserFlow(), which is what makes the atomic bind possible and why the policy write pins it; the passwordless policy is applied all-or-nothing only when RpEntityName is non-empty, with defaults substituted for omitted keys; and updateExecutions compares rep.getPriority(), so a body omitting `priority` would silently reshuffle the whole execution order. Gate coverage: a new whole-repo L0 check extracts the script and runs its --selftest (216 assertions, including a build killed at four different points each asserting the old flow is still bound and the bound subtree was never written), plus mutations.py (30 injected regressions, all caught) proving the selftest can fail. An earlier iteration of this work only AUDITED the flow and left the restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records that design and why it was reconsidered. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
1301af9a4c
commit
f5931944fa
9 files changed
+5200
-14
No files matched your search
+95
-14
@@ -31,6 +31,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
|
||||
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
||||
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
|
||||
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
|
||||
| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one) | **platform-config** | config |
|
||||
| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state) | **platform-config** | config |
|
||||
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
||||
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
||||
|
||||
@@ -154,7 +156,7 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
|
||||
|
||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
||||
@@ -170,7 +172,7 @@ comments, not repeated here. The headline choices:
|
||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
||||
|
||||
### This change does NOT reach the live realm. At all.
|
||||
### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
|
||||
|
||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
||||
2026-10-17:
|
||||
@@ -188,21 +190,100 @@ endpoint allow-list makes `/authentication` unreachable **by design** (see the t
|
||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
||||
|
||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
||||
an operator applies it by hand.** Two paths, both operator actions:
|
||||
`passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
|
||||
secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
|
||||
declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
|
||||
reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
|
||||
`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
|
||||
|
||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
||||
Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
|
||||
`ci/render-keycloak-passkey-policy.py` and gated by
|
||||
`conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
|
||||
That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
|
||||
and **can never be re-keyed**, so two declarations free to skew about the domain would be an
|
||||
unrecoverable-credential bug, not a cosmetic one.
|
||||
|
||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
||||
| Declared thing | Who applies it | Why |
|
||||
|---|---|---|
|
||||
| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
|
||||
| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
|
||||
| The `browser-mfa` flow restructure | **The reconciler. GitOps.** | Blue/green: build under a new alias, verify, flip `browserFlow` in one atomic PUT — see below |
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
||||
#### The flow converges blue/green — it never mutates the flow you log in through
|
||||
|
||||
Converging a five-flow / ten-execution restructure needs create-subflow,
|
||||
create-execution, set-requirement and bind-config as **separate** admin API calls, with no
|
||||
transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
|
||||
leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
|
||||
PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
|
||||
outage from a realm change, and the conditional-OTP sub-flow — which the declared
|
||||
structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
|
||||
|
||||
That risk is real but it is **not inherent**: it follows only from mutating the *bound*
|
||||
flow. So the reconciler does not.
|
||||
|
||||
1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
|
||||
hash of the desired shape, applied to every flow in the tree. Every multi-call write
|
||||
lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
|
||||
flow and **zero user-visible effect**.
|
||||
2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
|
||||
the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
|
||||
regression that once 403'd every platform-admin is caught **before** it can bite.
|
||||
3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow`.
|
||||
4. **Rollback is one write**, and nothing is ever deleted, so the old flow stays usable.
|
||||
|
||||
**The safety property is structural, not intentional.** The reconciler is *incapable* of
|
||||
writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
|
||||
whole subtree of sub-flow aliases and execution ids are read first and screened against
|
||||
every write before a socket opens. It is **fail-closed** — until that binding has been
|
||||
read, every flow write is refused, and if the bound flow's executions cannot be read the
|
||||
flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
|
||||
|
||||
**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
|
||||
would churn the login path on a schedule. The bound flow is compared structurally against
|
||||
the declared shape and, when they match, the run issues **zero writes and builds
|
||||
nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
|
||||
reconciler built carries a hash suffix, so comparing labels would make it permanently
|
||||
unrecognisable as converged.
|
||||
|
||||
**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
|
||||
later run reads it: if it matches desired it is a completed-but-unbound build and is
|
||||
simply **bound**; if not, it is left alone and the next candidate alias is used. The
|
||||
candidate count is **bounded**, so a persistently failing build fails loudly naming the
|
||||
flows to clean up rather than creating unbounded sprawl.
|
||||
|
||||
```sh
|
||||
# What the reconciler currently says. A normal run prints the full comparison.
|
||||
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
|
||||
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
|
||||
|
||||
# Force a run now rather than wait for the tick. NOTE: this runs the DEFAULT mode, which
|
||||
# converges (including flipping browserFlow once a new flow verifies). `kubectl create job
|
||||
# --from=cronjob/...` CANNOT override the container command, so there is no way to get
|
||||
# --audit or --no-bind out of it — for those, copy the CronJob's pod spec into a one-off
|
||||
# Job and append the flag.
|
||||
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
|
||||
passkey-run-once
|
||||
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
|
||||
|
||||
# ROLLBACK — one write. The run log names the previous alias ("ROLLBACK is one write:
|
||||
# set browserFlow back to ..."). The old flow is never deleted, so this always works.
|
||||
# browserFlow is a realm setting: Realm settings -> ... or
|
||||
# PUT /admin/realms/homelab with browserFlow set to the previous alias
|
||||
# (read-modify-write the realm representation; a partial body drops the passkey policy).
|
||||
# The admin console path is Authentication -> Flows -> <old flow> -> Action -> Bind flow
|
||||
# -> Browser flow. The console does this as one atomic change too.
|
||||
|
||||
# Regenerate the desired state after editing the passkey policy or the flow in the realm
|
||||
# import (the merge gate will demand it).
|
||||
python3 ci/render-keycloak-passkey-policy.py --write
|
||||
```
|
||||
|
||||
**If you would rather the go-live be a human act,** the reconciler takes `--no-bind`: it
|
||||
builds and verifies the flow and stops, printing the one write needed to bind it. That
|
||||
automates everything except one atomic operator action.
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-18)
|
||||
|
||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
||||
it, in order:
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
---
|
||||
# =============================================================================
|
||||
# GENERATED FILE — DO NOT EDIT BY HAND.
|
||||
#
|
||||
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
|
||||
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
|
||||
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
|
||||
# (runs this renderer with --check on every merge)
|
||||
#
|
||||
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
|
||||
# the WebAuthn passwordless policy, the enrolment required action, and the declared
|
||||
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
|
||||
# it and (a) converges the policy and the required action onto it, (b) AUDITS the
|
||||
# live browser flow against it and alarms on divergence without rewriting it.
|
||||
#
|
||||
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
|
||||
# once Done=True, so a realm-level field or a flow edited in git never reaches the
|
||||
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
|
||||
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
|
||||
# the level-triggered loop that closes that; this file is what it reconciles
|
||||
# TOWARDS.
|
||||
#
|
||||
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
|
||||
# user-verification requirement and the flow tree would be a second source of truth,
|
||||
# free to disagree with the realm import — the same "two declarations that can
|
||||
# disagree" shape as the bug being closed. It is rendered from the import and the
|
||||
# render is gated, so the two cannot drift.
|
||||
#
|
||||
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
|
||||
# substitution token exactly as the realm import writes it. Both files are rendered
|
||||
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
|
||||
# same value in both and the domain is decided in exactly one place.
|
||||
#
|
||||
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
|
||||
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
|
||||
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
|
||||
# nine more (a hard lock-out of every platform-admin). It is derived instead from
|
||||
# the presence of a webauthn-authenticator-passwordless execution in the flow:
|
||||
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
|
||||
# false, so enrolment stays opt-in.
|
||||
#
|
||||
# A field NOT declared on the realm in the import is simply ABSENT here, and the
|
||||
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: keycloak-passkey-desired-state
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-passkey-desired-state
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
passkey-desired-state.json: |
|
||||
{
|
||||
"authenticatorConfig": {
|
||||
"conditional-otp-platform-admin": {
|
||||
"condUserRole": "platform-admin",
|
||||
"negate": "false"
|
||||
}
|
||||
},
|
||||
"browserFlow": "browser-mfa",
|
||||
"flows": {
|
||||
"browser-mfa": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-cookie",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "identity-provider-redirector",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-forms",
|
||||
"requirement": "ALTERNATIVE"
|
||||
}
|
||||
],
|
||||
"topLevel": true
|
||||
},
|
||||
"browser-mfa-conditional-otp": {
|
||||
"executions": [
|
||||
{
|
||||
"configAlias": "conditional-otp-platform-admin",
|
||||
"kind": "authenticator",
|
||||
"name": "conditional-user-role",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-otp-form",
|
||||
"requirement": "REQUIRED"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-forms": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-username-form",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-passkey-or-password",
|
||||
"requirement": "REQUIRED"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-passkey-or-password": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "webauthn-authenticator-passwordless",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-password-and-otp",
|
||||
"requirement": "ALTERNATIVE"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-password-and-otp": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-password-form",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-conditional-otp",
|
||||
"requirement": "CONDITIONAL"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
}
|
||||
},
|
||||
"realm": "homelab",
|
||||
"requiredAction": {
|
||||
"alias": "webauthn-register-passwordless",
|
||||
"defaultAction": false,
|
||||
"enabled": true
|
||||
},
|
||||
"webAuthnPolicyPasswordless": {
|
||||
"webAuthnPolicyPasswordlessAcceptableAaguids": [],
|
||||
"webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
|
||||
"webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
|
||||
"webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
|
||||
"webAuthnPolicyPasswordlessCreateTimeout": 300,
|
||||
"webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
|
||||
"webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
|
||||
"webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
|
||||
"webAuthnPolicyPasswordlessSignatureAlgorithms": [
|
||||
"ES256",
|
||||
"RS256"
|
||||
],
|
||||
"webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user