home-network-k3s-impl/decisions/2026-10-18-keycloak-passkey-reconciler.md

date, branch, title

date	branch	title
2026-10-18	feat/keycloak-passkey-reconciler	Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated

What changed

Why this is a new branch and not an update to PR #72. The brief for this work said to extend PR #72 (feat/keycloak-passkeys) in place and not open a new PR. That premise was already stale when I started: PR #72 merged at 2026-10-18T05:44:04Z (Gitea API: state: closed, merged: true, merge_commit_sha: 4d489f2), and git log origin/main carries 4d489f2 Merge pull request '…' (#72). A merged PR cannot be updated, and CLAUDE.md §1 leaves exactly one path to main. So this is a fresh branch from origin/main in its own worktree, and a new PR. For the same reason this is a new decisions/ entry rather than an edit to decisions/2026-10-17-keycloak-passkeys.md, which is now merged history describing a different unit of work; it is the entry to read first for the passkey design decisions, which are not repeated here.

The problem. PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa flow in platform/keycloak/keycloak-realm-import.yaml. That declaration could not take effect. A KeycloakRealmImport is a one-shot; prod's reads Done=True / Started=False / HasErrors=False, so the operator will never re-run it and every realm-level field or flow declared after the first import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the 2026-09-06 client-secret incident, third field over.

What now makes it converge. A sibling reconciler, modelled on platform/keycloak/client-secret-reconciler.yaml:

• platform/keycloak/passkey-reconciler.yaml — ConfigMap (Python, stdlib only, zero dollar signs) + CronJob on */15 * * * *, automountServiceAccountToken: false, admin credential by secretKeyRef only. Three passes, all of which converge:
  1. WebAuthn passwordless policy. Strict read-modify-write of GET/PUT /admin/realms/homelab, changing only the webAuthnPolicyPasswordless* keys that differ, readback-verified.
  2. webauthn-register-passwordless required action. Strict read-modify-write of one literally-named alias; converges enabled: true / defaultAction: false only.
  3. The browser authentication flow — BLUE/GREEN. Build the declared tree under a new, unused alias; verify it by readback; flip browserFlow in one atomic PUT.
• ci/render-keycloak-passkey-policy.py + the generated platform/keycloak/passkey-desired-state.yaml — the desired state, rendered from the realm import, --checked by the merge gate.
• conformance/tests/l0-static/keycloak-passkey-reconciler/ — check.sh, chainsaw-test.yaml (scope: whole-repo), mutations.py.
• platform-config/kustomization.yaml — both new resources, in the same Kustomization as the realm import.
• platform/keycloak/README.md — the layout table, the section heading, and the "does NOT reach the live realm" section rewritten: all three declared things now converge, with the blue/green mechanics, the rollback command and the --no-bind fallback written out.

The flow design was reconsidered, and the first version of it was wrong

This is worth recording because the superseded reasoning was not silly, it was just wrongly scoped. The first design of pass 3 was audit-only: compare the live browser flow against git, report divergence loudly, and never write it — leaving the restructure as a one-time admin-console action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow, create-execution, set-requirement and bind-config as separate untransacted admin-API calls, so a run dying between them leaves a half-built flow bound as the login path for every SSO-gated app and kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every platform-admin.

That risk analysis was sound but it assumed the wrong shape. It is true only if you mutate the bound flow. The operator's instruction was "if this can be done through GitOps, it should", and audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already forces. So:

• All the multi-call construction happens on a flow nothing is using (the declared alias plus a hash of the desired shape, applied to every alias in the tree — reusing browser-mfa-forms would mean writing into the bound tree). A failure at any point leaves an orphaned, unbound flow and zero user-visible effect. The untransacted-calls problem dissolves.
• The built flow is read back and verified — nesting levels, ordering, authenticators, requirements, and the conditional-OTP config contents — before anything is bound. The regression the operator feared is now caught before it can bite rather than after.
• The flip is one atomic PUT of the realm setting browserFlow.
• Rollback is one write. The previous alias is printed in the run log, and there is no DELETE anywhere in the allow-list, so the old flow survives as a working rollback target.

The new safety property, enforced structurally rather than by intention. Blue/green needs flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone cannot express "but not the flow that is bound", because its write entries are alias-parameterised. So a second guard sits at the same choke point in _http(): the live browserFlow alias and its entire subtree of sub-flow aliases and execution ids are read first, and every write is screened against those sets before a socket opens. Protecting the whole subtree matters — adding an execution to browser-mfa-forms changes live login behaviour just as surely as touching browser-mfa. It is fail-closed: the sets start each run as "not determined", and while they are, every flow write is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass writes nothing.

Idempotence became load-bearing rather than merely nice. A rebuild-and-reflip every 15 minutes would churn the login path on a schedule, so the bound flow is compared structurally and a match produces zero writes and no new flow. Sub-flow and config aliases are deliberately not compared: a flow this program built carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is reused; a half-built one is left alone and the next candidate alias is used, with the candidate count bounded so a persistently failing build fails loudly instead of creating unbounded sprawl.

Where the rollback target is recorded, and why not as an annotation. The brief suggested recording the previous browserFlow alias as an annotation on the CronJob or in a ConfigMap. The pod cannot: it runs with automountServiceAccountToken: false and has no Kubernetes API access at all, which is the same property that made the mounted-generated-ConfigMap design preferable to reading the CR (and is why this change adds no **/rbac.yaml). Giving it write access to its own annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse bargain than the one it would buy.

So the record is in two places that cost nothing: the run log prints ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted., and — more durably — the realm itself, because nothing is ever deleted. Every superseded flow is still listed under Authentication -> Flows, so the rollback target is discoverable from the admin console even if every Job log has rotated away. The README carries the rollback procedure.

I achieved full convergence including the bind. --no-bind exists as the deliberate fallback (build and verify, leave the flip as a one-write human act) but it is not the default and was not needed.

No **/rbac.yaml was touched. The brief floated having the reconciler read the KeycloakRealmImport CR from the Kubernetes API, which would need a Role, a RoleBinding and a ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state generated from the import and mounted, so the pod keeps automountServiceAccountToken: false and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with --check, not by a second declaration being trusted.

Sensitive paths touched — checked against infra/gitea/approval-policy.json rather than from memory, which caught one I had initially missed. Three:

Path	Why it is in the policy	What I changed
ci/**	the pipeline itself	added ci/render-keycloak-passkey-policy.py
conformance/**	changing the tests changes what green means	added the L0 test directory
platform-config/kustomization.yaml	Flux wiring / which manifests are actually applied	added the two new resources

Not touched: **/rbac.yaml, clusters/**, platform/kyverno/**, .gitea/**, bootstrap/**, infra/gitea/**, .sops.yaml, any *.sops.yaml, CLAUDE.md.

So this PR requires an approving review from someone who is not its author, and ci/approval-policy.sh is fail-closed, so it will block until that exists. I did not post /approve-override — CLAUDE.md §1 names doing that on your own agent PR as exactly the self-approval the stage exists to prevent. The operator needs to either review it or post the override.

What I verified

Measured against the live cluster (read-only kubectl --context k3s-prod):

• keycloakrealmimport homelab: Done=True Started=False HasErrors=False — the import is a completed one-shot. This is the premise of the whole change and it is measured, not assumed.
• Keycloak image actually running: quay.io/keycloak/keycloak:26.0.6.
• kustomization/platform-config .status.lastAppliedRevision == gitrepository/flux-system artifact revision == main@sha1:83d3568…, which contains the PR #72 merge. So the inert declaration is genuinely applied and genuinely inert.

PUT /admin/realms/{realm} does not touch users, groups, clients or roles in 26.0.6 — established from source at tag 26.0.6, not from documentation or memory:

• RealmAdminResource.updateRealm is the only handler. Its javadoc: "Any user, roles or client information in the representation will be ignored. This will only update top-level attributes of the realm."
• It delegates to RepresentationToModel.updateRealm → DefaultExportImportManager.updateRealm (model/storage-private/.../datastore/), lines 725–894. I brace-matched that body and enumerated every rep.getX() call in it: it contains no getUsers(), getClients(), getGroups(), getRoles(), getRequiredActions(), getAuthenticationFlows() or getAuthenticatorConfig(). Those objects are unreachable from the endpoint, not merely ignored by convention.
• Two findings that changed the design, both from the same read:
  • updateRealm does read rep.getBrowserFlow(), so the endpoint can rebind the realm's login path. The write therefore carries browserFlow straight back from the GET and a hard assertion refuses the PUT if it would differ.
  • The passwordless policy is applied all-or-nothing and only when RpEntityName is non-empty (line 842), and getWebAuthnPolicyPasswordless(rep) substitutes a Keycloak default for every null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and one missing RpEntityName would drop the whole policy while returning 204. This is why the write is a strict read-modify-write of the full live representation.
• AuthenticationManagementResource.updateRequiredAction (line 1205) is per-alias and likewise replaces the provider from the representation (name, alias, priority, enabled, defaultAction, config), and 404s when the alias is unregistered. Hence read-modify-write there too.
• DefaultRequiredActions registers webauthn-register-passwordless with enabled=true, defaultAction=false, priority=80 — confirming the realm import's claim that declaring it would restate a default while deleting ten siblings.
• The realm GET really does return what the read-modify-write needs. This is load-bearing: if the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset them to defaults. ModelToRepresentation.toRepresentation (lines 471–482) sets all ten webAuthnPolicyPasswordless* fields plus ExtraOrigins, and line 496 sets browserFlow. I checked for an internal/export guard around that block — there is none in lines 380–500, so those fields are emitted unconditionally, not only on the export path.

The five flow-construction endpoints, read from 26.0.6 source before being used:

• POST /authentication/flows — AuthenticationManagementResource.createFlow, rejects a duplicate alias. Carries no alias in its path (the alias is in the body), which is why it needed its own body-based screen; without that it escaped the protected-subtree guard entirely. My selftest caught that, not my reading.
• POST /flows/{alias}/executions/flow — addExecutionFlow, body {alias, type, description}. Creates the sub-flow at requirement DISABLED, and refuses if the parent isBuiltIn().
• POST /flows/{alias}/executions/execution — addExecutionToFlow, body {provider}. Also DISABLED unless the factory has exactly one requirement choice. Both assign getNextPriority(), so insertion order is the resulting order — which is why the builder inserts depth-first in declared order and then verifies by readback.
• PUT /flows/{alias}/executions — updateExecutions. A trap I would have walked into: it compares rep.getPriority() against the stored priority, and a JSON body omitting priority deserialises it as the int 0 — which would silently reshuffle the whole execution order while only a requirement change was intended. Every requirement write therefore carries the execution's current priority back. There is a mutation for it and a selftest assertion on the resulting order.
• POST /executions/{id}/config — newExecutionConfig, creates an authenticator config and binds it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and the old flow already uses the declared alias.

Offline proofs, all green:

• The reconciler's --selftest, extracted from the shipped ConfigMap and run: 216 assertions pass. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the all-or-nothing policy rebuild, the required-action replace, the updateExecutions priority trap, and flow construction where everything is created DISABLED — so these are tested against the real semantics rather than a forgiving fake.
• A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds and never touches the live flow. Each case asserts the realm is still bound to the old alias, that no write targeted the bound subtree, and that every live flow survives. This is the assertion that makes blue/green a property rather than a claim.
• Verify-before-bind is exercised, not assumed. Two stub sabotages — a server that accepts the requirement writes and applies the wrong value, and one that writes the conditional-OTP role under condition.evaluated.user.role — both end with the flow built but NOT bound and the old flow still live.
• mutations.py: all 30 injected regressions caught, including the six that are the blue/green safety argument itself (build into the bound flow; never call the screen; reset the protected sets to empty; bind before verify; report a failed build as success; reuse an existing flow without verifying it).
• Mutation testing again found real gaps in my own work, which I fixed rather than weakening the mutations. Across the two designs: no readback coverage on the realm write, no coverage of an undeclared policy key surviving, a mis-indented anchor, and — in this design — no coverage at all of the bound flow being unreadable, which is the fail-closed path. It also showed me that two of my proposed mutations were not real regressions: the module-level initial value of the protected sets is dead code (run() resets them first), and removing the early return after a build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that are load-bearing rather than pretending the originals had been caught.
• The new L0 check.sh: rc 0, all assertions. It asserts the renderer --check; that the realm import and the desired state are rendered by the same Flux Kustomization; that the renderer's and reconciler's writable key tuples are identical (parsed out of both sources with ast); that defaultAction is false; that the realm import declares no requiredActions block (previously only a comment saying "DO NOT ADD ONE"); that the sibling reconciler's allow-list was not widened; that nothing is deletable and the execution-/config-id-scoped write endpoints stay off the list (they carry no flow alias, so they could not be screened); and — the load-bearing one — that the protected-subtree guard exists in the source, is wired into _http(), resets to "undetermined" every run, screens the body-only create-flow endpoint, and that the policy write still refuses to rebind browserFlow.
• ci/render-keycloak-passkey-policy.py --write output round-trips: the generated JSON is accepted by the reconciler's declared(), its 10 policy keys equal the reconciler's POLICY_FIELDS, and it flattens to the exact 11-step tree the realm import declares, with conditional-otp-platform-admin bound at level 4.
• ${BASE_DOMAIN} is carried into the generated ConfigMap verbatim, and both files sit in platform-config, so one substitution pass decides the RP ID.
• Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed manifest.

The conditional-OTP regression is detectable — and now also preventable. The comparison is key-by-key on the authenticator config contents, and because it runs before the bind, a flow carrying the bug never becomes the live login path. --selftest covers it three ways: the historical condition.evaluated.user.role key injected into a built flow (built, verified, not bound, old flow still live); the same bug already live in the bound flow (reported, then fixed by building and binding a correct flow); and the config disappearing entirely. mutations.py deletes the comparison and the undeclared-key report and requires the selftest to notice both. The suite also covers a requirement flipped from ALTERNATIVE to REQUIRED, the un-restructured legacy flow, a correct-but-unbound flow, and a cyclic declaration.

Gate: see the gate section of the PR / the run quoted in my report — rc and per-stage results are recorded there rather than paraphrased here.

What I did NOT verify

• The live realm's actual passkey state is entirely UNVERIFIED. I never authenticated to the Keycloak admin API. Doing so needs the keycloak-admin-credentials password, which CLAUDE.md §6 forbids decrypting or printing, so I did not read the realm's current webAuthnPolicyPasswordless* values, its required-action list, or its browser-mfa executions. I therefore do not know how much drift the first real run will report, nor that the live flow is in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
• Nothing has ever run against a real Keycloak. Every behavioural proof is against an in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This matters more for the blue/green flow builder than for anything else in the change, because it is five endpoints I have never exercised. Specifically unproven in reality:
  • That POST /flows/{alias}/executions/flow and .../executions/execution accept the bodies I send and assign priorities in insertion order. If they do not, the build produces a tree that fails verification — which is the safe failure (nothing is bound, login unaffected), but it means the flow would never converge and the Job would fail every 15 minutes.
  • That PUT /admin/realms/homelab accepts a full round-tripped representation. The analogous client endpoint rejects one with 400 "Cannot parse the JSON" (measured 2026-09-06), and the sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm endpoint differs; no measurement here confirms it. If it does not, both the policy pass and the atomic bind fail closed — loudly, writing nothing.
  • The exact JSON shapes compared: AcceptableAaguids as [] rather than absent or null, CreateTimeout as a JSON int, RequireResidentKey as the literal string "Yes". A shape I did not anticipate would show up as perpetual drift — a write every 15 minutes that never converges — rather than as an error.
  • That authenticationConfig on an execution-info response is the config id (I read it from the representation class, but never saw a live response).
• I did not verify the alias-collision behaviour against a real realm. The recovery logic assumes POST /flows returns a non-2xx on a duplicate alias (the source checks getFlowByAlias != null and raises ErrorResponse.exists), and that a config alias collision likewise fails. If either silently succeeded, the recovery path would behave differently than modelled.
• Flow sprawl is bounded but real. Every genuinely new desired shape creates a new flow and nothing is ever deleted — by design, because the old flow is the rollback target. Over many shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented beyond "an operator deletes them". I consider that the right trade (deletion is the one operation that could destroy a rollback target) but it is a cost, not a non-issue.
• No L2 behavioural test was added. The reconciler has an --audit mode built for one and the sibling has tests/l2-behavioral/keycloak-client-secrets-match-git, but I added no equivalent. And vars.CLUSTER_AVAILABLE is unset, so L2/L3 would not run in CI anyway. CI proves the offline subset only; it proves nothing about the live realm.
• No alerting. Unlike the first design there is no longer any failure that is "expected by design", so a PrometheusRule on kube_job_status_failed would now be meaningful — and I did not add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
• That passkey login works. No passkey has been enrolled and no browser has been pointed at this. Nothing offline can prove a WebAuthn ceremony completes, that the ALTERNATIVE fork degrades to the password form via CREDENTIAL_SETUP_REQUIRED, or that a platform-admin is still asked for TOTP on the password branch. The README's four-step manual check remains the only thing that would establish it, and it remains undone.
• That the RP ID is right for the origin. ${BASE_DOMAIN} is a legal registrable-domain suffix of keycloak.${BASE_DOMAIN} on paper, but a wrong RP ID fails in the browser, before Keycloak is involved, with no server-side log line. The reconciler will happily converge the realm on a wrong RP ID; it checks that git and the realm agree, never that git is correct.
• The first real run will change how everyone logs in. Once this merges and Flux applies it, the next tick builds a flow and — if it verifies — flips the realm's browser flow. The UX change is deliberate and documented (username screen, then password), but it is the first time it will have happened, unattended, on a schedule. The rollback is one write and the old flow survives; that is the mitigation, and it has not been rehearsed either.
• CI's own result. My gate run is local. gitea-settings cannot run locally, and ci/approval-policy.sh reports green N/A offline while this branch touches three sensitive paths, so the PR is guaranteed to need a non-author approving review and I have not seen CI report. I did not and will not post /approve-override on my own PR.
• I re-measured branch protection this session (protected: true, user_can_push: false, enable_status_check: true, context gate / gate (pull_request), required_approvals: 0), but the branches endpoint does not expose block_admin_merge_override, so that row of CLAUDE.md §2 is not re-confirmed by me.