Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Merge pull request 'fix(keycloak-passkey): passkey reconciler with review fixes — bind only on git's say-so (default false), protect real sub-flows, body-id screening, failure alert' (#82) from feat/keycloak-passkey-reconcile into main
gate / gate (push) Successful in 2m50s
Details
gate / conformance (push) Has been skipped
Details

Browse Source 
Reviewed-on: #82
Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #82.
This commit is contained in:
jakman85 committed 2026-10-30 05:59:11 +00:00
2 parents 523cd9118f 262c6e4a97
commit dfc432c6ea
15 files changed
+6148 -15
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

ci/render-keycloak-passkey-policy.py
+490
View File
@@ -0,0 +1,490 @@
#!/usr/bin/env python3
# =============================================================================
# ci/render-keycloak-passkey-policy.py
#
# Render platform/keycloak/passkey-desired-state.yaml FROM
# platform/keycloak/keycloak-realm-import.yaml.
#
# WHY THIS EXISTS
# ---------------
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
# after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
# Done=True Started=False HasErrors=False
#
# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
# is the level-triggered loop that closes that; this renderer supplies the DESIRED
# STATE it converges towards.
#
# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
# flow in a browser; make it the login path" -- not a fact derivable from the realm
# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
# file: --write preserves it, --check accepts any value of it that is exactly the
# string "true" or "false" and rejects everything else (an unquoted YAML true is a
# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
# Kustomization -- so it is refused HERE, at merge time). An absent file or key
# renders as "false". Going live is therefore a one-line PR flipping that line.
#
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
# automountServiceAccountToken: false, exactly like its client-secret sibling. It
# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
# must be MOUNTED. Hence a ConfigMap.
#
# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
# the user-verification requirement and the whole flow tree — free to disagree with
# the realm import. "Two declarations that can disagree" is the same shape as the
# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
# import and the generation is GATED:
# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
# with --check on every merge, and the merge fails if the committed file is not what
# this renderer produces. Same render==golden idiom as
# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
#
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
# rendered by the SAME Flux Kustomization (./platform-config, whose
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
# pass with the same value and cannot disagree about what the base domain is. This
# renderer never parses, splits or reconstructs a hostname and never reads
# cluster-vars — there is no second place the domain is decided.
#
# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
# -----------------------------------------------------
# The enrolment required action, `webauthn-register-passwordless`. It is
# deliberately NOT declared in the realm import, and the import's own header
# explains at length why declaring it there would be actively DANGEROUS: Keycloak
# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
# so declaring one action imports exactly that action plus delete_account and skips
# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
# lock-out of every platform-admin, because auth-otp-form in the browser flow still
# demands a TOTP they could then never enrol.
#
# So the required action cannot be copied from the import, because it is correctly
# absent from it. It is instead DERIVED, as a consequence of what the import DOES
# declare:
#
# IF the declared browser flow contains a `webauthn-authenticator-passwordless`
# execution, THEN passkey logins are offered, THEREFORE the enrolment action must
# be registered and enabled — and NOT a defaultAction, because the import's whole
# stated posture is "additive and optional" and a defaultAction forces enrolment
# on every user at next login.
#
# This is a derivation, not a second declaration: there is no value here that a
# human could edit into disagreement with the import. If the import stops declaring
# the passkey execution, this renderer stops emitting the required action (and
# fails, because then the whole desired state is empty of passkey intent).
#
# WHAT IS DELIBERATELY NOT EXPORTED
# * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
# `webauthn-authenticator` two-factor authenticator; the passwordless
# authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
# into the two-factor family is a silent no-op. Exporting it would hand the
# reconciler a write surface with no purpose.
# * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
# SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
# and users/groups/roles are nobody's — prod's real user and the media-users /
# platform-admins / viewers groups must survive every run of everything.
# * execution `priority`. It is consumed here to ORDER the exported executions
# (Keycloak returns them in priority order), but the numbers themselves are not
# exported: the reconciler compares the resulting SEQUENCE, which is what
# actually determines behaviour. Exporting the integers would make a
# priority renumbering that changes no ordering read as drift.
#
# USAGE
# python3 ci/render-keycloak-passkey-policy.py # print the rendered file
# python3 ci/render-keycloak-passkey-policy.py --write # write it (regenerate)
# python3 ci/render-keycloak-passkey-policy.py --check # exit 1 if it has drifted
# =============================================================================
from __future__ import annotations
import argparse
import difflib
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(__file__).resolve().parent.parent
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
DATA_KEY = "passkey-desired-state.json"
CONFIGMAP_NAME = "keycloak-passkey-desired-state"
NAMESPACE = "keycloak"
# The realm keys this renderer exports, and therefore the ONLY realm keys the
# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
# tuples are identical, so they cannot drift apart silently.
POLICY_STR_FIELDS = (
"webAuthnPolicyPasswordlessRpId",
"webAuthnPolicyPasswordlessRpEntityName",
"webAuthnPolicyPasswordlessRequireResidentKey",
"webAuthnPolicyPasswordlessUserVerificationRequirement",
"webAuthnPolicyPasswordlessAuthenticatorAttachment",
"webAuthnPolicyPasswordlessAttestationConveyancePreference",
)
POLICY_LIST_FIELDS = (
"webAuthnPolicyPasswordlessSignatureAlgorithms",
"webAuthnPolicyPasswordlessAcceptableAaguids",
)
POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+ POLICY_BOOL_FIELDS)
# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
# (DefaultExportImportManager.updateRealm: the policy is applied only when
# RpEntityName is non-null and non-empty). A desired state without it would produce
# an accepted PUT that changes nothing, so refuse to render one.
RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
# The execution whose presence is what makes passkey enrolment necessary.
PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
ENROL_ACTION = "webauthn-register-passwordless"
BIND_KEY = "bind-browser-flow"
BIND_VALUES = ("true", "false")
def committed_bind(root: Path) -> str:
"""The operator's bind-browser-flow value from the COMMITTED file, or "false".
This is the only input read from the output file, and it is validated, not trusted:
anything other than the exact strings "true" / "false" is a hard failure."""
target = root / OUT_FILE
if not target.is_file():
return "false"
docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
data = cm.get("data") or {}
if BIND_KEY not in data:
return "false"
v = data[BIND_KEY]
if not isinstance(v, str) or v not in BIND_VALUES:
raise SystemExit(
"FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
"\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
"at apply time and which would fail the whole platform-config Kustomization"
% (OUT_FILE, BIND_KEY, v))
return v
def render(root: Path) -> str:
"""Return the full text of passkey-desired-state.yaml as it should be committed."""
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
if imp is None:
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
realm = (imp.get("spec") or {}).get("realm") or {}
realm_name = realm.get("realm")
if not isinstance(realm_name, str) or not realm_name:
raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
# ---- the passwordless policy, copied VERBATIM ----------------------------------
policy: dict[str, object] = {}
for f in POLICY_STR_FIELDS:
if f not in realm:
continue # not declared => NOT MANAGED
v = realm[f]
# A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
# a YAML quoting mistake that must not be silently stringified.
if not isinstance(v, str):
raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
"quote it in the realm import" % (IMPORT_FILE, f, v))
policy[f] = v
for f in POLICY_LIST_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
raise SystemExit("FATAL: %s realm field %s is not a list of strings"
% (IMPORT_FILE, f))
policy[f] = list(v) # verbatim, order as authored
for f in POLICY_INT_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, int) or isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not an integer"
% (IMPORT_FILE, f))
policy[f] = v
for f in POLICY_BOOL_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not a boolean"
% (IMPORT_FILE, f))
policy[f] = v
if not policy:
# Fail closed. An empty policy would make the reconciler believe every field is
# unmanaged and silently stop reconciling — the realm import's own fail-open.
raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
"refusing to render an empty desired state" % IMPORT_FILE)
if not policy.get(RP_ENTITY_NAME):
raise SystemExit(
"FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is set, so every realm PUT would "
"be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
# ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
browser_flow = realm.get("browserFlow")
if not isinstance(browser_flow, str) or not browser_flow:
raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
"be perfect and still not be the one anyone logs in through"
% IMPORT_FILE)
raw_flows = realm.get("authenticationFlows") or []
if not isinstance(raw_flows, list) or not raw_flows:
raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
flows: dict[str, object] = {}
for fl in raw_flows:
alias = fl.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authentication flow with no alias"
% IMPORT_FILE)
if alias in flows:
raise SystemExit("FATAL: %s declares the flow %r twice"
% (IMPORT_FILE, alias))
execs = fl.get("authenticationExecutions") or []
if not isinstance(execs, list) or not execs:
raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
% (IMPORT_FILE, alias))
ordered = []
for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
is_flow = bool(ex.get("authenticatorFlow"))
name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
if not isinstance(name, str) or not name:
raise SystemExit(
"FATAL: %s flow %r has an execution with neither an authenticator "
"nor a flowAlias" % (IMPORT_FILE, alias))
requirement = ex.get("requirement")
if not isinstance(requirement, str) or not requirement:
raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
% (IMPORT_FILE, alias, name))
entry: dict[str, object] = {
"kind": "flow" if is_flow else "authenticator",
"name": name,
"requirement": requirement,
}
cfg = ex.get("authenticatorConfig")
if cfg is not None:
if not isinstance(cfg, str) or not cfg:
raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
"authenticatorConfig" % (IMPORT_FILE, alias, name))
entry["configAlias"] = cfg
ordered.append(entry)
flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
if browser_flow not in flows:
raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
% (IMPORT_FILE, browser_flow))
# Every referenced subflow must be declared, or the reconciler's flattening walks
# into a hole and the audit silently covers less than it claims.
for alias, spec in flows.items():
for ex in spec["executions"]:
if ex["kind"] == "flow" and ex["name"] not in flows:
raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
"declared" % (IMPORT_FILE, alias, ex["name"]))
# ---- authenticator configs, copied verbatim ------------------------------------
configs: dict[str, dict] = {}
for cfg in realm.get("authenticatorConfig") or []:
alias = cfg.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
% IMPORT_FILE)
body = cfg.get("config") or {}
if not isinstance(body, dict) or not body:
raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
% (IMPORT_FILE, alias))
# Values are stringified because Keycloak stores authenticator config values as
# strings. `negate: "false"` in the import and a stored "false" are the same
# declaration, and a YAML-unquoted `false` must not become a Python bool that
# then never matches.
configs[alias] = {k: str(v) for k, v in sorted(body.items())}
# Every configAlias a flow references must be defined, else the reconciler reports
# "git declares config X but defines no key/value for it" on every run forever.
for alias, spec in flows.items():
for ex in spec["executions"]:
ca = ex.get("configAlias")
if ca and ca not in configs:
raise SystemExit(
"FATAL: %s flow %r execution %r references authenticatorConfig %r, "
"which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
# ---- the enrolment required action, DERIVED (see the header) -------------------
offers_passkey = any(
ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
for spec in flows.values() for ex in spec["executions"])
if not offers_passkey:
raise SystemExit(
"FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
"in any flow. The policy decides what a passkey must BE; without that "
"execution nothing ever OFFERS one, so the policy is unreachable "
"configuration and this desired state would be meaningless"
% (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
want = {
"realm": realm_name,
"webAuthnPolicyPasswordless": policy,
# enabled: passkey enrolment must be possible, since the flow offers passkeys.
# defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
# enrol a passkey at next login — the platform-wide lock-out the realm import's
# "additive and optional" posture exists to avoid. Derived, not copied; see the
# header for why the import cannot declare this itself.
"requiredAction": {"alias": ENROL_ACTION, "enabled": True,
"defaultAction": False},
"browserFlow": browser_flow,
"flows": flows,
"authenticatorConfig": configs,
}
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+ " %s: |\n" % DATA_KEY + indented + "\n")
HEADER = """---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
# the WebAuthn passwordless policy, the enrolment required action, and the declared
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
# it and (a) converges the policy and the required action onto it, (b) builds the
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
# writing the flow users are logging in through, and (c) BINDS that verified flow as
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
#
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
# operator decision, carried through by the renderer from this committed file (see
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a realm-level field or a flow edited in git never reaches the
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
# the level-triggered loop that closes that; this file is what it reconciles
# TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
# user-verification requirement and the flow tree would be a second source of truth,
# free to disagree with the realm import — the same "two declarations that can
# disagree" shape as the bug being closed. It is rendered from the import and the
# render is gated, so the two cannot drift.
#
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
# substitution token exactly as the realm import writes it. Both files are rendered
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
# same value in both and the domain is decided in exactly one place.
#
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
# nine more (a hard lock-out of every platform-admin). It is derived instead from
# the presence of a webauthn-authenticator-passwordless execution in the flow:
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
# false, so enrolment stays opt-in.
#
# A field NOT declared on the realm in the import is simply ABSENT here, and the
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: %s
namespace: %s
labels:
app.kubernetes.io/name: %s
app.kubernetes.io/part-of: keycloak
data:
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
BIND_BLOCK = """ # ---------------------------------------------------------------------------
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
# "false", QUOTED.
#
# "false" the reconciler builds and verifies the candidate browser flow and
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
# Whatever the realm is bound to -- including a rollback -- is left alone.
# "true" it binds the verified candidate in one atomic PUT, and keeps the
# realm bound to it: a binding changed by hand is changed back.
#
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
# "Passkeys"), then a PR setting this to "true".
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
# one write setting browserFlow back to the previous alias from the Job log.
# ---------------------------------------------------------------------------
%s: "%s"
"""
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group()
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
g.add_argument("--check", action="store_true",
help="exit 1 if the committed file has drifted from the source")
args = ap.parse_args()
want = render(ROOT)
target = ROOT / OUT_FILE
if args.write:
target.write_text(want, encoding="utf-8")
print("wrote %s" % OUT_FILE)
return 0
if args.check:
if not target.is_file():
print("CHECK FAIL: %s does not exist. Run: python3 "
"ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
file=sys.stderr)
return 1
have = target.read_text(encoding="utf-8")
if have == want:
print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
return 0
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
"realm on a passkey policy the realm import does not declare, and build "
"the login flow from a tree nobody authored.\n"
"Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
fromfile="committed " + OUT_FILE,
tofile="rendered from " + IMPORT_FILE):
sys.stderr.write(line)
return 1
sys.stdout.write(want)
return 0
if __name__ == "__main__":
raise SystemExit(main())
conformance/tests/l0-static/keycloak-passkey-reconciler/chainsaw-test.yaml
+108
View File
@@ -0,0 +1,108 @@
# =============================================================================
# L0 STATIC — the passkey configuration is derived from one declaration, continuously
# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
# Measured on prod 2026-10-18, hours after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
# them changed nothing about how anyone logs in, and nothing anywhere said so.
# Declarative config that silently does not apply is not declarative config.
#
# GIVEN the repository as authored,
# THEN the reconciler's desired state is GENERATED from the realm import (so the RP ID —
# which is baked into every passkey at enrolment and can NEVER be re-keyed — is
# declared exactly once); both files are rendered by the SAME Flux Kustomization, so
# their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
# policy keys and the reconciler's writable keys are the SAME tuple; the realm
# import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
# delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
# enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
# no ServiceAccount token, mounts only what it needs, passes no mode flag, and
# contains no dollar sign; the endpoint allow-list contains NO delete, NO
# single-execution or config-edit write, and NO required-action write for any alias
# other than the one named literally; the protected-subtree guard exists, is wired
# into the single request choke point and is fail-closed; the SIBLING client-secret
# reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
# (extracted from the shipped ConfigMap and executed) proves compare-then-write,
# idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
# its sub-flows unwritable, the git bind switch and a rollback that sticks, the
# conditional-OTP regression and fail-closed — with mutations.py proving that
# selftest can FAIL.
#
# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
# than only argued: converging a five-flow, ten-execution restructure needs
# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
# calls with no transaction. Doing that to the BOUND flow means a run that dies between
# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
# which is verified by readback before anything is bound. Binding it is then ONE atomic
# PUT of browserFlow, made only when passkey-desired-state.yaml sets
# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
# a schedule tick, and with the switch off an operator's rollback is never undone.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
# Test wraps the SAME script.
#
# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
# password form (needs a real browser against a real realm); that the five flow-building
# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: keycloak-passkey-reconciler
labels:
conformance.home.k3s/R-IDP-03: "true"
conformance.home.k3s/R-IDP-04: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
# check declares it: this invariant spans platform/ (the realm import, the reconciler,
# the generated desired state), platform-config/ (which Flux Kustomization renders
# them, and therefore which substitution pass they share), ci/ (the renderer) and
# clusters/ (that both are reachable from both entrypoints). A change that breaks it
# need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
# then filter this check out of exactly the change that introduces the bug — which is
# how 12 unusable prod OIDC clients sat behind an entirely green cluster on
# 2026-09-06.
conformance.home.k3s/scope: "whole-repo"
# Not safety-critical in the harness's sense (it mutates nothing and needs no
# cluster), but it guards a component that holds WRITE access to the identity
# provider's realm settings.
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The realm's passkey configuration — policy, enrolment action AND browser login
flow — is declared once in the realm import, derived from it by a gated renderer,
and continuously reconciled; the flow converges blue/green, so a one-shot import
that never re-runs can neither leave the configuration inert nor let a reconciler
half-rewrite the login path of the whole platform.
spec:
steps:
- name: audit-passkey-declaration-renderer-and-reconciler
description: >-
Assert the desired state is derived from the realm import and shares its
substitution pass, that the reconciler cannot write the bound flow or delete
anything and binds only on git's bind-browser-flow, that the sibling reconciler
is unwidened, and run the reconciler's own selftest plus the mutation suite that
proves it can fail.
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh
Executable
+754
View File
@@ -0,0 +1,754 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
# — spec docs/15-identity-and-access.md)
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
# — spec docs/15-identity-and-access.md)
#
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
# ---------------------------------------------------------------
# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
# hours after that declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
# re-runs it, so every realm-level field and every flow declared after the FIRST
# import is inert — in git, applied by Flux, reported Ready, reaching the realm
# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
# the same fix: a level-triggered reconciler. This check is the static half.
#
# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
#
# GIVEN the repository as authored,
# THEN (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
# (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
# import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
# renderer with --check, so a hand edit that makes the two disagree is
# unmergeable. The RP ID is the sharpest case: it is baked into every passkey
# at enrolment and can NEVER be re-keyed, so two declarations that can skew
# about the domain is an unrecoverable-credential bug, not a cosmetic one.
# (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
# postBuild.substituteFrom is per-Kustomization, so a split would let the
# ${BASE_DOMAIN} token in the realm import and the one in the desired state
# resolve to two different values — and an RP ID mismatch invalidates every
# enrolled passkey with no error the user can act on.
# (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
# declares no realm key outside the reconciler's write surface.
# (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
# reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
# could emit a key the reconciler refuses (fail-closed forever) or the
# reconciler could accept a key the renderer never emits (dead surface).
# (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
# entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
# ServiceAccount token, and contains no dollar sign (it ships inside a
# SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
# rewritten before any container runs).
# (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
# SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
# alias, verify, flip browserFlow in one atomic PUT), so flow-create and
# execution-add endpoints ARE on the allow-list and their presence is not the
# hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
# is the rollback target); the execution- and config-id-scoped write endpoints
# stay off the list because they carry no flow alias and so cannot be screened;
# no required-action write reaches any alias but the one named literally; and
# the guard the allow-list CANNOT express exists in the source and is wired
# into the single request choke point, FAIL-CLOSED until the live binding has
# been read. That last one is what the whole design rests on: a reconciler that
# half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
# this repo has had one (2026-09-06).
# (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
# bind-browser-flow as exactly "true" or "false", the script reads it from the
# SAME mount, and the CronJob passes no flag that could widen it. With "false"
# (the default) the reconciler builds and verifies but never PUTs browserFlow,
# so an operator's rollback sticks; the selftest proves that end to end.
# (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
# allow-list is NOT widened by this work. Its --selftest asserts it refuses
# /users, /groups, /roles, /authentication and the whole-realm endpoint, and
# its guarantee that prod's real user and the media-users / platform-admins /
# viewers groups survive every run must stay LITERALLY true.
# (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
# script from the ConfigMap and executes its `--selftest`, which drives the
# real code paths against an in-process stub Keycloak that MODELS the three
# Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
# passwordless policy, the all-or-nothing policy rebuild that defaults every
# omitted key, and the required-action representation replace.
# (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
# one at a time and requires --selftest to CATCH every one. A passing suite
# proves nothing if it cannot fail, and "reported success while the realm was
# broken" is the exact failure this component exists to end.
# AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
# `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
# via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
# ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
# An unreferenced test is precisely the "looks applied and isn't" failure this
# repo keeps hitting.
#
# NOT proven here (named honestly, in the style of the sibling L0 checks):
# - that the LIVE realm's policy, enrolment action or browser flow match git. That is
# cluster state. The reconciler's own `--audit` mode is the read-only probe for it
# and is what an L2 test should run; no such L2 test is added by this change, so the
# live realm is UNVERIFIED by CI.
# - that the declared browser flow actually WORKS as a login path. Nothing offline can
# prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
# password form, or that a platform-admin is still asked for TOTP. Those need a real
# browser against a real realm.
# - that the reconciler's first real run against prod will succeed. The flow is built
# through five admin-API endpoints whose request shapes were read from the 26.0.6
# source but never exercised against a live Keycloak.
# - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
# offline; the plaintext is not, and must not be.
# - that the CronJob's image can be pulled or its schedule fires.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
RENDERER="ci/render-keycloak-passkey-policy.py"
SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
# Honest SKIP (never green) if the thing under test is absent entirely.
for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
if [ ! -f "${ROOT}/${f}" ]; then
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
exit 3
fi
done
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit of the Keycloak passkey declaration, the renderer that derives the
reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
is one definition of "what a Flux path renders" in the harness."""
from __future__ import annotations
import ast
import json
import re
import subprocess
import sys
import tempfile
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
# test directory is named relative to the repo root the wrapper passes in.
DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
/ "keycloak-passkey-reconciler")
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
import status as st # noqa: E402
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
RENDERER = "ci/render-keycloak-passkey-policy.py"
SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
STATE_CM_NAME = "keycloak-passkey-desired-state"
STATE_DATA_KEY = "passkey-desired-state.json"
BIND_KEY = "bind-browser-flow"
SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
ENROL_ACTION = "webauthn-register-passwordless"
DOLLAR = chr(36)
failures: list[str] = []
def check(cond, msg):
if not cond:
failures.append(msg)
def load(rel):
return list(yaml.safe_load_all((ROOT / rel).read_text()))
# ================================================== (1) the desired state is DERIVED
proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
capture_output=True, text=True, timeout=120)
check(proc.returncode == 0,
f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
f".py --check failed). The reconciler would converge the realm on a passkey policy "
f"the realm import does not declare, and audit the login flow against a tree "
f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
f"domain is an unrecoverable-credential bug.\n "
+ "\n ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
# The realm import must actually declare the things the renderer derives from. If it
# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
# renderer would otherwise be the only signal.
imp_docs = [d for d in load(IMPORT_FILE) if d]
imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
realm = ((imp or {}).get("spec") or {}).get("realm") or {}
policy_keys_in_import = sorted(k for k in realm
if k.startswith("webAuthnPolicyPasswordless"))
check(policy_keys_in_import,
f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
f"passkey policy for the reconciler to converge")
check(realm.get("browserFlow"),
f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
f"BOUND changes nothing about how anyone logs in")
# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
# representation as REPLACE, not merge, so declaring ONE action imports exactly that
# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
# NOT ADD ONE" is advice; this is the gate.
check("requiredActions" not in realm,
f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
f"as REPLACE, not merge: declaring any action imports exactly those plus "
f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
f"realm's browser flow still demands a TOTP they could no longer enrol. The "
f"enrolment action is already registered by default (priority 80, enabled, "
f"defaultAction false) and the passkey reconciler converges it through the "
f"per-alias admin API, which has no such hazard")
# ============================== (2)(3) the two files share one substitution pass
state_docs = [d for d in load(STATE_FILE) if d]
state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
desired = {}
if state_cm is not None:
check(state_cm["metadata"]["name"] == STATE_CM_NAME,
f"{STATE_FILE}: the ConfigMap is named "
f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
data = state_cm.get("data") or {}
check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
f"PASSKEY_BIND_FILE defaults would not resolve")
# THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
# at apply time -- failing the whole platform-config Kustomization, realm import
# included. The renderer refuses it too; asserted here so the reason is on record.
check(data.get(BIND_KEY) in ("true", "false"),
f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
f"string \"true\" or \"false\"")
raw = data.get(STATE_DATA_KEY, "")
try:
desired = json.loads(raw)
except ValueError as e:
check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
f"({e}) — the reconciler would fail closed on every run")
# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
import_owners = set()
state_owners = set()
recon_owners = set()
for env in ("dev", "prod"):
entry = f"clusters/{env}"
if not (ROOT / entry).is_dir():
continue
applied, _ = st.reconciled_files(ROOT, entry)
check(RECON_FILE in applied,
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
f"healer would exist on one cluster only")
check(STATE_FILE in applied,
f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
f"would fail closed on that cluster forever")
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
# listed as resources of the same kustomization.yaml.
for kpath in ROOT.glob("*/kustomization.yaml"):
try:
k = yaml.safe_load(kpath.read_text()) or {}
except Exception:
continue
res = [str(r) for r in (k.get("resources") or [])]
base = kpath.parent.name
if any(r.endswith("keycloak-realm-import.yaml") for r in res):
import_owners.add(base)
if any(r.endswith("passkey-desired-state.yaml") for r in res):
state_owners.add(base)
if any(r.endswith("passkey-reconciler.yaml") for r in res):
recon_owners.add(base)
check(import_owners and import_owners == state_owners,
f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
check(recon_owners == state_owners,
f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
f"{sorted(state_owners)} — the reconciler could be applied without its input")
# (3) no substitution token the realm definition does not also use.
import_text = (ROOT / IMPORT_FILE).read_text()
TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
import_tokens = set(TOKEN.findall(import_text))
# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
# policy uses, so a subset check is the right assertion rather than an exact match.
check(state_tokens <= import_tokens,
f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
f"silently — which for an RP ID means every passkey ceremony fails")
# ======================================= (4) the two write surfaces are the SAME tuple
recon_docs = [d for d in load(RECON_FILE) if d]
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
script = ""
if cm is not None:
data = cm.get("data") or {}
check(len(data) == 1,
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
f"(found {sorted(data)}) — a second copy would be untested by this gate")
check(SCRIPT_KEY in data,
f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
script = next(iter(data.values()), "")
check(len(script.splitlines()) > 200,
f"{RECON_FILE}: the extracted script is implausibly short "
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
f"selftest below would prove nothing")
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
# postBuild.substituteFrom, so envsubst rewrites it before any container runs.
check(DOLLAR not in script,
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
f"never needs one; chr(36) spells it where a literal is genuinely required")
def tuple_literal(src, name):
"""The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
than importing it keeps this check offline and side-effect free, and means the thing
asserted is the text that ships."""
try:
tree = ast.parse(src)
except SyntaxError:
return None
for node in tree.body:
if isinstance(node, ast.Assign):
for t in node.targets:
if isinstance(t, ast.Name) and t.id == name:
try:
return ast.literal_eval(node.value)
except ValueError:
return None
return None
renderer_src = (ROOT / RENDERER).read_text()
# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
# their parts from the same vocabulary; a part present in one and not the other simply
# contributes nothing, which is why the comparison below is of the resolved SETS.
def resolve_policy(src):
parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
"POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
"POLICY_BOOL_FIELDS")
out = []
for p in parts:
v = tuple_literal(src, p)
if v:
out.extend(v)
return sorted(set(out))
recon_keys = resolve_policy(script)
rend_keys = resolve_policy(renderer_src)
check(recon_keys and recon_keys == rend_keys,
f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
f"every run fails closed forever; if the reconciler accepts a key the renderer "
f"never emits, that is dead write surface on the identity provider")
# Every exported key must actually be a passwordless key. The two-factor
# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
# with no effect.
check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
f"the reconciler's writable keys include a non-passwordless field: "
f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
f"writing it is a silent no-op")
# And the desired state must declare only those.
if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
check(not extra,
f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
f"surface — it would fail closed on every run")
# The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
check(desired["webAuthnPolicyPasswordless"].get(
"webAuthnPolicyPasswordlessRpEntityName"),
f"{STATE_FILE} carries no non-empty "
f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
f"passwordless policy only when that field is set "
f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
f"accepted with 204 and change nothing")
# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
# import's stated "additive and optional" decision. It is one boolean, so it is gated.
ra = desired.get("requiredAction") or {}
check(ra.get("alias") == ENROL_ACTION,
f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
f"any other value is unreachable and the pass would fail closed")
check(ra.get("enabled") is True,
f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
f"impossible while the login flow offers passkeys")
check(ra.get("defaultAction") is False,
f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
f"must be false. True forces EVERY user to enrol a passkey at next login — this "
f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
# ========================================= (5) the reconciler manifest is safe + wired
if cj is not None:
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
ctr = pod["containers"][0]
vols = {v["name"]: v for v in pod.get("volumes", [])}
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
env = {e["name"]: e for e in ctr.get("env", [])}
# NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
# narrower than reading the KeycloakRealmImport CR through the API: no Role, no
# RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
check(pod.get("automountServiceAccountToken") is False,
f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
f"ServiceAccount token is pure blast radius (R-SEC-02)")
check("serviceAccountName" not in pod,
f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
check(STATE_CM_NAME in names,
f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
check(cm["metadata"]["name"] in names,
f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
# NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
# secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
check(not [v for v in vols.values() if "secret" in v],
f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
f"credential is referenced by secretKeyRef only; it must never be projected "
f"as a file here")
# The mount paths must be the ones the code DEFAULTS to, or the container starts and
# fails closed forever while every manifest looks correct.
state_vol = next((n for n, v in cm_vols.items()
if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
if state_vol:
mp = (mounts.get(state_vol) or {}).get("mountPath")
check(mp == "/etc/keycloak-passkey",
f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
f"script's PASSKEY_STATE_FILE default is "
f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
f"the mount path — they are a pair and must move together")
check((mounts.get(state_vol) or {}).get("readOnly") is True,
f"{RECON_FILE}: the desired-state mount must be readOnly")
check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
f"(safe, but git's \"true\" would then silently never take effect)")
check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
f"keys may not be projected")
# The bind decision belongs to git. A container arg could only NARROW it today
# (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
# behaviour must be readable from passkey-desired-state.yaml alone.
cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
check(not [x for x in cmd if x.startswith("--")],
f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
# Retries burn the bounded candidate-alias budget on a deterministic failure.
check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
f"within a tick, and each retried build consumes another candidate alias")
# The admin credential, referenced BY NAME. Never decrypted, never a literal.
for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
e = env.get(key) or {}
ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
check(ref.get("name") == "keycloak-admin-credentials",
f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
f"Secret by secretKeyRef, got {e!r}")
check("value" not in e,
f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
f"manifest is a plaintext secret in git")
check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
f"{RECON_FILE}: KEYCLOAK_REALM is "
f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
f"wrong realm, or none")
check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
f"{RECON_FILE}: readOnlyRootFilesystem must be true")
check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
f"{RECON_FILE}: allowPrivilegeEscalation must be false")
check(pod["securityContext"].get("runAsNonRoot") is True,
f"{RECON_FILE}: runAsNonRoot must be true")
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
f"would double the write attempts against the IdP during an outage recovery")
check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
#
# This component converges the browser flow BLUE/GREEN: it builds the target structure
# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
# first design of this check the mere presence of a flow write is no longer the thing to
# forbid. What must hold instead is narrower and stronger:
#
# (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
# (b) the single-execution and config-edit endpoints stay off the list — requirements
# are set through the flow-scoped endpoint, which the guard in (d) screens;
# (c) no required-action write may reach any alias but the one named LITERALLY;
# (d) and the guard the allow-list CANNOT express must exist in the source: a
# method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
# flow's subtree, FAIL-CLOSED until that binding has been read.
#
# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
# change (2026-09-06).
allow = tuple_literal(script, "_ALLOWED") or ()
check(bool(allow),
f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
f"script. This check's central assertions about the write surface cannot be made, "
f"so it must FAIL rather than pass silently")
if allow:
check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
f"allow-list cannot distinguish reading a flow from writing one, which this "
f"component's whole safety argument rests on")
writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
# (a) nothing is ever deleted.
check(not [(m, p) for m, p in allow if m == "DELETE"],
f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
f"does requires deleting anything: a superseded flow is left unbound as the "
f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
# (b) the endpoints that would let it edit live objects in place.
banned = [(m, p) for m, p in writes
if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
r"\[\^/\]\+", p)
or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
p)
or p.endswith("/authentication/register-required-action")]
check(not banned,
f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
f"and config-edit endpoints must stay off it: requirements are set through the "
f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
f"therefore screenable against the bound subtree. An execution- or "
f"config-id-scoped write is not, so it would be an unscreened hole")
# (c) no required-action write may reach any other alias.
ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
f"permanent lock-out of every platform-admin, because auth-otp-form in the "
f"browser flow still demands a TOTP that could no longer be enrolled")
check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
f"must be named LITERALLY so no other action is writable by construction")
# Nothing outside the realm + authentication surface at all.
stray = [(m, p) for m, p in allow
if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
check(not stray,
f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
f"be unreachable (prod's real user and the media-users / platform-admins / "
f"viewers groups must survive every run); clients are the SIBLING "
f"reconciler's surface and the two must not overlap")
# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
# Asserted against the SOURCE, because this is the property the blue/green design rests
# on and a comment claiming it is worth nothing. Three things must all be true.
check(re.search(r"^def _screen_write\(", script, re.M) is not None,
f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
f"one thing the allow-list cannot express, because its write entries are "
f"alias-parameterised. Without it this component can mutate the live login path")
# It must be CALLED from the single choke point, not merely defined.
check(re.search(r"_screen_write\(method, path", script) is not None,
f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
f"request's method and path. A guard that is not on the request path is decoration")
# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
# the top of every run must put them back there.
reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
check(reset_fn is not None
and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
f"must start each run UNDETERMINED, because while they are, every flow write is "
f"refused — that is the fail-closed half of the guard, and it is what stops a "
f"build running before the live binding has been read")
check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
f"protected sets are not reset per run and a previous run's binding could be "
f"trusted for this one")
# The create-flow endpoint carries NO alias in its path, so it must be screened on its
# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
# future edit cannot silently undo it.)
check("_FLOW_CREATE_PATH" in script,
f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
f"That endpoint carries no alias in its path — the alias is in the request BODY — "
f"so without a body-based screen it bypasses the protected-subtree guard "
f"altogether")
# And the one function allowed to change the live binding must be distinct from the
# policy write, which must refuse to.
# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
# the top-level flow alone while every selftest passed against a stub that returned it.
check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*? name = ex\.get\("displayName"\)',
script, re.M) is not None,
f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
f"None and the protected subtree silently shrinks to the top-level flow")
# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
f"and ignores the path alias, so a path-only screen lets it write the bound flow")
check("def declared_bind(" in script and "BIND_FILE" in script,
f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
check("def bind_browser_flow(" in script,
f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
f"one named, auditable operation, not an incidental field in some other write")
check(re.search(r"never rebinds the browser flow", script) is not None,
f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
f"can rebind the login path by accident — to an UNVERIFIED flow")
# ======================================== (7) the SIBLING reconciler is NOT widened
if (ROOT / SIBLING_FILE).is_file():
sib_docs = [d for d in load(SIBLING_FILE) if d]
sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
check(bool(sib_allow),
f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
f"prove the sibling was not widened")
if sib_allow:
# The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
# whole-realm endpoint and the /authentication tree; assert the same here so a
# future edit that "just adds the realm endpoint there instead" is caught.
bad = [p for p in sib_allow
if isinstance(p, str)
and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
or "authentication" in p or "/users" in p or "/groups" in p
or "/roles" in p)]
check(not bad,
f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
f"stated guarantee — prod's real user and the media-users / "
f"platform-admins / viewers groups survive every run — is expressed as "
f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
f"The passkey work is a SIBLING program with its own narrow list precisely "
f"so that guarantee stays literally true. Revert the widening")
# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
if script:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "reconcile.py"
p.write_text(script)
proc = subprocess.run([sys.executable, str(p), "--selftest"],
capture_output=True, text=True, timeout=300)
tail = (proc.stdout + proc.stderr).strip().splitlines()
check(proc.returncode == 0,
"the passkey reconciler's --selftest FAILED (compare-then-write / "
"idempotence / strict read-modify-write / the RpEntityName gate / "
"flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
" " + "\n ".join(tail[-14:]))
if proc.returncode == 0:
print(" passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
# ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
# A passing suite proves nothing unless it can be shown to FAIL. mutations.py
# injects plausible regressions one at a time -- a partial policy PUT, a
# rebind of browserFlow, a widened allow-list that makes the flow writable, a
# fail-open on an unreadable input -- and requires --selftest to catch every
# one. This matters more than usual here: the whole incident class is a
# component that REPORTED SUCCESS while the realm was broken, and a green test
# that cannot fail is the same lie one level up.
mut = DIR_PATH / "mutations.py"
if mut.is_file():
mproc = subprocess.run([sys.executable, str(mut), str(p)],
capture_output=True, text=True, timeout=900)
mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
check(mproc.returncode == 0,
"the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
"one injected regression survived it, so the suite does not prove "
"what it claims:\n " + "\n ".join(mtail[-16:]))
if mproc.returncode == 0:
print(" passkey reconciler mutations: "
+ (mtail[-1] if mtail else "(no output)"))
else:
check(False,
"conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
"is missing — nothing proves the --selftest can fail, and an "
"unfalsifiable proof is the failure mode this component exists to end")
# ======================================================= (9) this check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
is not None,
"keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
"only when a change happens to attribute one of the requirements it is labelled "
"with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
"change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
"prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
"stage is a purely requirement-scoped run again and the scope annotation is inert "
"decoration — this check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
"ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
"import or the reconciler would attribute NO requirement and the keycloak tests "
"would be filtered out of the merge gate's touched subset")
# =============================================================================
if failures:
print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
for f in failures:
print(" FAIL %s" % f, file=sys.stderr)
raise SystemExit(1)
print("keycloak-passkey-reconciler: all assertions passed "
"(desired state derived from the realm import, one substitution pass, write "
"surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
"deletable, sibling allow-list unwidened, selftest green and falsifiable)")
PY
conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py
+456
View File
@@ -0,0 +1,456 @@
#!/usr/bin/env python3
# =============================================================================
# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
#
# WHY THIS EXISTS
# ---------------
# A selftest that passes proves nothing on its own: a selftest asserting `True ==
# True` also passes. This reconciler exists because a one-shot realm import
# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
# with the whole passkey block inert, measured on prod 2026-10-18), and the same
# trap applies one level up -- a green test suite that cannot fail is the same lie
# in a different place.
#
# It matters more here than for the sibling client reconciler, because this program
# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
# untested safety property on that surface is not a documentation gap, it is an
# outage waiting for a schedule tick.
#
# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
# a time, and requires that --selftest FAILS for every one. A mutation that
# SURVIVES means the suite does not actually check that property, and the check
# fails, naming it.
#
# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
# write -- never mutating the flow users are logging in through. Each of those
# properties is enforced in code; these mutations are the proof that the enforcement
# is load-bearing rather than commentary. Several model the obvious "simplification"
# someone will propose:
# * "just build it under the declared alias, the suffix is ugly"
# * "bind it first, then check -- the check is just belt and braces"
# * "the protected-alias screen is redundant, the allow-list covers it"
# Every one of those would be a total SSO outage waiting to happen.
#
# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
# The sibling mutations.py dedents by four because its anchors are copied out of the
# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
# which is one less transformation to get wrong. check.sh passes the extracted file,
# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~60s (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster).
# =============================================================================
from __future__ import annotations
import re
import subprocess
import sys
import tempfile
from pathlib import Path
# (name, the mistake being modelled, anchor, replacement)
MUTATIONS: list[tuple[str, str, str, str]] = [
# ---------------------------------------------------------------- blue/green safety
("build INTO the currently bound flow",
"'the suffix is ugly, just build under the declared alias' -- every "
"create-execution and set-requirement call would then land on the flow users are "
"logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
"every SSO app and kubectl. This is the single regression the whole design exists "
"to make impossible",
" def mapped(alias):\n"
" return \"%s-%s\" % (alias, suffix)",
" def mapped(alias):\n return alias"),
("the protected-subtree screen is never called",
"'the allow-list already covers it' -- it does not: the allow-list's write entries "
"are alias-parameterised, so only this screen can say 'but not the flow that is "
"bound'. Without it the reconciler may write anywhere in the authentication tree",
" _screen_write(method, path, body if isinstance(body, dict) else None)",
" pass"),
("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
"reset them to frozenset() so the code reads more simply -- it silently removes "
"the fail-closed property, so a build that ran before the live binding had been "
"read would be PERMITTED, and could write straight into the live flow. (The "
"module-level initial value is deliberately NOT the anchor here: run() resets "
"these at the start of every run, so the initialiser is unreachable and mutating "
"it would prove nothing.)",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = None\n"
" _PROTECTED_EXECUTION_IDS = None",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = frozenset()\n"
" _PROTECTED_EXECUTION_IDS = frozenset()"),
("bind BEFORE verifying the build",
"'bind it first, the readback is just belt and braces' -- it is not: the readback "
"is what catches a build that was accepted but came out wrong, including the "
"conditional-OTP config landing under the wrong key. Binding first makes that the "
"live login path before anyone notices",
" verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
" \"the newly built flow %r\" % target)",
" verify = []"),
("a failed build is reported as SUCCESS",
"swallow the build error and return a converged result -- the realm is then left "
"on the old flow (which blue/green makes safe) but the run says it converged, so "
"nobody ever learns the passkey flow was never built. That is the one-shot "
"import's fail-open, reintroduced. NOTE: simply removing the early return here "
"is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
"is defence in depth working as intended -- so the mutation targets the reporting",
" problems.append(\n"
" \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
" \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
" \"place, unbound and harmless\" % (candidate, e))\n"
" return 0, problems",
" return 1, []"),
("an existing candidate flow is reused WITHOUT verifying it",
"'if the alias exists we already built it' -- a half-built leftover from a crashed "
"run lives under exactly that alias, so this binds a partially constructed login "
"path",
" if cand_shape is not None and not compare_shape(\n"
" cand_shape, expected, base, realm, token, configs,\n"
" \"the existing flow %r\" % candidate):",
" if cand_shape is not None:"),
# ------------------------------------- 2026-10-29 review findings, each made falsifiable
("sub-flows identified by `alias` again",
"read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
"recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
"sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
"a write into browser-mfa-forms would NOT be refused",
" name = ex.get(\"displayName\")\n"
" return name if isinstance(name, str) and name else None",
" name = ex.get(\"alias\")\n"
" return name if isinstance(name, str) and name else None"),
("an unnameable sub-flow is tolerated",
"carry on when a sub-flow row has no displayName -- the protected subtree is then "
"SMALLER than the live tree, which is the fail-open the guard exists to prevent",
" if name is None:\n"
" return None, None",
" if name is None:\n"
" name = \"\""),
("the requirement PUT is screened on its path only",
"trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
"body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
"the path can carry a BOUND execution's id in the body",
" targets.append((\"execution\", (body or {}).get(\"id\")))",
" pass"),
("execution-addressed writes are not held to the build tree",
"'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
"outside it too, so a stray id would edit an unrelated flow",
" if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
" or target not in _BUILD_EXECUTION_IDS):",
" if False:"),
("the requirement PUT may carry any key",
"let the body carry flowId / displayName -- the same handler then RENAMES whatever "
"flow flowId names, bound or not",
" if extra:\n"
" raise Refused(",
" if False:\n"
" raise Refused("),
("git's bind switch is ignored",
"bind whenever verification passes, as the first version did -- the first live use of "
"an untested login flow becomes a schedule tick, and an operator's rollback is "
"undone within 15 minutes, forever",
" bind = git_bind and not no_bind",
" bind = not no_bind"),
("bind=false still binds inside the flow pass",
"keep the flag but forget to honour it where the bind happens",
" if not bind:\n"
" # THE BIND SWITCH IS OFF.",
" if False:\n"
" # THE BIND SWITCH IS OFF."),
("an absent bind switch means bind",
"default to true when the key is missing -- a ConfigMap predating the switch, or a "
"deleted line, would silently put an untested flow live",
" except FileNotFoundError:\n"
" return False",
" except FileNotFoundError:\n"
" return True"),
("an invalid bind switch is guessed at",
"read anything that is not \"true\" as false -- a typo then passes silently instead of "
"stopping the run, and the operator believes the flow is live",
" if raw == \"false\":\n"
" return False\n"
" raise ValueError(",
" return False\n"
" raise ValueError("),
("bind=false rebuilds beside a hand-edited bound flow",
"treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
"off this builds an -r2 next to the operator's hotfix on every tick",
" if candidate in have and candidate == bound and not bind:",
" if False:"),
# ------------------------------------------------------------------- idempotence
("sub-flow aliases are compared as names",
"compare the sub-flow alias like any other field -- the flows this program builds "
"carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
"converged: it would be rebuilt and the login path reflipped every fifteen "
"minutes, forever",
" out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
" out.append((level, ex[\"kind\"], ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
("the converged short-circuit is removed",
"drop the 'bound flow already matches git' early return -- every run rebuilds and "
"reflips, which is a permanent write loop against the IdP and churns the realm's "
"login path on a schedule",
" if not drift:\n"
" log(\"flow ok (the bound flow %r matches git: %d execution(s), no \"\n"
" \"write)\" % (bound, len(expected)))\n"
" return 0, problems",
" if False:\n return 0, problems"),
("the execution priority is dropped from the requirement write",
"omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
"missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
"the execution order while only a requirement change was intended. The passkey "
"branch would end up in the wrong place in the tree",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
" \"priority\": ex.get(\"priority\") or 0})",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
("the alias budget is unbounded",
"loop forever looking for a free alias -- a persistently failing build then creates "
"unbounded flow sprawl in the realm, and nothing here can delete any of it",
" for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
" for attempt in range(1, 10000):"),
("fail-open when the bound flow cannot be read",
"carry on when the live flow's executions are unreadable -- the program then cannot "
"know which flows are live, so it cannot prove a build would miss them, and it "
"would build with the protected sets still undetermined",
" return 0, [\n"
" \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
" \"be read, so this program cannot establish which flows are live. Flow \"\n"
" \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
" % bound]",
" _set_protected(set(), set())\n bound_shape, bound_raw = None, None"),
# ------------------------------------------------------- the conditional-OTP regression
("authenticator config contents no longer compared",
"check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
"regression: the original bug wrote the role under "
"`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
"to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
"and every platform-admin got a 403 from every SSO-gated app",
" if str(have_cfg.get(key)) != str(want_cfg[key]):",
" if False:"),
("undeclared config keys are ignored",
"stop reporting keys the realm holds and git does not -- the stale "
"`condition.evaluated.user.role` key is the signature of the original null-role "
"bug, and a flow carrying both keys would look correct",
" extra = sorted(k for k in have_cfg if k not in want_cfg)",
" extra = []"),
("flow structure divergence detected but not reported",
"leave the per-step comparison out -- the realm's login flow could be the old "
"pre-passkey shape forever and every run would report the realm converged, which "
"is the exact fail-open of the one-shot import this program replaces",
" if observed[idx][:4] != expected[idx][:4]:",
" if False:"),
("the browserFlow BINDING is no longer compared",
"stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
"passkey flow that nothing logs in through changes nothing at all",
" if bound_shape is not None:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
" if False:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
# ------------------------------------------------------------- realm / policy writes
("partial policy PUT instead of read-modify-write",
"send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
"passwordless policy from the representation and substitutes its own DEFAULT for "
"every key the body omits, so a partial PUT silently resets everything it did not "
"mention while returning 204",
" body = dict(realm_rep)\n"
" for key in drift:\n"
" body[key] = wanted[key]",
" body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
" BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
" for key in drift:\n"
" body[key] = wanted[key]"),
("the policy pass rebinds browserFlow as a side effect",
"'while we are here, let the policy write fix the flow binding too' -- the realm "
"PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
"rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
"the passkey policy",
" if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
" raise RuntimeError(",
" body[BROWSER_FLOW] = \"browser\"\n"
" if False:\n"
" raise RuntimeError("),
("the RpEntityName gate removed from desired-state validation",
"accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is non-empty, so every realm PUT would "
"be accepted with 204 and change nothing: reported success, zero convergence",
" if not policy.get(RP_ENTITY_NAME):\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the RpEntityName gate removed from the write path",
"drop the second, defence-in-depth guard -- it is unreachable while the "
"desired-state validation holds, which is exactly why deleting it would otherwise "
"be an invisible regression",
" if not body.get(RP_ENTITY_NAME):\n"
" raise RuntimeError(",
" if False:\n raise RuntimeError("),
("no readback verification on the policy write",
"trust the 204 -- a server that accepts the PUT and does not apply it is reported "
"as REPAIRED, which is 'reports success while the realm never converged' again, "
"and it is precisely what the RpEntityName gate does when it is missed",
" still = policy_drift(after, wanted)",
" still = {}"),
("AAGUID comparison made order-sensitive",
"compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
"is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
"realm on every run forever while every run still reports success",
" if stored != sorted(declared_value):\n"
" drift[key] = (stored, sorted(declared_value))",
" if stored != declared_value:\n"
" drift[key] = (stored, declared_value)"),
("signature-algorithm order treated as insignificant",
"sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
"to the authenticator, so ES256-then-RS256 is not the same declaration as "
"RS256-then-ES256, and a reorder in the realm would never be corrected",
" if stored != list(declared_value):",
" if sorted(stored) != sorted(declared_value):"),
("unmanaged policy keys forced to a value",
"iterate every writable key instead of only the declared ones -- a "
"webAuthnPolicyPasswordless field the realm holds and git does not declare would "
"be overwritten, turning 'not declared' into 'must equal my default'",
" for key in sorted(wanted):\n declared_value = wanted[key]",
" for key in POLICY_FIELDS:\n declared_value = wanted.get(key, \"\")"),
# ------------------------------------------------------------------ required actions
("required-action PUT sends only the two drifted fields",
"a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
"the representation, so name, priority and config are all blanked: the enrolment "
"action keeps working but loses its priority and display name",
" body = dict(rep)\n for key in drift:\n body[key] = wanted[key]",
" body = {}\n for key in drift:\n body[key] = wanted[key]"),
("the guarded default required actions are no longer audited",
"stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
"exist -- their disappearance is the signature of a requiredActions block in the "
"realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
"lock-out of every platform-admin",
" missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
" missing = []"),
# ----------------------------------------------------------------- inputs / coupling
("the three passes are coupled",
"abort the run at the first failing pass -- a policy write that fails for an "
"unrelated reason would then suppress the browser-flow pass, which is the most "
"safety-critical thing here",
" problems.extend(found)\n if found:\n rc = 1",
" problems.extend(found)\n if found:\n rc = 1\n"
" break"),
("policy-key validation removed from the desired state",
"accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
"writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
"bruteForceProtected or any other top-level realm field on the live IdP",
" if key not in POLICY_FIELDS:",
" if False:"),
("fail-open on an unreadable desired state",
"treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
"reconciling nothing, which is precisely the fail-open of the realm import this "
"whole component exists to replace",
" if not isinstance(raw, dict) or not raw:\n"
" raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
" if not isinstance(raw, dict) or not raw:\n return {}"),
("an undeclared authenticatorConfig reference is tolerated",
"stop validating that every configAlias a flow references is actually declared -- "
"the build then fails at the config-binding step, AFTER flows and executions have "
"already been created, leaving leftovers on every run",
" if ca and ca not in configs:\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the flow-cycle guard is removed",
"drop the `seen` check -- a cyclic flow declaration then recurses until the "
"interpreter dies, in a CronJob that retries",
" if alias in seen:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
" if False:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
]
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
problems: list[str] = []
killed = 0
for name, why, old, new in MUTATIONS:
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n))
continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(src.replace(old, new, 1))
path = fh.name
try:
proc = subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300)
finally:
Path(path).unlink(missing_ok=True)
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how))
print()
if problems:
print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
% (len(problems), len(MUTATIONS)), file=sys.stderr)
for p in problems:
print(" FAIL %s" % p, file=sys.stderr)
return 1
print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
"(the suite is falsifiable, not decorative)" % killed)
return 0
if __name__ == "__main__":
raise SystemExit(main())
decisions/2026-10-18-keycloak-passkey-reconciler.md
+334
View File
@@ -0,0 +1,334 @@
---
date: 2026-10-18
branch: feat/keycloak-passkey-reconciler
title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
---
## What changed
**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
here.
**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
so the operator will never re-run it and every realm-level field or flow declared after the first
import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
2026-09-06 client-secret incident, third field over.
**What now makes it converge.** A **sibling** reconciler, modelled on
`platform/keycloak/client-secret-reconciler.yaml`:
- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
`secretKeyRef` only. Three passes, **all of which converge**:
1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
**literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
- `ci/render-keycloak-passkey-policy.py` + the **generated**
`platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
import, `--check`ed by the merge gate.
- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
(`scope: whole-repo`), `mutations.py`.
- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
realm import.
- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
live realm" section rewritten: all three declared things now converge, with the blue/green
mechanics, the rollback command and the `--no-bind` fallback written out.
### The flow design was reconsidered, and the first version of it was wrong
**This is worth recording because the superseded reasoning was not silly, it was just wrongly
scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
platform-admin.
**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
forces. So:
- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
and **zero user-visible effect**. The untransacted-calls problem dissolves.
- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
regression the operator feared is now caught *before* it can bite rather than after.
- The flip is **one atomic PUT** of the realm setting `browserFlow`.
- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
`DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
**The new safety property, enforced structurally rather than by intention.** Blue/green needs
flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
writes nothing.
**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
compared: a flow this program built carries a hash suffix, so comparing labels would make it
permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
half-built one is left alone and the next candidate alias is used, with the candidate count
**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
bargain than the one it would buy.
So the record is in two places that cost nothing: the **run log** prints
`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
even if every Job log has rotated away. The README carries the rollback procedure.
**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
(build and verify, leave the flip as a one-write human act) but it is not the default and was not
needed.
**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
`--check`, not by a second declaration being trusted.
**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
memory, which caught one I had initially missed. **Three**:
| Path | Why it is in the policy | What I changed |
|---|---|---|
| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
So this PR **requires an approving review from someone who is not its author**, and
`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
self-approval the stage exists to prevent. The operator needs to either review it or post the
override.
## What I verified
**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
- `kustomization/platform-config` `.status.lastAppliedRevision` ==
`gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
merge. So the inert declaration is genuinely applied and genuinely inert.
**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
established from source at tag `26.0.6`, not from documentation or memory:
- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
information in the representation will be ignored. This will only update top-level attributes of
the realm."*
- It delegates to `RepresentationToModel.updateRealm` →
`DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
**no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
`getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
the endpoint, not merely ignored by convention.
- Two findings that changed the design, both from the same read:
- `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
refuses the PUT if it would differ.
- The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
(line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
is a strict read-modify-write of the full live representation.
- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
**replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
config), and 404s when the alias is unregistered. Hence read-modify-write there too.
- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
`defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
restate a default while deleting ten siblings.
- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
`webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
those fields are emitted unconditionally, not only on the export path.
**The five flow-construction endpoints, read from 26.0.6 source before being used:**
- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
caught that, not my reading.
- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
so **insertion order is the resulting order** — which is why the builder inserts depth-first in
declared order and then verifies by readback.
- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
while only a requirement change was intended. Every requirement write therefore carries the
execution's current priority back. There is a mutation for it and a selftest assertion on the
resulting order.
- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
the old flow already uses the declared alias.
**Offline proofs, all green:**
- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
and flow construction where everything is created DISABLED — so these are tested against the real
semantics rather than a forgiving fake.
- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
that no write targeted the bound subtree, and that every live flow survives. This is the assertion
that makes blue/green a property rather than a claim.
- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
`condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
still live.
- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
safety argument itself (build into the bound flow; never call the screen; reset the protected sets
to empty; bind before verify; report a failed build as success; reuse an existing flow without
verifying it).
- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
**two of my proposed mutations were not real regressions**: the module-level initial value of the
protected sets is dead code (`run()` resets them first), and removing the early return after a
build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
*are* load-bearing rather than pretending the originals had been caught.
- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
`defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
(previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
stay off the list (they carry no flow alias, so they could not be screened); and — the
load-bearing one — that the **protected-subtree guard exists in the source, is wired into
`_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
that the policy write still refuses to rebind `browserFlow`**.
- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
it flattens to the exact 11-step tree the realm import declares, with
`conditional-otp-platform-admin` bound at level 4.
- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
`platform-config`, so one substitution pass decides the RP ID.
- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
manifest.
**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
the comparison and the undeclared-key report and requires the selftest to notice both. The suite
also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
a correct-but-unbound flow, and a cyclic declaration.
**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
are recorded there rather than paraphrased here.
## What I did NOT verify
- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
forbids decrypting or printing, so I did not read the realm's current
`webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
therefore **do not know** how much drift the first real run will report, nor that the live flow is
in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
matters more for the blue/green flow builder than for anything else in the change, because it is
**five endpoints I have never exercised**. Specifically unproven in reality:
- That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
send and assign priorities in insertion order. If they do not, the build produces a tree that
fails verification — which is the safe failure (nothing is bound, login unaffected), but it
means **the flow would never converge** and the Job would fail every 15 minutes.
- That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
*client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
the atomic bind fail closed** — loudly, writing nothing.
- The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
`CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
converges — rather than as an error.
- That `authenticationConfig` on an execution-info response is the config **id** (I read it from
the representation class, but never saw a live response).
- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
`getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
likewise fails. If either silently succeeded, the recovery path would behave differently than
modelled.
- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
**nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
that could destroy a rollback target) but it is a cost, not a non-issue.
- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
offline subset only; it proves nothing about the live realm.**
- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
asked for TOTP on the password branch. The README's four-step manual check remains the only thing
that would establish it, and it remains undone.
- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
is involved, with no server-side log line. The reconciler will happily converge the realm on a
wrong RP ID; it checks that git and the realm agree, never that git is correct.
- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
is deliberate and documented (username screen, then password), but it is the first time it will
have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
that is the mitigation, and it has not been rehearsed either.
- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
`ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
did not and will not post `/approve-override` on my own PR.
- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
`enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
§2 is **not re-confirmed** by me.
decisions/2026-10-29-keycloak-passkey-reconcile.md
+172
View File
@@ -0,0 +1,172 @@
---
date: 2026-10-29
branch: feat/keycloak-passkey-reconcile
title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
---
## What changed
This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
review of it. Read that entry for the design; this one records what the review found and what
changed.
**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
building and binding an `-r2`.
- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
`"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
/ `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
which would have failed the whole `platform-config` Kustomization, realm import included.
- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
`WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
is built beside it).
- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
and says not to hotfix the flow while it is `"true"`.
- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
git's value. The L0 check asserts the CronJob carries no `--` argument.
- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
order and why it matters, plus how to test the candidate on one client (a per-client
`account-console` browser-flow override) before the go-live PR.
**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
rows, which is why all 216 assertions passed anyway.
- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
`getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
leaves the flow pass fail-closed.
- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
`alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
provider id) and the config alias as `alias`. Nulls are omitted.
- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
the protected set, and an execution add or requirement write into them is refused. A missing
`displayName` fails closed.
**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
`authenticationFlow` + `displayName`. Now:
- `body["id"]` is screened against the protected execution ids.
- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
created, open only inside `build_flow`.
- The config `POST /executions/{id}/config` is held to the same build scope.
- The body may carry only `id`, `requirement` and `priority`.
- Selftest cases cover each of these.
**4. Stale docs on sensitive paths** now describe what ships, including the switch:
`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
renderer template, the renderer's own header and `--check` message, and the README.
**5.**
- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
another of the 5 candidate aliases.
- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
`for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
shape as the sibling's. It has promtool tests in
`platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
"the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
`runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
enables the `kubernetes` manager over `platform/**`. The image is pinned to
`python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
neither today's index nor today's amd64 manifest.
**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
in the script, where it ships next to the code and its selftest. A short header above the
ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
into `_finish_build()` so the build scope can be opened and closed around it.
Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
`platform-config/kustomization.yaml` (comment only). This needs the operator's
`/approve-override` or a non-author review. I did not post one.
## What I verified
- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
(including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
`alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
`SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
skipped by design (no cluster). rc 3 is **not** green.
- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
promtool 3.14.0.
- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
check.
- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
one fix:
- sub-flows read by `alias` again;
- an unnameable sub-flow tolerated;
- the requirement PUT screened on its path only;
- execution writes not held to the build tree;
- extra body keys allowed;
- the bind switch ignored in `run()`;
- the bind switch ignored in `reconcile_flow()`;
- an absent switch reading as true;
- an invalid switch guessed at;
- `-r2` built beside a hand-edited bound flow.
"Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
rather than failing an assertion). It is still caught.
- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
checked this by hand-editing the committed file and running `--check`.
- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
(lines ~225-231, 658-724 and 747-806 of the file at that tag).
- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
(2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
passkey reconciler is **not** live.
## What I did NOT verify
- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
source, but they are still my reading of that source, not a captured response. In particular,
I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
differs, the failure is fail-closed (no build), not fail-open.
- **The candidate flow has never been tested in a browser.** That is now a precondition of the
go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
I also have not checked that the per-client `account-console` override behaves as described on
26.0.6.
- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
realm.** They set the passwordless policy and enable the enrolment action. That action is
enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
The window is one HTTP round trip and only exists when the policy has drifted.
- **The alert has never seen a real series.** The CronJob has never run, so
`kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
shape is copied from the sibling's rule, which does have live series.
- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
`tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
run here.
- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
operator overrides.
### Follow-ups (out of scope here, deliberately not done)
- A dedicated least-privilege service-account client for the reconciler, instead of the
master-realm break-glass admin it shares with the client-secret reconciler.
- A NetworkPolicy for the `keycloak` namespace.
- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
decision, currently "no TOTP after a passkey".
- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
Only the failure alert was in scope.
platform-config/kustomization.yaml
+24
View File
@@ -127,6 +127,30 @@ resources:
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations # so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
# cannot disagree about the domain. # cannot disagree about the domain.
- ../platform/keycloak/client-urls.yaml - ../platform/keycloak/client-urls.yaml
# PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
# browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
# merged: the KeycloakRealmImport read Done=True / Started=False with the whole
# webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
# CronJob converges the policy and the enrolment required action, and builds the
# declared browser flow BLUE/GREEN: under a new alias, verified by readback, and
# structurally unable to write the flow that is currently bound (a half-rewritten bound
# flow is a total SSO outage, and this repo has had one). It BINDS the verified flow
# (one atomic PUT of browserFlow) ONLY when passkey-desired-state.yaml sets
# bind-browser-flow: "true"; the default "false" leaves the realm's binding untouched.
# See the script header in that file for the full rationale.
- ../platform/keycloak/passkey-reconciler.yaml
# Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
# ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
# client-urls.yaml above: the reconciler pod has no Kubernetes API access
# (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
# and its input must be MOUNTED.
#
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
# the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
# ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
# two different values. An RP ID is baked into every passkey at enrolment and can never
# be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
- ../platform/keycloak/passkey-desired-state.yaml
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC) - ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
- ../platform/keycloak/httproute.yaml - ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml - ../platform/keycloak/servicemonitor.yaml
platform/keycloak/README.md
+153 -15
View File
@@ -31,6 +31,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config | | `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config | | `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config | | `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one, **bound only when git says so**) | **platform-config** | config |
| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state — plus the one hand-edited line, `bind-browser-flow`) | **platform-config** | config |
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config | | `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config | | `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
@@ -154,7 +156,7 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
a realm edit landed. a realm edit landed.
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17) ## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless** `keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
@@ -170,7 +172,7 @@ comments, not repeated here. The headline choices:
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey | | Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that | | OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
### This change does NOT reach the live realm. At all. ### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
Same one-shot failure this file already documents twice above, third field over. Measured on prod Same one-shot failure this file already documents twice above, third field over. Measured on prod
2026-10-17: 2026-10-17:
@@ -188,24 +190,160 @@ endpoint allow-list makes `/authentication` unreachable **by design** (see the t
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
mapper or a changed scope still needs the realm import to actually re-run" case named above. mapper or a changed scope still needs the realm import to actually re-run" case named above.
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until `passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
an operator applies it by hand.** Two paths, both operator actions: secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy* Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to `ci/render-keycloak-passkey-policy.py` and gated by
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form `conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
into Username Form + Password Form and add the two sub-flows exactly as declared. That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against and **can never be re-keyed**, so two declarations free to skew about the domain would be an
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode unrecoverable-credential bug, not a cosmetic one.
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
Either way the realm is then configured by hand and git is only the record of intent, which is a | Declared thing | Who applies it | Why |
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it. |---|---|---|
| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
| The `browser-mfa` flow restructure | **The reconciler builds and verifies it. Git decides whether it is bound.** | Blue/green: build under a new alias, verify, and flip `browserFlow` in one atomic PUT **only when `bind-browser-flow: "true"`** — see below |
### Verifying it actually works (nobody has, as of 2026-10-17) #### The flow converges blue/green — it never mutates the flow you log in through
Converging a five-flow / ten-execution restructure needs create-subflow,
create-execution, set-requirement and bind-config as **separate** admin API calls, with no
transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
outage from a realm change, and the conditional-OTP sub-flow — which the declared
structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
That risk is real but it is **not inherent**: it follows only from mutating the *bound*
flow. So the reconciler does not.
1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
hash of the desired shape, applied to every flow in the tree. Every multi-call write
lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
flow and **zero user-visible effect**.
2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
regression that once 403'd every platform-admin is caught **before** it can bite.
3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow` — **only if git says
so** (next section). Otherwise the run stops here and logs `WOULD BIND browser-mfa-<hash>`.
4. **Rollback is a PR plus one write** (below), and nothing is ever deleted, so the old flow
stays usable.
#### The bind switch: `bind-browser-flow` in `passkey-desired-state.yaml`
That generated file carries exactly one hand-edited line, `bind-browser-flow`, and it is
`"false"` by default. The renderer carries it through on `--write`, and `--check` refuses
anything but the quoted strings `"true"` / `"false"` (an unquoted `true` is a YAML boolean,
which a ConfigMap rejects at apply time and which would fail the whole `platform-config`
Kustomization). The CronJob reads it from the same mount as the desired state; no flag can
widen it.
| `bind-browser-flow` | What the reconciler does to the realm's `browserFlow` |
|---|---|
| `"false"` (default) | **Never writes it.** It builds and verifies the candidate (or reuses an already verified one), logs `WOULD BIND <alias>`, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it). |
| `"true"` | Binds the verified candidate in one atomic PUT, **and keeps asserting it**: a binding changed by hand is changed back within 15 minutes, and a hand-edit *inside* the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is `"true"`. |
**Merging with `"false"` changes nothing about how anyone logs in.** Passes 1 and 2 do write
the realm (the passwordless policy and the enrolment action, neither of which any login
step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but
`browserFlow` is untouched.
**Going live** is a later one-line PR — `bind-browser-flow: "true"` — made **after** the
candidate has been tested in a browser (see "Testing the candidate flow before binding it").
**Rolling back**, in this order:
1. A PR setting `bind-browser-flow: "false"`, merged and applied by Flux. **First** — while it
is `"true"`, the next run rebinds the candidate over your rollback.
2. One write: bind the previous flow again. The Job log of the bind printed it
(`ROLLBACK is one write: set browserFlow back to ...`); it is `browser-mfa` for the first
go-live. Admin console: *Authentication → Flows → <previous flow> → Action → Bind flow →
Browser flow*, or `PUT /admin/realms/homelab` as a read-modify-write of the full realm
representation (a partial body drops the passkey policy).
If login is broken *right now*, do step 2 immediately and step 1 straight after: the worst
case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux
applies the PR, and you repeat step 2.
**The safety property is structural, not intentional.** The reconciler is *incapable* of
writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
whole subtree of sub-flow aliases and execution ids are read first and screened against
every write before a socket opens. It is **fail-closed** — until that binding has been
read, every flow write is refused, and if the bound flow's executions cannot be read the
flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
would churn the login path on a schedule. The bound flow is compared structurally against
the declared shape and, when they match, the run issues **zero writes and builds
nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
reconciler built carries a hash suffix, so comparing labels would make it permanently
unrecognisable as converged.
**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
later run reads it: if it matches desired it is a completed-but-unbound build and is
simply **bound**; if not, it is left alone and the next candidate alias is used. The
candidate count is **bounded**, so a persistently failing build fails loudly naming the
flows to clean up rather than creating unbounded sprawl.
```sh
# What the reconciler currently says. A normal run prints the full comparison.
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
# Which candidate it would bind (bind-browser-flow "false"):
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
| grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'
# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
# override the container command, so --audit / --no-bind need a one-off Job with the flag
# appended (--no-bind can only NARROW git's switch, never widen it).
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
passkey-run-once
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
# Browser flow). The old flow is never deleted, so this always works.
# Regenerate the desired state after editing the passkey policy or the flow in the realm
# import (the merge gate will demand it).
python3 ci/render-keycloak-passkey-policy.py --write
```
### Testing the candidate flow before binding it
With `bind-browser-flow: "false"` the candidate exists but nothing logs in through it. Test it
on **one client** without touching the realm binding, using a per-client flow override:
1. Find the candidate: the Job log line `WOULD BIND browser-mfa-<hash>`.
2. Admin console → *Clients → `account-console` → Advanced → Authentication flow overrides →
Browser Flow* = `browser-mfa-<hash>`. This is a temporary, hand-made client change; remove it
in step 7.
3. **Password fallback.** Private window → `https://keycloak.<base domain>/realms/homelab/account`.
Expect a **username** screen, then a choice: passkey, or the password form. As a user with no
passkey, choose password and sign in.
4. **Platform-admin TOTP on the password branch.** Repeat step 3 as a `platform-admins` member:
after the password, a TOTP prompt **must** appear. If it does not — or you get a 403 — the
conditional-OTP step is wrong: stop, do not bind.
5. **Passkey enrolment.** Signed in to the account console → *Signing in → Passkey → Set up*. A
browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and
nothing server-side will say so.
6. **Passkey login.** Sign out; private window again; username, then choose passkey. Expect to be
signed in with **no** TOTP prompt, even as a platform-admin.
7. Remove the override from step 2.
8. Only now: the one-line PR `bind-browser-flow: "true"`. The next run binds the candidate you
tested (same alias — the hash is of the declared shape).
### Verifying it actually works (nobody has, as of 2026-10-29)
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
it, in order: it on the BOUND flow, after go-live, in order:
```sh ```sh
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL) # 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
platform/keycloak/passkey-desired-state.yaml
+189
View File
@@ -0,0 +1,189 @@
---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
# the WebAuthn passwordless policy, the enrolment required action, and the declared
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
# it and (a) converges the policy and the required action onto it, (b) builds the
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
# writing the flow users are logging in through, and (c) BINDS that verified flow as
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
#
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
# operator decision, carried through by the renderer from this committed file (see
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a realm-level field or a flow edited in git never reaches the
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
# the level-triggered loop that closes that; this file is what it reconciles
# TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
# user-verification requirement and the flow tree would be a second source of truth,
# free to disagree with the realm import — the same "two declarations that can
# disagree" shape as the bug being closed. It is rendered from the import and the
# render is gated, so the two cannot drift.
#
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
# substitution token exactly as the realm import writes it. Both files are rendered
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
# same value in both and the domain is decided in exactly one place.
#
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
# nine more (a hard lock-out of every platform-admin). It is derived instead from
# the presence of a webauthn-authenticator-passwordless execution in the flow:
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
# false, so enrolment stays opt-in.
#
# A field NOT declared on the realm in the import is simply ABSENT here, and the
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: keycloak-passkey-desired-state
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-passkey-desired-state
app.kubernetes.io/part-of: keycloak
data:
# ---------------------------------------------------------------------------
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
# "false", QUOTED.
#
# "false" the reconciler builds and verifies the candidate browser flow and
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
# Whatever the realm is bound to -- including a rollback -- is left alone.
# "true" it binds the verified candidate in one atomic PUT, and keeps the
# realm bound to it: a binding changed by hand is changed back.
#
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
# "Passkeys"), then a PR setting this to "true".
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
# one write setting browserFlow back to the previous alias from the Job log.
# ---------------------------------------------------------------------------
bind-browser-flow: "false"
passkey-desired-state.json: |
{
"authenticatorConfig": {
"conditional-otp-platform-admin": {
"condUserRole": "platform-admin",
"negate": "false"
}
},
"browserFlow": "browser-mfa",
"flows": {
"browser-mfa": {
"executions": [
{
"kind": "authenticator",
"name": "auth-cookie",
"requirement": "ALTERNATIVE"
},
{
"kind": "authenticator",
"name": "identity-provider-redirector",
"requirement": "ALTERNATIVE"
},
{
"kind": "flow",
"name": "browser-mfa-forms",
"requirement": "ALTERNATIVE"
}
],
"topLevel": true
},
"browser-mfa-conditional-otp": {
"executions": [
{
"configAlias": "conditional-otp-platform-admin",
"kind": "authenticator",
"name": "conditional-user-role",
"requirement": "REQUIRED"
},
{
"kind": "authenticator",
"name": "auth-otp-form",
"requirement": "REQUIRED"
}
],
"topLevel": false
},
"browser-mfa-forms": {
"executions": [
{
"kind": "authenticator",
"name": "auth-username-form",
"requirement": "REQUIRED"
},
{
"kind": "flow",
"name": "browser-mfa-passkey-or-password",
"requirement": "REQUIRED"
}
],
"topLevel": false
},
"browser-mfa-passkey-or-password": {
"executions": [
{
"kind": "authenticator",
"name": "webauthn-authenticator-passwordless",
"requirement": "ALTERNATIVE"
},
{
"kind": "flow",
"name": "browser-mfa-password-and-otp",
"requirement": "ALTERNATIVE"
}
],
"topLevel": false
},
"browser-mfa-password-and-otp": {
"executions": [
{
"kind": "authenticator",
"name": "auth-password-form",
"requirement": "REQUIRED"
},
{
"kind": "flow",
"name": "browser-mfa-conditional-otp",
"requirement": "CONDITIONAL"
}
],
"topLevel": false
}
},
"realm": "homelab",
"requiredAction": {
"alias": "webauthn-register-passwordless",
"defaultAction": false,
"enabled": true
},
"webAuthnPolicyPasswordless": {
"webAuthnPolicyPasswordlessAcceptableAaguids": [],
"webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
"webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
"webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
"webAuthnPolicyPasswordlessCreateTimeout": 300,
"webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
"webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
"webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
"webAuthnPolicyPasswordlessSignatureAlgorithms": [
"ES256",
"RS256"
],
"webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
}
}
platform/keycloak/passkey-reconciler.yaml
+3276
View File
File diff suppressed because it is too large. Load diff
platform/monitoring/alerts.yaml
+30
View File
@@ -1445,6 +1445,36 @@ spec:
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
that would catch the healer being quietly absent during a domain migration. that would catch the healer being quietly absent during a domain migration.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets" runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
# enrolment required action, and the browser login flow (blue/green; bound only when
# git's bind-browser-flow is "true"). There is no failure that is "expected by
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
# single transient failure while still firing within about two ticks.
# A separate rule from the client reconciler's on purpose: different CronJob,
# different job_name, different runbook — one cause, one alert.
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
# did not exist yet. The metric and label are the ones the sibling rule above uses.
- alert: KeycloakPasskeyReconcilerFailing
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak passkey reconciler failing"
description: >-
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
policy, the passkey enrolment action or the browser login flow may differ from
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
never writes the flow users are logging in through, so a failure here does NOT
by itself mean login is broken. Read the Job log: every PROBLEM line names the
pass and the exact divergence.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
# ===================================================================================== # =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10) # The monitor's own capacity — don't let the monitor blind itself (docs/10)
platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml
+90
View File
@@ -0,0 +1,90 @@
# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
# group platform-health.identity).
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
# flow, so a failed Job is always news. What can go wrong with the rule:
# * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
# * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
# failures already have their own alert and runbook; one cause, one alert;
# * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# =============================================================================================
# 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
# that forgot the `> 0` would fire here.
# =============================================================================================
- interval: 1m
name: passkey reconciler succeeding - quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
values: '0+0x60'
- series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
values: '1+0x60'
alert_rule_test:
- eval_time: 60m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
# =============================================================================================
# 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
# =============================================================================================
- interval: 1m
name: passkey reconciler Job failed - fires after 15m
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
values: '1+0x60'
alert_rule_test:
- eval_time: 10m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
- eval_time: 20m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts:
- exp_labels:
severity: warning
exp_annotations:
summary: "Keycloak passkey reconciler failing"
description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
# =============================================================================================
# 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
# same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
# =============================================================================================
- interval: 1m
name: only the client-secret reconciler failing - passkey alert quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
values: '1+0x60'
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
values: '0+0x60'
alert_rule_test:
- eval_time: 60m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
# =============================================================================================
# 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
# series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
# =============================================================================================
- interval: 1m
name: single transient failure cleared within 15m - quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
values: '1+0x5'
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
values: '_x15 0+0x45'
alert_rule_test:
- eval_time: 30m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
runbooks/README.md
+2
View File
@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable. - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed. every green signal missed.
- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
rather than a 401, the browser follows it silently against a live SSO session, and the call rather than a 401, the browser follows it silently against a live SSO session, and the call
runbooks/keycloak-passkey-reconciler.md
+69
View File
@@ -0,0 +1,69 @@
# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
```
## What this means here
`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
at `Done=True`:
1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
(`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
`bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
default) a run that builds and verifies a candidate but does not bind it is a **success**.
So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
is structurally unable to write the flow users are currently logging in through, so **a failing Job
does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
`CONFIGURE_TOTP` required action (see below).
## First commands
Every command runs in the pinned toolbox — kubectl is not on the workstation:
```sh
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
# What git currently asks for (the bind switch):
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
```
## Likely causes, most likely first
1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
(Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
but the readback differs (the log names the step). Login is unaffected. If it names the
conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
platform-admin, caught before it went live.
3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
`requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
lock-out for platform-admins: re-register the action in the admin console
(Authentication → Required actions). The reconciler cannot repair it.
4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
`keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
(`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
transient; if persistent, the realm is bound to a flow that no longer exists.
## When to escalate or stop
- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
`"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
alias printed in the Job log).
- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.
runbooks/kustomization.yaml
+1
View File
@@ -77,6 +77,7 @@ configMapGenerator:
- image-automation=image-automation.md - image-automation=image-automation.md
- image-policy-stalled=image-policy-stalled.md - image-policy-stalled=image-policy-stalled.md
- keycloak-client-secrets=keycloak-client-secrets.md - keycloak-client-secrets=keycloak-client-secrets.md
- keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
- keycloak-db=keycloak-db.md - keycloak-db=keycloak-db.md
- keycloak-down=keycloak-down.md - keycloak-down=keycloak-down.md
- kubevirt-control-plane=kubevirt-control-plane.md - kubevirt-control-plane=kubevirt-control-plane.md