Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
262c6e4a97 | ||
|
|
523cd9118f | ||
|
|
d2c4bd5194 | ||
|
|
0dd57fbaf0 | ||
|
|
6bb8af1d70 | ||
|
|
7c4eb439b2 | ||
|
|
17eed6e8ae | ||
|
|
f5931944fa |
No files matched your search
@@ -0,0 +1,490 @@
|
||||
#!/usr/bin/env python3
|
||||
# =============================================================================
|
||||
# ci/render-keycloak-passkey-policy.py
|
||||
#
|
||||
# Render platform/keycloak/passkey-desired-state.yaml FROM
|
||||
# platform/keycloak/keycloak-realm-import.yaml.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# ---------------
|
||||
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
|
||||
# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
|
||||
# after the passkey declaration merged:
|
||||
#
|
||||
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
|
||||
# Done=True Started=False HasErrors=False
|
||||
#
|
||||
# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
|
||||
# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
|
||||
# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
|
||||
# is the level-triggered loop that closes that; this renderer supplies the DESIRED
|
||||
# STATE it converges towards.
|
||||
#
|
||||
# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
|
||||
# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
|
||||
# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
|
||||
# flow in a browser; make it the login path" -- not a fact derivable from the realm
|
||||
# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
|
||||
# file: --write preserves it, --check accepts any value of it that is exactly the
|
||||
# string "true" or "false" and rejects everything else (an unquoted YAML true is a
|
||||
# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
|
||||
# Kustomization -- so it is refused HERE, at merge time). An absent file or key
|
||||
# renders as "false". Going live is therefore a one-line PR flipping that line.
|
||||
#
|
||||
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
|
||||
# automountServiceAccountToken: false, exactly like its client-secret sibling. It
|
||||
# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
|
||||
# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
|
||||
# must be MOUNTED. Hence a ConfigMap.
|
||||
#
|
||||
# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
|
||||
# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
|
||||
# the user-verification requirement and the whole flow tree — free to disagree with
|
||||
# the realm import. "Two declarations that can disagree" is the same shape as the
|
||||
# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
|
||||
# import and the generation is GATED:
|
||||
# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
|
||||
# with --check on every merge, and the merge fails if the committed file is not what
|
||||
# this renderer produces. Same render==golden idiom as
|
||||
# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
|
||||
#
|
||||
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
|
||||
# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
|
||||
# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
|
||||
# rendered by the SAME Flux Kustomization (./platform-config, whose
|
||||
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
|
||||
# pass with the same value and cannot disagree about what the base domain is. This
|
||||
# renderer never parses, splits or reconstructs a hostname and never reads
|
||||
# cluster-vars — there is no second place the domain is decided.
|
||||
#
|
||||
# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
|
||||
# -----------------------------------------------------
|
||||
# The enrolment required action, `webauthn-register-passwordless`. It is
|
||||
# deliberately NOT declared in the realm import, and the import's own header
|
||||
# explains at length why declaring it there would be actively DANGEROUS: Keycloak
|
||||
# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
|
||||
# so declaring one action imports exactly that action plus delete_account and skips
|
||||
# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
|
||||
# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
|
||||
# lock-out of every platform-admin, because auth-otp-form in the browser flow still
|
||||
# demands a TOTP they could then never enrol.
|
||||
#
|
||||
# So the required action cannot be copied from the import, because it is correctly
|
||||
# absent from it. It is instead DERIVED, as a consequence of what the import DOES
|
||||
# declare:
|
||||
#
|
||||
# IF the declared browser flow contains a `webauthn-authenticator-passwordless`
|
||||
# execution, THEN passkey logins are offered, THEREFORE the enrolment action must
|
||||
# be registered and enabled — and NOT a defaultAction, because the import's whole
|
||||
# stated posture is "additive and optional" and a defaultAction forces enrolment
|
||||
# on every user at next login.
|
||||
#
|
||||
# This is a derivation, not a second declaration: there is no value here that a
|
||||
# human could edit into disagreement with the import. If the import stops declaring
|
||||
# the passkey execution, this renderer stops emitting the required action (and
|
||||
# fails, because then the whole desired state is empty of passkey intent).
|
||||
#
|
||||
# WHAT IS DELIBERATELY NOT EXPORTED
|
||||
# * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
|
||||
# `webauthn-authenticator` two-factor authenticator; the passwordless
|
||||
# authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
|
||||
# into the two-factor family is a silent no-op. Exporting it would hand the
|
||||
# reconciler a write surface with no purpose.
|
||||
# * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
|
||||
# SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
|
||||
# and users/groups/roles are nobody's — prod's real user and the media-users /
|
||||
# platform-admins / viewers groups must survive every run of everything.
|
||||
# * execution `priority`. It is consumed here to ORDER the exported executions
|
||||
# (Keycloak returns them in priority order), but the numbers themselves are not
|
||||
# exported: the reconciler compares the resulting SEQUENCE, which is what
|
||||
# actually determines behaviour. Exporting the integers would make a
|
||||
# priority renumbering that changes no ordering read as drift.
|
||||
#
|
||||
# USAGE
|
||||
# python3 ci/render-keycloak-passkey-policy.py # print the rendered file
|
||||
# python3 ci/render-keycloak-passkey-policy.py --write # write it (regenerate)
|
||||
# python3 ci/render-keycloak-passkey-policy.py --check # exit 1 if it has drifted
|
||||
# =============================================================================
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
|
||||
OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
|
||||
DATA_KEY = "passkey-desired-state.json"
|
||||
CONFIGMAP_NAME = "keycloak-passkey-desired-state"
|
||||
NAMESPACE = "keycloak"
|
||||
|
||||
# The realm keys this renderer exports, and therefore the ONLY realm keys the
|
||||
# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
|
||||
# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
|
||||
# tuples are identical, so they cannot drift apart silently.
|
||||
POLICY_STR_FIELDS = (
|
||||
"webAuthnPolicyPasswordlessRpId",
|
||||
"webAuthnPolicyPasswordlessRpEntityName",
|
||||
"webAuthnPolicyPasswordlessRequireResidentKey",
|
||||
"webAuthnPolicyPasswordlessUserVerificationRequirement",
|
||||
"webAuthnPolicyPasswordlessAuthenticatorAttachment",
|
||||
"webAuthnPolicyPasswordlessAttestationConveyancePreference",
|
||||
)
|
||||
POLICY_LIST_FIELDS = (
|
||||
"webAuthnPolicyPasswordlessSignatureAlgorithms",
|
||||
"webAuthnPolicyPasswordlessAcceptableAaguids",
|
||||
)
|
||||
POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
|
||||
POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
|
||||
POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
|
||||
+ POLICY_BOOL_FIELDS)
|
||||
|
||||
# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
|
||||
# (DefaultExportImportManager.updateRealm: the policy is applied only when
|
||||
# RpEntityName is non-null and non-empty). A desired state without it would produce
|
||||
# an accepted PUT that changes nothing, so refuse to render one.
|
||||
RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
|
||||
|
||||
# The execution whose presence is what makes passkey enrolment necessary.
|
||||
PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
|
||||
ENROL_ACTION = "webauthn-register-passwordless"
|
||||
|
||||
|
||||
BIND_KEY = "bind-browser-flow"
|
||||
BIND_VALUES = ("true", "false")
|
||||
|
||||
|
||||
def committed_bind(root: Path) -> str:
|
||||
"""The operator's bind-browser-flow value from the COMMITTED file, or "false".
|
||||
|
||||
This is the only input read from the output file, and it is validated, not trusted:
|
||||
anything other than the exact strings "true" / "false" is a hard failure."""
|
||||
target = root / OUT_FILE
|
||||
if not target.is_file():
|
||||
return "false"
|
||||
docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
|
||||
cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
|
||||
data = cm.get("data") or {}
|
||||
if BIND_KEY not in data:
|
||||
return "false"
|
||||
v = data[BIND_KEY]
|
||||
if not isinstance(v, str) or v not in BIND_VALUES:
|
||||
raise SystemExit(
|
||||
"FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
|
||||
"\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
|
||||
"at apply time and which would fail the whole platform-config Kustomization"
|
||||
% (OUT_FILE, BIND_KEY, v))
|
||||
return v
|
||||
|
||||
|
||||
def render(root: Path) -> str:
|
||||
"""Return the full text of passkey-desired-state.yaml as it should be committed."""
|
||||
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
|
||||
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
|
||||
if imp is None:
|
||||
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
|
||||
realm = (imp.get("spec") or {}).get("realm") or {}
|
||||
realm_name = realm.get("realm")
|
||||
if not isinstance(realm_name, str) or not realm_name:
|
||||
raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
|
||||
|
||||
# ---- the passwordless policy, copied VERBATIM ----------------------------------
|
||||
policy: dict[str, object] = {}
|
||||
for f in POLICY_STR_FIELDS:
|
||||
if f not in realm:
|
||||
continue # not declared => NOT MANAGED
|
||||
v = realm[f]
|
||||
# A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
|
||||
# a YAML quoting mistake that must not be silently stringified.
|
||||
if not isinstance(v, str):
|
||||
raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
|
||||
"quote it in the realm import" % (IMPORT_FILE, f, v))
|
||||
policy[f] = v
|
||||
for f in POLICY_LIST_FIELDS:
|
||||
if f not in realm:
|
||||
continue
|
||||
v = realm[f]
|
||||
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
|
||||
raise SystemExit("FATAL: %s realm field %s is not a list of strings"
|
||||
% (IMPORT_FILE, f))
|
||||
policy[f] = list(v) # verbatim, order as authored
|
||||
for f in POLICY_INT_FIELDS:
|
||||
if f not in realm:
|
||||
continue
|
||||
v = realm[f]
|
||||
if not isinstance(v, int) or isinstance(v, bool):
|
||||
raise SystemExit("FATAL: %s realm field %s is not an integer"
|
||||
% (IMPORT_FILE, f))
|
||||
policy[f] = v
|
||||
for f in POLICY_BOOL_FIELDS:
|
||||
if f not in realm:
|
||||
continue
|
||||
v = realm[f]
|
||||
if not isinstance(v, bool):
|
||||
raise SystemExit("FATAL: %s realm field %s is not a boolean"
|
||||
% (IMPORT_FILE, f))
|
||||
policy[f] = v
|
||||
|
||||
if not policy:
|
||||
# Fail closed. An empty policy would make the reconciler believe every field is
|
||||
# unmanaged and silently stop reconciling — the realm import's own fail-open.
|
||||
raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
|
||||
"refusing to render an empty desired state" % IMPORT_FILE)
|
||||
if not policy.get(RP_ENTITY_NAME):
|
||||
raise SystemExit(
|
||||
"FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
|
||||
"passwordless policy ONLY when that field is set, so every realm PUT would "
|
||||
"be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
|
||||
|
||||
# ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
|
||||
browser_flow = realm.get("browserFlow")
|
||||
if not isinstance(browser_flow, str) or not browser_flow:
|
||||
raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
|
||||
"be perfect and still not be the one anyone logs in through"
|
||||
% IMPORT_FILE)
|
||||
raw_flows = realm.get("authenticationFlows") or []
|
||||
if not isinstance(raw_flows, list) or not raw_flows:
|
||||
raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
|
||||
|
||||
flows: dict[str, object] = {}
|
||||
for fl in raw_flows:
|
||||
alias = fl.get("alias")
|
||||
if not isinstance(alias, str) or not alias:
|
||||
raise SystemExit("FATAL: %s has an authentication flow with no alias"
|
||||
% IMPORT_FILE)
|
||||
if alias in flows:
|
||||
raise SystemExit("FATAL: %s declares the flow %r twice"
|
||||
% (IMPORT_FILE, alias))
|
||||
execs = fl.get("authenticationExecutions") or []
|
||||
if not isinstance(execs, list) or not execs:
|
||||
raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
|
||||
% (IMPORT_FILE, alias))
|
||||
ordered = []
|
||||
for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
|
||||
is_flow = bool(ex.get("authenticatorFlow"))
|
||||
name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
|
||||
if not isinstance(name, str) or not name:
|
||||
raise SystemExit(
|
||||
"FATAL: %s flow %r has an execution with neither an authenticator "
|
||||
"nor a flowAlias" % (IMPORT_FILE, alias))
|
||||
requirement = ex.get("requirement")
|
||||
if not isinstance(requirement, str) or not requirement:
|
||||
raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
|
||||
% (IMPORT_FILE, alias, name))
|
||||
entry: dict[str, object] = {
|
||||
"kind": "flow" if is_flow else "authenticator",
|
||||
"name": name,
|
||||
"requirement": requirement,
|
||||
}
|
||||
cfg = ex.get("authenticatorConfig")
|
||||
if cfg is not None:
|
||||
if not isinstance(cfg, str) or not cfg:
|
||||
raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
|
||||
"authenticatorConfig" % (IMPORT_FILE, alias, name))
|
||||
entry["configAlias"] = cfg
|
||||
ordered.append(entry)
|
||||
flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
|
||||
|
||||
if browser_flow not in flows:
|
||||
raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
|
||||
% (IMPORT_FILE, browser_flow))
|
||||
# Every referenced subflow must be declared, or the reconciler's flattening walks
|
||||
# into a hole and the audit silently covers less than it claims.
|
||||
for alias, spec in flows.items():
|
||||
for ex in spec["executions"]:
|
||||
if ex["kind"] == "flow" and ex["name"] not in flows:
|
||||
raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
|
||||
"declared" % (IMPORT_FILE, alias, ex["name"]))
|
||||
|
||||
# ---- authenticator configs, copied verbatim ------------------------------------
|
||||
configs: dict[str, dict] = {}
|
||||
for cfg in realm.get("authenticatorConfig") or []:
|
||||
alias = cfg.get("alias")
|
||||
if not isinstance(alias, str) or not alias:
|
||||
raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
|
||||
% IMPORT_FILE)
|
||||
body = cfg.get("config") or {}
|
||||
if not isinstance(body, dict) or not body:
|
||||
raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
|
||||
% (IMPORT_FILE, alias))
|
||||
# Values are stringified because Keycloak stores authenticator config values as
|
||||
# strings. `negate: "false"` in the import and a stored "false" are the same
|
||||
# declaration, and a YAML-unquoted `false` must not become a Python bool that
|
||||
# then never matches.
|
||||
configs[alias] = {k: str(v) for k, v in sorted(body.items())}
|
||||
|
||||
# Every configAlias a flow references must be defined, else the reconciler reports
|
||||
# "git declares config X but defines no key/value for it" on every run forever.
|
||||
for alias, spec in flows.items():
|
||||
for ex in spec["executions"]:
|
||||
ca = ex.get("configAlias")
|
||||
if ca and ca not in configs:
|
||||
raise SystemExit(
|
||||
"FATAL: %s flow %r execution %r references authenticatorConfig %r, "
|
||||
"which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
|
||||
|
||||
# ---- the enrolment required action, DERIVED (see the header) -------------------
|
||||
offers_passkey = any(
|
||||
ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
|
||||
for spec in flows.values() for ex in spec["executions"])
|
||||
if not offers_passkey:
|
||||
raise SystemExit(
|
||||
"FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
|
||||
"in any flow. The policy decides what a passkey must BE; without that "
|
||||
"execution nothing ever OFFERS one, so the policy is unreachable "
|
||||
"configuration and this desired state would be meaningless"
|
||||
% (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
|
||||
|
||||
want = {
|
||||
"realm": realm_name,
|
||||
"webAuthnPolicyPasswordless": policy,
|
||||
# enabled: passkey enrolment must be possible, since the flow offers passkeys.
|
||||
# defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
|
||||
# enrol a passkey at next login — the platform-wide lock-out the realm import's
|
||||
# "additive and optional" posture exists to avoid. Derived, not copied; see the
|
||||
# header for why the import cannot declare this itself.
|
||||
"requiredAction": {"alias": ENROL_ACTION, "enabled": True,
|
||||
"defaultAction": False},
|
||||
"browserFlow": browser_flow,
|
||||
"flows": flows,
|
||||
"authenticatorConfig": configs,
|
||||
}
|
||||
|
||||
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
|
||||
return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
|
||||
+ " %s: |\n" % DATA_KEY + indented + "\n")
|
||||
|
||||
|
||||
HEADER = """---
|
||||
# =============================================================================
|
||||
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
|
||||
#
|
||||
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
|
||||
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
|
||||
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
|
||||
# (runs this renderer with --check on every merge)
|
||||
#
|
||||
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
|
||||
# the WebAuthn passwordless policy, the enrolment required action, and the declared
|
||||
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
|
||||
# it and (a) converges the policy and the required action onto it, (b) builds the
|
||||
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
|
||||
# writing the flow users are logging in through, and (c) BINDS that verified flow as
|
||||
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
|
||||
#
|
||||
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
|
||||
# operator decision, carried through by the renderer from this committed file (see
|
||||
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
|
||||
#
|
||||
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
|
||||
# once Done=True, so a realm-level field or a flow edited in git never reaches the
|
||||
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
|
||||
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
|
||||
# the level-triggered loop that closes that; this file is what it reconciles
|
||||
# TOWARDS.
|
||||
#
|
||||
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
|
||||
# user-verification requirement and the flow tree would be a second source of truth,
|
||||
# free to disagree with the realm import — the same "two declarations that can
|
||||
# disagree" shape as the bug being closed. It is rendered from the import and the
|
||||
# render is gated, so the two cannot drift.
|
||||
#
|
||||
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
|
||||
# substitution token exactly as the realm import writes it. Both files are rendered
|
||||
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
|
||||
# same value in both and the domain is decided in exactly one place.
|
||||
#
|
||||
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
|
||||
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
|
||||
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
|
||||
# nine more (a hard lock-out of every platform-admin). It is derived instead from
|
||||
# the presence of a webauthn-authenticator-passwordless execution in the flow:
|
||||
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
|
||||
# false, so enrolment stays opt-in.
|
||||
#
|
||||
# A field NOT declared on the realm in the import is simply ABSENT here, and the
|
||||
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: %s
|
||||
namespace: %s
|
||||
labels:
|
||||
app.kubernetes.io/name: %s
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
|
||||
|
||||
BIND_BLOCK = """ # ---------------------------------------------------------------------------
|
||||
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
|
||||
# "false", QUOTED.
|
||||
#
|
||||
# "false" the reconciler builds and verifies the candidate browser flow and
|
||||
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
|
||||
# Whatever the realm is bound to -- including a rollback -- is left alone.
|
||||
# "true" it binds the verified candidate in one atomic PUT, and keeps the
|
||||
# realm bound to it: a binding changed by hand is changed back.
|
||||
#
|
||||
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
|
||||
# "Passkeys"), then a PR setting this to "true".
|
||||
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
|
||||
# one write setting browserFlow back to the previous alias from the Job log.
|
||||
# ---------------------------------------------------------------------------
|
||||
%s: "%s"
|
||||
"""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
g = ap.add_mutually_exclusive_group()
|
||||
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
|
||||
g.add_argument("--check", action="store_true",
|
||||
help="exit 1 if the committed file has drifted from the source")
|
||||
args = ap.parse_args()
|
||||
|
||||
want = render(ROOT)
|
||||
target = ROOT / OUT_FILE
|
||||
|
||||
if args.write:
|
||||
target.write_text(want, encoding="utf-8")
|
||||
print("wrote %s" % OUT_FILE)
|
||||
return 0
|
||||
|
||||
if args.check:
|
||||
if not target.is_file():
|
||||
print("CHECK FAIL: %s does not exist. Run: python3 "
|
||||
"ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
have = target.read_text(encoding="utf-8")
|
||||
if have == want:
|
||||
print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
|
||||
return 0
|
||||
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
|
||||
"realm on a passkey policy the realm import does not declare, and build "
|
||||
"the login flow from a tree nobody authored.\n"
|
||||
"Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
|
||||
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
|
||||
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
|
||||
fromfile="committed " + OUT_FILE,
|
||||
tofile="rendered from " + IMPORT_FILE):
|
||||
sys.stderr.write(line)
|
||||
return 1
|
||||
|
||||
sys.stdout.write(want)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,108 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the passkey configuration is derived from one declaration, continuously
|
||||
# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
|
||||
# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
|
||||
# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
|
||||
# Measured on prod 2026-10-18, hours after the passkey declaration merged:
|
||||
#
|
||||
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
# Done=True Started=False HasErrors=False
|
||||
#
|
||||
# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
|
||||
# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
|
||||
# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
|
||||
# them changed nothing about how anyone logs in, and nothing anywhere said so.
|
||||
# Declarative config that silently does not apply is not declarative config.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN the reconciler's desired state is GENERATED from the realm import (so the RP ID —
|
||||
# which is baked into every passkey at enrolment and can NEVER be re-keyed — is
|
||||
# declared exactly once); both files are rendered by the SAME Flux Kustomization, so
|
||||
# their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
|
||||
# policy keys and the reconciler's writable keys are the SAME tuple; the realm
|
||||
# import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
|
||||
# delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
|
||||
# enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
|
||||
# no ServiceAccount token, mounts only what it needs, passes no mode flag, and
|
||||
# contains no dollar sign; the endpoint allow-list contains NO delete, NO
|
||||
# single-execution or config-edit write, and NO required-action write for any alias
|
||||
# other than the one named literally; the protected-subtree guard exists, is wired
|
||||
# into the single request choke point and is fail-closed; the SIBLING client-secret
|
||||
# reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
|
||||
# (extracted from the shipped ConfigMap and executed) proves compare-then-write,
|
||||
# idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
|
||||
# its sub-flows unwritable, the git bind switch and a rollback that sticks, the
|
||||
# conditional-OTP regression and fail-closed — with mutations.py proving that
|
||||
# selftest can FAIL.
|
||||
#
|
||||
# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
|
||||
# than only argued: converging a five-flow, ten-execution restructure needs
|
||||
# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
|
||||
# calls with no transaction. Doing that to the BOUND flow means a run that dies between
|
||||
# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
|
||||
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
|
||||
# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
|
||||
# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
|
||||
# which is verified by readback before anything is bound. Binding it is then ONE atomic
|
||||
# PUT of browserFlow, made only when passkey-desired-state.yaml sets
|
||||
# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
|
||||
# a schedule tick, and with the switch off an operator's rollback is never undone.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
|
||||
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
||||
# Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
|
||||
# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
|
||||
# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
|
||||
# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
|
||||
# password form (needs a real browser against a real realm); that the five flow-building
|
||||
# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
|
||||
# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: keycloak-passkey-reconciler
|
||||
labels:
|
||||
conformance.home.k3s/R-IDP-03: "true"
|
||||
conformance.home.k3s/R-IDP-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
|
||||
# check declares it: this invariant spans platform/ (the realm import, the reconciler,
|
||||
# the generated desired state), platform-config/ (which Flux Kustomization renders
|
||||
# them, and therefore which substitution pass they share), ci/ (the renderer) and
|
||||
# clusters/ (that both are reachable from both entrypoints). A change that breaks it
|
||||
# need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
|
||||
# then filter this check out of exactly the change that introduces the bug — which is
|
||||
# how 12 unusable prod OIDC clients sat behind an entirely green cluster on
|
||||
# 2026-09-06.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
# Not safety-critical in the harness's sense (it mutates nothing and needs no
|
||||
# cluster), but it guards a component that holds WRITE access to the identity
|
||||
# provider's realm settings.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
The realm's passkey configuration — policy, enrolment action AND browser login
|
||||
flow — is declared once in the realm import, derived from it by a gated renderer,
|
||||
and continuously reconciled; the flow converges blue/green, so a one-shot import
|
||||
that never re-runs can neither leave the configuration inert nor let a reconciler
|
||||
half-rewrite the login path of the whole platform.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-passkey-declaration-renderer-and-reconciler
|
||||
description: >-
|
||||
Assert the desired state is derived from the realm import and shares its
|
||||
substitution pass, that the reconciler cannot write the bound flow or delete
|
||||
anything and binds only on git's bind-browser-flow, that the sibling reconciler
|
||||
is unwidened, and run the reconciler's own selftest plus the mutation suite that
|
||||
proves it can fail.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,754 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
|
||||
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
|
||||
# — spec docs/15-identity-and-access.md)
|
||||
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
|
||||
# — spec docs/15-identity-and-access.md)
|
||||
#
|
||||
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
|
||||
# ---------------------------------------------------------------
|
||||
# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
|
||||
# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
|
||||
# hours after that declaration merged:
|
||||
#
|
||||
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
# Done=True Started=False HasErrors=False
|
||||
#
|
||||
# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
|
||||
# re-runs it, so every realm-level field and every flow declared after the FIRST
|
||||
# import is inert — in git, applied by Flux, reported Ready, reaching the realm
|
||||
# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
|
||||
# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
|
||||
# the same fix: a level-triggered reconciler. This check is the static half.
|
||||
#
|
||||
# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
|
||||
# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
|
||||
# (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
|
||||
# import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
|
||||
# renderer with --check, so a hand edit that makes the two disagree is
|
||||
# unmergeable. The RP ID is the sharpest case: it is baked into every passkey
|
||||
# at enrolment and can NEVER be re-keyed, so two declarations that can skew
|
||||
# about the domain is an unrecoverable-credential bug, not a cosmetic one.
|
||||
# (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
|
||||
# postBuild.substituteFrom is per-Kustomization, so a split would let the
|
||||
# ${BASE_DOMAIN} token in the realm import and the one in the desired state
|
||||
# resolve to two different values — and an RP ID mismatch invalidates every
|
||||
# enrolled passkey with no error the user can act on.
|
||||
# (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
|
||||
# declares no realm key outside the reconciler's write surface.
|
||||
# (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
|
||||
# reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
|
||||
# could emit a key the reconciler refuses (fail-closed forever) or the
|
||||
# reconciler could accept a key the renderer never emits (dead surface).
|
||||
# (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
|
||||
# entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
|
||||
# ServiceAccount token, and contains no dollar sign (it ships inside a
|
||||
# SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
|
||||
# rewritten before any container runs).
|
||||
# (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
|
||||
# SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
|
||||
# alias, verify, flip browserFlow in one atomic PUT), so flow-create and
|
||||
# execution-add endpoints ARE on the allow-list and their presence is not the
|
||||
# hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
|
||||
# is the rollback target); the execution- and config-id-scoped write endpoints
|
||||
# stay off the list because they carry no flow alias and so cannot be screened;
|
||||
# no required-action write reaches any alias but the one named literally; and
|
||||
# the guard the allow-list CANNOT express exists in the source and is wired
|
||||
# into the single request choke point, FAIL-CLOSED until the live binding has
|
||||
# been read. That last one is what the whole design rests on: a reconciler that
|
||||
# half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
|
||||
# this repo has had one (2026-09-06).
|
||||
# (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
|
||||
# bind-browser-flow as exactly "true" or "false", the script reads it from the
|
||||
# SAME mount, and the CronJob passes no flag that could widen it. With "false"
|
||||
# (the default) the reconciler builds and verifies but never PUTs browserFlow,
|
||||
# so an operator's rollback sticks; the selftest proves that end to end.
|
||||
# (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
|
||||
# allow-list is NOT widened by this work. Its --selftest asserts it refuses
|
||||
# /users, /groups, /roles, /authentication and the whole-realm endpoint, and
|
||||
# its guarantee that prod's real user and the media-users / platform-admins /
|
||||
# viewers groups survive every run must stay LITERALLY true.
|
||||
# (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
|
||||
# script from the ConfigMap and executes its `--selftest`, which drives the
|
||||
# real code paths against an in-process stub Keycloak that MODELS the three
|
||||
# Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
|
||||
# passwordless policy, the all-or-nothing policy rebuild that defaults every
|
||||
# omitted key, and the required-action representation replace.
|
||||
# (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
|
||||
# one at a time and requires --selftest to CATCH every one. A passing suite
|
||||
# proves nothing if it cannot fail, and "reported success while the realm was
|
||||
# broken" is the exact failure this component exists to end.
|
||||
# AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
|
||||
# `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
|
||||
# via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
|
||||
# ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
|
||||
# An unreferenced test is precisely the "looks applied and isn't" failure this
|
||||
# repo keeps hitting.
|
||||
#
|
||||
# NOT proven here (named honestly, in the style of the sibling L0 checks):
|
||||
# - that the LIVE realm's policy, enrolment action or browser flow match git. That is
|
||||
# cluster state. The reconciler's own `--audit` mode is the read-only probe for it
|
||||
# and is what an L2 test should run; no such L2 test is added by this change, so the
|
||||
# live realm is UNVERIFIED by CI.
|
||||
# - that the declared browser flow actually WORKS as a login path. Nothing offline can
|
||||
# prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
|
||||
# password form, or that a platform-admin is still asked for TOTP. Those need a real
|
||||
# browser against a real realm.
|
||||
# - that the reconciler's first real run against prod will succeed. The flow is built
|
||||
# through five admin-API endpoints whose request shapes were read from the 26.0.6
|
||||
# source but never exercised against a live Keycloak.
|
||||
# - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
|
||||
# offline; the plaintext is not, and must not be.
|
||||
# - that the CronJob's image can be pulled or its schedule fires.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
|
||||
RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
|
||||
STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
|
||||
RENDERER="ci/render-keycloak-passkey-policy.py"
|
||||
SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
|
||||
|
||||
# Honest SKIP (never green) if the thing under test is absent entirely.
|
||||
for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
|
||||
if [ ! -f "${ROOT}/${f}" ]; then
|
||||
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
done
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit of the Keycloak passkey declaration, the renderer that derives the
|
||||
reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
|
||||
manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
|
||||
is one definition of "what a Flux path renders" in the harness."""
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
|
||||
# test directory is named relative to the repo root the wrapper passes in.
|
||||
DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
|
||||
/ "keycloak-passkey-reconciler")
|
||||
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
||||
import status as st # noqa: E402
|
||||
|
||||
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
|
||||
RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
|
||||
STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
|
||||
RENDERER = "ci/render-keycloak-passkey-policy.py"
|
||||
SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
|
||||
STATE_CM_NAME = "keycloak-passkey-desired-state"
|
||||
STATE_DATA_KEY = "passkey-desired-state.json"
|
||||
BIND_KEY = "bind-browser-flow"
|
||||
SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
|
||||
ENROL_ACTION = "webauthn-register-passwordless"
|
||||
DOLLAR = chr(36)
|
||||
|
||||
failures: list[str] = []
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
if not cond:
|
||||
failures.append(msg)
|
||||
|
||||
|
||||
def load(rel):
|
||||
return list(yaml.safe_load_all((ROOT / rel).read_text()))
|
||||
|
||||
|
||||
# ================================================== (1) the desired state is DERIVED
|
||||
proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
check(proc.returncode == 0,
|
||||
f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
|
||||
f".py --check failed). The reconciler would converge the realm on a passkey policy "
|
||||
f"the realm import does not declare, and audit the login flow against a tree "
|
||||
f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
|
||||
f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
|
||||
f"domain is an unrecoverable-credential bug.\n "
|
||||
+ "\n ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
|
||||
|
||||
# The realm import must actually declare the things the renderer derives from. If it
|
||||
# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
|
||||
# renderer would otherwise be the only signal.
|
||||
imp_docs = [d for d in load(IMPORT_FILE) if d]
|
||||
imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
|
||||
check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
|
||||
realm = ((imp or {}).get("spec") or {}).get("realm") or {}
|
||||
policy_keys_in_import = sorted(k for k in realm
|
||||
if k.startswith("webAuthnPolicyPasswordless"))
|
||||
check(policy_keys_in_import,
|
||||
f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
|
||||
f"passkey policy for the reconciler to converge")
|
||||
check(realm.get("browserFlow"),
|
||||
f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
|
||||
f"BOUND changes nothing about how anyone logs in")
|
||||
|
||||
# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
|
||||
# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
|
||||
# representation as REPLACE, not merge, so declaring ONE action imports exactly that
|
||||
# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
|
||||
# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
|
||||
# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
|
||||
# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
|
||||
# NOT ADD ONE" is advice; this is the gate.
|
||||
check("requiredActions" not in realm,
|
||||
f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
|
||||
f"as REPLACE, not merge: declaring any action imports exactly those plus "
|
||||
f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
|
||||
f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
|
||||
f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
|
||||
f"realm's browser flow still demands a TOTP they could no longer enrol. The "
|
||||
f"enrolment action is already registered by default (priority 80, enabled, "
|
||||
f"defaultAction false) and the passkey reconciler converges it through the "
|
||||
f"per-alias admin API, which has no such hazard")
|
||||
|
||||
# ============================== (2)(3) the two files share one substitution pass
|
||||
state_docs = [d for d in load(STATE_FILE) if d]
|
||||
state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
|
||||
check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
|
||||
desired = {}
|
||||
if state_cm is not None:
|
||||
check(state_cm["metadata"]["name"] == STATE_CM_NAME,
|
||||
f"{STATE_FILE}: the ConfigMap is named "
|
||||
f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
|
||||
data = state_cm.get("data") or {}
|
||||
check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
|
||||
f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
|
||||
f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
|
||||
f"PASSKEY_BIND_FILE defaults would not resolve")
|
||||
# THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
|
||||
# at apply time -- failing the whole platform-config Kustomization, realm import
|
||||
# included. The renderer refuses it too; asserted here so the reason is on record.
|
||||
check(data.get(BIND_KEY) in ("true", "false"),
|
||||
f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
|
||||
f"string \"true\" or \"false\"")
|
||||
raw = data.get(STATE_DATA_KEY, "")
|
||||
try:
|
||||
desired = json.loads(raw)
|
||||
except ValueError as e:
|
||||
check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
|
||||
f"({e}) — the reconciler would fail closed on every run")
|
||||
|
||||
# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
|
||||
# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
|
||||
# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
|
||||
import_owners = set()
|
||||
state_owners = set()
|
||||
recon_owners = set()
|
||||
for env in ("dev", "prod"):
|
||||
entry = f"clusters/{env}"
|
||||
if not (ROOT / entry).is_dir():
|
||||
continue
|
||||
applied, _ = st.reconciled_files(ROOT, entry)
|
||||
check(RECON_FILE in applied,
|
||||
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
|
||||
f"healer would exist on one cluster only")
|
||||
check(STATE_FILE in applied,
|
||||
f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
|
||||
f"would fail closed on that cluster forever")
|
||||
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
|
||||
|
||||
# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
|
||||
# listed as resources of the same kustomization.yaml.
|
||||
for kpath in ROOT.glob("*/kustomization.yaml"):
|
||||
try:
|
||||
k = yaml.safe_load(kpath.read_text()) or {}
|
||||
except Exception:
|
||||
continue
|
||||
res = [str(r) for r in (k.get("resources") or [])]
|
||||
base = kpath.parent.name
|
||||
if any(r.endswith("keycloak-realm-import.yaml") for r in res):
|
||||
import_owners.add(base)
|
||||
if any(r.endswith("passkey-desired-state.yaml") for r in res):
|
||||
state_owners.add(base)
|
||||
if any(r.endswith("passkey-reconciler.yaml") for r in res):
|
||||
recon_owners.add(base)
|
||||
check(import_owners and import_owners == state_owners,
|
||||
f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
|
||||
f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
|
||||
f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
|
||||
f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
|
||||
f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
|
||||
f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
|
||||
check(recon_owners == state_owners,
|
||||
f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
|
||||
f"{sorted(state_owners)} — the reconciler could be applied without its input")
|
||||
|
||||
# (3) no substitution token the realm definition does not also use.
|
||||
import_text = (ROOT / IMPORT_FILE).read_text()
|
||||
TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
|
||||
state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
|
||||
import_tokens = set(TOKEN.findall(import_text))
|
||||
# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
|
||||
# policy uses, so a subset check is the right assertion rather than an exact match.
|
||||
check(state_tokens <= import_tokens,
|
||||
f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
|
||||
f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
|
||||
f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
|
||||
f"silently — which for an RP ID means every passkey ceremony fails")
|
||||
|
||||
# ======================================= (4) the two write surfaces are the SAME tuple
|
||||
recon_docs = [d for d in load(RECON_FILE) if d]
|
||||
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
|
||||
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
|
||||
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
|
||||
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
|
||||
|
||||
script = ""
|
||||
if cm is not None:
|
||||
data = cm.get("data") or {}
|
||||
check(len(data) == 1,
|
||||
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
|
||||
f"(found {sorted(data)}) — a second copy would be untested by this gate")
|
||||
check(SCRIPT_KEY in data,
|
||||
f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
|
||||
script = next(iter(data.values()), "")
|
||||
check(len(script.splitlines()) > 200,
|
||||
f"{RECON_FILE}: the extracted script is implausibly short "
|
||||
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
|
||||
f"selftest below would prove nothing")
|
||||
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
|
||||
# postBuild.substituteFrom, so envsubst rewrites it before any container runs.
|
||||
check(DOLLAR not in script,
|
||||
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
|
||||
f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
|
||||
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
|
||||
f"never needs one; chr(36) spells it where a literal is genuinely required")
|
||||
|
||||
|
||||
def tuple_literal(src, name):
|
||||
"""The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
|
||||
than importing it keeps this check offline and side-effect free, and means the thing
|
||||
asserted is the text that ships."""
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
except SyntaxError:
|
||||
return None
|
||||
for node in tree.body:
|
||||
if isinstance(node, ast.Assign):
|
||||
for t in node.targets:
|
||||
if isinstance(t, ast.Name) and t.id == name:
|
||||
try:
|
||||
return ast.literal_eval(node.value)
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
renderer_src = (ROOT / RENDERER).read_text()
|
||||
|
||||
|
||||
# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
|
||||
# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
|
||||
# their parts from the same vocabulary; a part present in one and not the other simply
|
||||
# contributes nothing, which is why the comparison below is of the resolved SETS.
|
||||
def resolve_policy(src):
|
||||
parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
|
||||
"POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
|
||||
"POLICY_BOOL_FIELDS")
|
||||
out = []
|
||||
for p in parts:
|
||||
v = tuple_literal(src, p)
|
||||
if v:
|
||||
out.extend(v)
|
||||
return sorted(set(out))
|
||||
|
||||
|
||||
recon_keys = resolve_policy(script)
|
||||
rend_keys = resolve_policy(renderer_src)
|
||||
check(recon_keys and recon_keys == rend_keys,
|
||||
f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
|
||||
f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
|
||||
f"every run fails closed forever; if the reconciler accepts a key the renderer "
|
||||
f"never emits, that is dead write surface on the identity provider")
|
||||
# Every exported key must actually be a passwordless key. The two-factor
|
||||
# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
|
||||
# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
|
||||
# with no effect.
|
||||
check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
|
||||
f"the reconciler's writable keys include a non-passwordless field: "
|
||||
f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
|
||||
f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
|
||||
f"writing it is a silent no-op")
|
||||
# And the desired state must declare only those.
|
||||
if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
|
||||
extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
|
||||
check(not extra,
|
||||
f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
|
||||
f"surface — it would fail closed on every run")
|
||||
# The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
|
||||
check(desired["webAuthnPolicyPasswordless"].get(
|
||||
"webAuthnPolicyPasswordlessRpEntityName"),
|
||||
f"{STATE_FILE} carries no non-empty "
|
||||
f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
|
||||
f"passwordless policy only when that field is set "
|
||||
f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
|
||||
f"accepted with 204 and change nothing")
|
||||
|
||||
# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
|
||||
# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
|
||||
# import's stated "additive and optional" decision. It is one boolean, so it is gated.
|
||||
ra = desired.get("requiredAction") or {}
|
||||
check(ra.get("alias") == ENROL_ACTION,
|
||||
f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
|
||||
f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
|
||||
f"any other value is unreachable and the pass would fail closed")
|
||||
check(ra.get("enabled") is True,
|
||||
f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
|
||||
f"impossible while the login flow offers passkeys")
|
||||
check(ra.get("defaultAction") is False,
|
||||
f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
|
||||
f"must be false. True forces EVERY user to enrol a passkey at next login — this "
|
||||
f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
|
||||
f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
|
||||
f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
|
||||
|
||||
# ========================================= (5) the reconciler manifest is safe + wired
|
||||
if cj is not None:
|
||||
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
|
||||
ctr = pod["containers"][0]
|
||||
vols = {v["name"]: v for v in pod.get("volumes", [])}
|
||||
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
|
||||
env = {e["name"]: e for e in ctr.get("env", [])}
|
||||
|
||||
# NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
|
||||
# narrower than reading the KeycloakRealmImport CR through the API: no Role, no
|
||||
# RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
|
||||
check(pod.get("automountServiceAccountToken") is False,
|
||||
f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
|
||||
f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
|
||||
f"ServiceAccount token is pure blast radius (R-SEC-02)")
|
||||
check("serviceAccountName" not in pod,
|
||||
f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
|
||||
|
||||
cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
|
||||
names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
|
||||
check(STATE_CM_NAME in names,
|
||||
f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
|
||||
f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
|
||||
check(cm["metadata"]["name"] in names,
|
||||
f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
|
||||
# NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
|
||||
# secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
|
||||
check(not [v for v in vols.values() if "secret" in v],
|
||||
f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
|
||||
f"credential is referenced by secretKeyRef only; it must never be projected "
|
||||
f"as a file here")
|
||||
|
||||
# The mount paths must be the ones the code DEFAULTS to, or the container starts and
|
||||
# fails closed forever while every manifest looks correct.
|
||||
state_vol = next((n for n, v in cm_vols.items()
|
||||
if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
|
||||
if state_vol:
|
||||
mp = (mounts.get(state_vol) or {}).get("mountPath")
|
||||
check(mp == "/etc/keycloak-passkey",
|
||||
f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
|
||||
f"script's PASSKEY_STATE_FILE default is "
|
||||
f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
|
||||
check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
|
||||
f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
|
||||
f"the mount path — they are a pair and must move together")
|
||||
check((mounts.get(state_vol) or {}).get("readOnly") is True,
|
||||
f"{RECON_FILE}: the desired-state mount must be readOnly")
|
||||
check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
|
||||
f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
|
||||
f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
|
||||
f"(safe, but git's \"true\" would then silently never take effect)")
|
||||
check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
|
||||
f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
|
||||
f"keys may not be projected")
|
||||
|
||||
# The bind decision belongs to git. A container arg could only NARROW it today
|
||||
# (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
|
||||
# behaviour must be readable from passkey-desired-state.yaml alone.
|
||||
cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
|
||||
check(not [x for x in cmd if x.startswith("--")],
|
||||
f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
|
||||
f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
|
||||
# Retries burn the bounded candidate-alias budget on a deterministic failure.
|
||||
check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
|
||||
f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
|
||||
f"within a tick, and each retried build consumes another candidate alias")
|
||||
|
||||
# The admin credential, referenced BY NAME. Never decrypted, never a literal.
|
||||
for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
|
||||
e = env.get(key) or {}
|
||||
ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
|
||||
check(ref.get("name") == "keycloak-admin-credentials",
|
||||
f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
|
||||
f"Secret by secretKeyRef, got {e!r}")
|
||||
check("value" not in e,
|
||||
f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
|
||||
f"manifest is a plaintext secret in git")
|
||||
check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
|
||||
f"{RECON_FILE}: KEYCLOAK_REALM is "
|
||||
f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
|
||||
f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
|
||||
f"wrong realm, or none")
|
||||
|
||||
check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
|
||||
f"{RECON_FILE}: readOnlyRootFilesystem must be true")
|
||||
check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
|
||||
f"{RECON_FILE}: allowPrivilegeEscalation must be false")
|
||||
check(pod["securityContext"].get("runAsNonRoot") is True,
|
||||
f"{RECON_FILE}: runAsNonRoot must be true")
|
||||
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
|
||||
f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
|
||||
f"would double the write attempts against the IdP during an outage recovery")
|
||||
check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
|
||||
f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
|
||||
|
||||
# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
|
||||
#
|
||||
# This component converges the browser flow BLUE/GREEN: it builds the target structure
|
||||
# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
|
||||
# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
|
||||
# first design of this check the mere presence of a flow write is no longer the thing to
|
||||
# forbid. What must hold instead is narrower and stronger:
|
||||
#
|
||||
# (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
|
||||
# (b) the single-execution and config-edit endpoints stay off the list — requirements
|
||||
# are set through the flow-scoped endpoint, which the guard in (d) screens;
|
||||
# (c) no required-action write may reach any alias but the one named LITERALLY;
|
||||
# (d) and the guard the allow-list CANNOT express must exist in the source: a
|
||||
# method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
|
||||
# flow's subtree, FAIL-CLOSED until that binding has been read.
|
||||
#
|
||||
# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
|
||||
# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
|
||||
# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
|
||||
# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
|
||||
# change (2026-09-06).
|
||||
allow = tuple_literal(script, "_ALLOWED") or ()
|
||||
check(bool(allow),
|
||||
f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
|
||||
f"script. This check's central assertions about the write surface cannot be made, "
|
||||
f"so it must FAIL rather than pass silently")
|
||||
if allow:
|
||||
check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
|
||||
f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
|
||||
f"allow-list cannot distinguish reading a flow from writing one, which this "
|
||||
f"component's whole safety argument rests on")
|
||||
writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
|
||||
# (a) nothing is ever deleted.
|
||||
check(not [(m, p) for m, p in allow if m == "DELETE"],
|
||||
f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
|
||||
f"does requires deleting anything: a superseded flow is left unbound as the "
|
||||
f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
|
||||
# (b) the endpoints that would let it edit live objects in place.
|
||||
banned = [(m, p) for m, p in writes
|
||||
if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
|
||||
r"\[\^/\]\+", p)
|
||||
or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
|
||||
p)
|
||||
or p.endswith("/authentication/register-required-action")]
|
||||
check(not banned,
|
||||
f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
|
||||
f"and config-edit endpoints must stay off it: requirements are set through the "
|
||||
f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
|
||||
f"therefore screenable against the bound subtree. An execution- or "
|
||||
f"config-id-scoped write is not, so it would be an unscreened hole")
|
||||
# (c) no required-action write may reach any other alias.
|
||||
ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
|
||||
check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
|
||||
f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
|
||||
f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
|
||||
f"permanent lock-out of every platform-admin, because auth-otp-form in the "
|
||||
f"browser flow still demands a TOTP that could no longer be enrolled")
|
||||
check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
|
||||
f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
|
||||
f"must be named LITERALLY so no other action is writable by construction")
|
||||
# Nothing outside the realm + authentication surface at all.
|
||||
stray = [(m, p) for m, p in allow
|
||||
if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
|
||||
check(not stray,
|
||||
f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
|
||||
f"be unreachable (prod's real user and the media-users / platform-admins / "
|
||||
f"viewers groups must survive every run); clients are the SIBLING "
|
||||
f"reconciler's surface and the two must not overlap")
|
||||
|
||||
# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
|
||||
# Asserted against the SOURCE, because this is the property the blue/green design rests
|
||||
# on and a comment claiming it is worth nothing. Three things must all be true.
|
||||
check(re.search(r"^def _screen_write\(", script, re.M) is not None,
|
||||
f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
|
||||
f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
|
||||
f"one thing the allow-list cannot express, because its write entries are "
|
||||
f"alias-parameterised. Without it this component can mutate the live login path")
|
||||
# It must be CALLED from the single choke point, not merely defined.
|
||||
check(re.search(r"_screen_write\(method, path", script) is not None,
|
||||
f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
|
||||
f"request's method and path. A guard that is not on the request path is decoration")
|
||||
# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
|
||||
# the top of every run must put them back there.
|
||||
reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
|
||||
check(reset_fn is not None
|
||||
and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
|
||||
and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
|
||||
f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
|
||||
f"must start each run UNDETERMINED, because while they are, every flow write is "
|
||||
f"refused — that is the fail-closed half of the guard, and it is what stops a "
|
||||
f"build running before the live binding has been read")
|
||||
check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
|
||||
f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
|
||||
f"protected sets are not reset per run and a previous run's binding could be "
|
||||
f"trusted for this one")
|
||||
# The create-flow endpoint carries NO alias in its path, so it must be screened on its
|
||||
# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
|
||||
# future edit cannot silently undo it.)
|
||||
check("_FLOW_CREATE_PATH" in script,
|
||||
f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
|
||||
f"That endpoint carries no alias in its path — the alias is in the request BODY — "
|
||||
f"so without a body-based screen it bypasses the protected-subtree guard "
|
||||
f"altogether")
|
||||
# And the one function allowed to change the live binding must be distinct from the
|
||||
# policy write, which must refuse to.
|
||||
# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
|
||||
# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
|
||||
# the top-level flow alone while every selftest passed against a stub that returned it.
|
||||
check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*? name = ex\.get\("displayName"\)',
|
||||
script, re.M) is not None,
|
||||
f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
|
||||
f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
|
||||
f"None and the protected subtree silently shrinks to the top-level flow")
|
||||
# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
|
||||
check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
|
||||
f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
|
||||
f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
|
||||
f"and ignores the path alias, so a path-only screen lets it write the bound flow")
|
||||
check("def declared_bind(" in script and "BIND_FILE" in script,
|
||||
f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
|
||||
check("def bind_browser_flow(" in script,
|
||||
f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
|
||||
f"one named, auditable operation, not an incidental field in some other write")
|
||||
check(re.search(r"never rebinds the browser flow", script) is not None,
|
||||
f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
|
||||
f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
|
||||
f"can rebind the login path by accident — to an UNVERIFIED flow")
|
||||
|
||||
# ======================================== (7) the SIBLING reconciler is NOT widened
|
||||
if (ROOT / SIBLING_FILE).is_file():
|
||||
sib_docs = [d for d in load(SIBLING_FILE) if d]
|
||||
sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
|
||||
sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
|
||||
sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
|
||||
check(bool(sib_allow),
|
||||
f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
|
||||
f"prove the sibling was not widened")
|
||||
if sib_allow:
|
||||
# The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
|
||||
# whole-realm endpoint and the /authentication tree; assert the same here so a
|
||||
# future edit that "just adds the realm endpoint there instead" is caught.
|
||||
bad = [p for p in sib_allow
|
||||
if isinstance(p, str)
|
||||
and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
|
||||
or "authentication" in p or "/users" in p or "/groups" in p
|
||||
or "/roles" in p)]
|
||||
check(not bad,
|
||||
f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
|
||||
f"stated guarantee — prod's real user and the media-users / "
|
||||
f"platform-admins / viewers groups survive every run — is expressed as "
|
||||
f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
|
||||
f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
|
||||
f"The passkey work is a SIBLING program with its own narrow list precisely "
|
||||
f"so that guarantee stays literally true. Revert the widening")
|
||||
|
||||
# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
|
||||
if script:
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "reconcile.py"
|
||||
p.write_text(script)
|
||||
proc = subprocess.run([sys.executable, str(p), "--selftest"],
|
||||
capture_output=True, text=True, timeout=300)
|
||||
tail = (proc.stdout + proc.stderr).strip().splitlines()
|
||||
check(proc.returncode == 0,
|
||||
"the passkey reconciler's --selftest FAILED (compare-then-write / "
|
||||
"idempotence / strict read-modify-write / the RpEntityName gate / "
|
||||
"flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
|
||||
" " + "\n ".join(tail[-14:]))
|
||||
if proc.returncode == 0:
|
||||
print(" passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
|
||||
|
||||
# ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
|
||||
# A passing suite proves nothing unless it can be shown to FAIL. mutations.py
|
||||
# injects plausible regressions one at a time -- a partial policy PUT, a
|
||||
# rebind of browserFlow, a widened allow-list that makes the flow writable, a
|
||||
# fail-open on an unreadable input -- and requires --selftest to catch every
|
||||
# one. This matters more than usual here: the whole incident class is a
|
||||
# component that REPORTED SUCCESS while the realm was broken, and a green test
|
||||
# that cannot fail is the same lie one level up.
|
||||
mut = DIR_PATH / "mutations.py"
|
||||
if mut.is_file():
|
||||
mproc = subprocess.run([sys.executable, str(mut), str(p)],
|
||||
capture_output=True, text=True, timeout=900)
|
||||
mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
|
||||
check(mproc.returncode == 0,
|
||||
"the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
|
||||
"one injected regression survived it, so the suite does not prove "
|
||||
"what it claims:\n " + "\n ".join(mtail[-16:]))
|
||||
if mproc.returncode == 0:
|
||||
print(" passkey reconciler mutations: "
|
||||
+ (mtail[-1] if mtail else "(no output)"))
|
||||
else:
|
||||
check(False,
|
||||
"conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
|
||||
"is missing — nothing proves the --selftest can fail, and an "
|
||||
"unfalsifiable proof is the failure mode this component exists to end")
|
||||
|
||||
# ======================================================= (9) this check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
|
||||
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
|
||||
SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
|
||||
is not None,
|
||||
"keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
|
||||
"only when a change happens to attribute one of the requirements it is labelled "
|
||||
"with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
|
||||
"change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
|
||||
"prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
|
||||
"stage is a purely requirement-scoped run again and the scope annotation is inert "
|
||||
"decoration — this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
|
||||
and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
|
||||
"ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
|
||||
"import or the reconciler would attribute NO requirement and the keycloak tests "
|
||||
"would be filtered out of the merge gate's touched subset")
|
||||
|
||||
# =============================================================================
|
||||
if failures:
|
||||
print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
|
||||
for f in failures:
|
||||
print(" FAIL %s" % f, file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print("keycloak-passkey-reconciler: all assertions passed "
|
||||
"(desired state derived from the realm import, one substitution pass, write "
|
||||
"surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
|
||||
"deletable, sibling allow-list unwidened, selftest green and falsifiable)")
|
||||
PY
|
||||
@@ -0,0 +1,456 @@
|
||||
#!/usr/bin/env python3
|
||||
# =============================================================================
|
||||
# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# ---------------
|
||||
# A selftest that passes proves nothing on its own: a selftest asserting `True ==
|
||||
# True` also passes. This reconciler exists because a one-shot realm import
|
||||
# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
|
||||
# with the whole passkey block inert, measured on prod 2026-10-18), and the same
|
||||
# trap applies one level up -- a green test suite that cannot fail is the same lie
|
||||
# in a different place.
|
||||
#
|
||||
# It matters more here than for the sibling client reconciler, because this program
|
||||
# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
|
||||
# untested safety property on that surface is not a documentation gap, it is an
|
||||
# outage waiting for a schedule tick.
|
||||
#
|
||||
# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
|
||||
# a time, and requires that --selftest FAILS for every one. A mutation that
|
||||
# SURVIVES means the suite does not actually check that property, and the check
|
||||
# fails, naming it.
|
||||
#
|
||||
# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
|
||||
# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
|
||||
# write -- never mutating the flow users are logging in through. Each of those
|
||||
# properties is enforced in code; these mutations are the proof that the enforcement
|
||||
# is load-bearing rather than commentary. Several model the obvious "simplification"
|
||||
# someone will propose:
|
||||
# * "just build it under the declared alias, the suffix is ugly"
|
||||
# * "bind it first, then check -- the check is just belt and braces"
|
||||
# * "the protected-alias screen is redundant, the allow-list covers it"
|
||||
# Every one of those would be a total SSO outage waiting to happen.
|
||||
#
|
||||
# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
|
||||
# The sibling mutations.py dedents by four because its anchors are copied out of the
|
||||
# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
|
||||
# which is one less transformation to get wrong. check.sh passes the extracted file,
|
||||
# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
|
||||
#
|
||||
# python3 mutations.py <path-to-extracted-reconciler.py>
|
||||
#
|
||||
# Runtime is ~60s (one subprocess per mutation, each running the full selftest
|
||||
# against an in-process stub Keycloak; no network, no cluster).
|
||||
# =============================================================================
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# (name, the mistake being modelled, anchor, replacement)
|
||||
MUTATIONS: list[tuple[str, str, str, str]] = [
|
||||
# ---------------------------------------------------------------- blue/green safety
|
||||
("build INTO the currently bound flow",
|
||||
"'the suffix is ugly, just build under the declared alias' -- every "
|
||||
"create-execution and set-requirement call would then land on the flow users are "
|
||||
"logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
|
||||
"every SSO app and kubectl. This is the single regression the whole design exists "
|
||||
"to make impossible",
|
||||
" def mapped(alias):\n"
|
||||
" return \"%s-%s\" % (alias, suffix)",
|
||||
" def mapped(alias):\n return alias"),
|
||||
|
||||
("the protected-subtree screen is never called",
|
||||
"'the allow-list already covers it' -- it does not: the allow-list's write entries "
|
||||
"are alias-parameterised, so only this screen can say 'but not the flow that is "
|
||||
"bound'. Without it the reconciler may write anywhere in the authentication tree",
|
||||
" _screen_write(method, path, body if isinstance(body, dict) else None)",
|
||||
" pass"),
|
||||
|
||||
("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
|
||||
"reset them to frozenset() so the code reads more simply -- it silently removes "
|
||||
"the fail-closed property, so a build that ran before the live binding had been "
|
||||
"read would be PERMITTED, and could write straight into the live flow. (The "
|
||||
"module-level initial value is deliberately NOT the anchor here: run() resets "
|
||||
"these at the start of every run, so the initialiser is unreachable and mutating "
|
||||
"it would prove nothing.)",
|
||||
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
|
||||
" _PROTECTED_FLOW_ALIASES = None\n"
|
||||
" _PROTECTED_EXECUTION_IDS = None",
|
||||
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
|
||||
" _PROTECTED_FLOW_ALIASES = frozenset()\n"
|
||||
" _PROTECTED_EXECUTION_IDS = frozenset()"),
|
||||
|
||||
("bind BEFORE verifying the build",
|
||||
"'bind it first, the readback is just belt and braces' -- it is not: the readback "
|
||||
"is what catches a build that was accepted but came out wrong, including the "
|
||||
"conditional-OTP config landing under the wrong key. Binding first makes that the "
|
||||
"live login path before anyone notices",
|
||||
" verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
|
||||
" \"the newly built flow %r\" % target)",
|
||||
" verify = []"),
|
||||
|
||||
("a failed build is reported as SUCCESS",
|
||||
"swallow the build error and return a converged result -- the realm is then left "
|
||||
"on the old flow (which blue/green makes safe) but the run says it converged, so "
|
||||
"nobody ever learns the passkey flow was never built. That is the one-shot "
|
||||
"import's fail-open, reintroduced. NOTE: simply removing the early return here "
|
||||
"is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
|
||||
"is defence in depth working as intended -- so the mutation targets the reporting",
|
||||
" problems.append(\n"
|
||||
" \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
|
||||
" \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
|
||||
" \"place, unbound and harmless\" % (candidate, e))\n"
|
||||
" return 0, problems",
|
||||
" return 1, []"),
|
||||
|
||||
("an existing candidate flow is reused WITHOUT verifying it",
|
||||
"'if the alias exists we already built it' -- a half-built leftover from a crashed "
|
||||
"run lives under exactly that alias, so this binds a partially constructed login "
|
||||
"path",
|
||||
" if cand_shape is not None and not compare_shape(\n"
|
||||
" cand_shape, expected, base, realm, token, configs,\n"
|
||||
" \"the existing flow %r\" % candidate):",
|
||||
" if cand_shape is not None:"),
|
||||
|
||||
# ------------------------------------- 2026-10-29 review findings, each made falsifiable
|
||||
("sub-flows identified by `alias` again",
|
||||
"read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
|
||||
"recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
|
||||
"sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
|
||||
"a write into browser-mfa-forms would NOT be refused",
|
||||
" name = ex.get(\"displayName\")\n"
|
||||
" return name if isinstance(name, str) and name else None",
|
||||
" name = ex.get(\"alias\")\n"
|
||||
" return name if isinstance(name, str) and name else None"),
|
||||
|
||||
("an unnameable sub-flow is tolerated",
|
||||
"carry on when a sub-flow row has no displayName -- the protected subtree is then "
|
||||
"SMALLER than the live tree, which is the fail-open the guard exists to prevent",
|
||||
" if name is None:\n"
|
||||
" return None, None",
|
||||
" if name is None:\n"
|
||||
" name = \"\""),
|
||||
|
||||
("the requirement PUT is screened on its path only",
|
||||
"trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
|
||||
"body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
|
||||
"the path can carry a BOUND execution's id in the body",
|
||||
" targets.append((\"execution\", (body or {}).get(\"id\")))",
|
||||
" pass"),
|
||||
|
||||
("execution-addressed writes are not held to the build tree",
|
||||
"'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
|
||||
"outside it too, so a stray id would edit an unrelated flow",
|
||||
" if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
|
||||
" or target not in _BUILD_EXECUTION_IDS):",
|
||||
" if False:"),
|
||||
|
||||
("the requirement PUT may carry any key",
|
||||
"let the body carry flowId / displayName -- the same handler then RENAMES whatever "
|
||||
"flow flowId names, bound or not",
|
||||
" if extra:\n"
|
||||
" raise Refused(",
|
||||
" if False:\n"
|
||||
" raise Refused("),
|
||||
|
||||
("git's bind switch is ignored",
|
||||
"bind whenever verification passes, as the first version did -- the first live use of "
|
||||
"an untested login flow becomes a schedule tick, and an operator's rollback is "
|
||||
"undone within 15 minutes, forever",
|
||||
" bind = git_bind and not no_bind",
|
||||
" bind = not no_bind"),
|
||||
|
||||
("bind=false still binds inside the flow pass",
|
||||
"keep the flag but forget to honour it where the bind happens",
|
||||
" if not bind:\n"
|
||||
" # THE BIND SWITCH IS OFF.",
|
||||
" if False:\n"
|
||||
" # THE BIND SWITCH IS OFF."),
|
||||
|
||||
("an absent bind switch means bind",
|
||||
"default to true when the key is missing -- a ConfigMap predating the switch, or a "
|
||||
"deleted line, would silently put an untested flow live",
|
||||
" except FileNotFoundError:\n"
|
||||
" return False",
|
||||
" except FileNotFoundError:\n"
|
||||
" return True"),
|
||||
|
||||
("an invalid bind switch is guessed at",
|
||||
"read anything that is not \"true\" as false -- a typo then passes silently instead of "
|
||||
"stopping the run, and the operator believes the flow is live",
|
||||
" if raw == \"false\":\n"
|
||||
" return False\n"
|
||||
" raise ValueError(",
|
||||
" return False\n"
|
||||
" raise ValueError("),
|
||||
|
||||
("bind=false rebuilds beside a hand-edited bound flow",
|
||||
"treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
|
||||
"off this builds an -r2 next to the operator's hotfix on every tick",
|
||||
" if candidate in have and candidate == bound and not bind:",
|
||||
" if False:"),
|
||||
|
||||
# ------------------------------------------------------------------- idempotence
|
||||
("sub-flow aliases are compared as names",
|
||||
"compare the sub-flow alias like any other field -- the flows this program builds "
|
||||
"carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
|
||||
"converged: it would be rebuilt and the login path reflipped every fifteen "
|
||||
"minutes, forever",
|
||||
" out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
|
||||
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
|
||||
" out.append((level, ex[\"kind\"], ex[\"name\"],\n"
|
||||
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
|
||||
|
||||
("the converged short-circuit is removed",
|
||||
"drop the 'bound flow already matches git' early return -- every run rebuilds and "
|
||||
"reflips, which is a permanent write loop against the IdP and churns the realm's "
|
||||
"login path on a schedule",
|
||||
" if not drift:\n"
|
||||
" log(\"flow ok (the bound flow %r matches git: %d execution(s), no \"\n"
|
||||
" \"write)\" % (bound, len(expected)))\n"
|
||||
" return 0, problems",
|
||||
" if False:\n return 0, problems"),
|
||||
|
||||
("the execution priority is dropped from the requirement write",
|
||||
"omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
|
||||
"missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
|
||||
"the execution order while only a requirement change was intended. The passkey "
|
||||
"branch would end up in the wrong place in the tree",
|
||||
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
|
||||
" \"priority\": ex.get(\"priority\") or 0})",
|
||||
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
|
||||
|
||||
("the alias budget is unbounded",
|
||||
"loop forever looking for a free alias -- a persistently failing build then creates "
|
||||
"unbounded flow sprawl in the realm, and nothing here can delete any of it",
|
||||
" for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
|
||||
" for attempt in range(1, 10000):"),
|
||||
|
||||
("fail-open when the bound flow cannot be read",
|
||||
"carry on when the live flow's executions are unreadable -- the program then cannot "
|
||||
"know which flows are live, so it cannot prove a build would miss them, and it "
|
||||
"would build with the protected sets still undetermined",
|
||||
" return 0, [\n"
|
||||
" \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
|
||||
" \"be read, so this program cannot establish which flows are live. Flow \"\n"
|
||||
" \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
|
||||
" % bound]",
|
||||
" _set_protected(set(), set())\n bound_shape, bound_raw = None, None"),
|
||||
|
||||
# ------------------------------------------------------- the conditional-OTP regression
|
||||
("authenticator config contents no longer compared",
|
||||
"check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
|
||||
"regression: the original bug wrote the role under "
|
||||
"`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
|
||||
"to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
|
||||
"and every platform-admin got a 403 from every SSO-gated app",
|
||||
" if str(have_cfg.get(key)) != str(want_cfg[key]):",
|
||||
" if False:"),
|
||||
|
||||
("undeclared config keys are ignored",
|
||||
"stop reporting keys the realm holds and git does not -- the stale "
|
||||
"`condition.evaluated.user.role` key is the signature of the original null-role "
|
||||
"bug, and a flow carrying both keys would look correct",
|
||||
" extra = sorted(k for k in have_cfg if k not in want_cfg)",
|
||||
" extra = []"),
|
||||
|
||||
("flow structure divergence detected but not reported",
|
||||
"leave the per-step comparison out -- the realm's login flow could be the old "
|
||||
"pre-passkey shape forever and every run would report the realm converged, which "
|
||||
"is the exact fail-open of the one-shot import this program replaces",
|
||||
" if observed[idx][:4] != expected[idx][:4]:",
|
||||
" if False:"),
|
||||
|
||||
("the browserFlow BINDING is no longer compared",
|
||||
"stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
|
||||
"passkey flow that nothing logs in through changes nothing at all",
|
||||
" if bound_shape is not None:\n"
|
||||
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
|
||||
" if False:\n"
|
||||
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
|
||||
|
||||
# ------------------------------------------------------------- realm / policy writes
|
||||
("partial policy PUT instead of read-modify-write",
|
||||
"send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
|
||||
"passwordless policy from the representation and substitutes its own DEFAULT for "
|
||||
"every key the body omits, so a partial PUT silently resets everything it did not "
|
||||
"mention while returning 204",
|
||||
" body = dict(realm_rep)\n"
|
||||
" for key in drift:\n"
|
||||
" body[key] = wanted[key]",
|
||||
" body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
|
||||
" BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
|
||||
" for key in drift:\n"
|
||||
" body[key] = wanted[key]"),
|
||||
|
||||
("the policy pass rebinds browserFlow as a side effect",
|
||||
"'while we are here, let the policy write fix the flow binding too' -- the realm "
|
||||
"PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
|
||||
"rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
|
||||
"the passkey policy",
|
||||
" if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
|
||||
" raise RuntimeError(",
|
||||
" body[BROWSER_FLOW] = \"browser\"\n"
|
||||
" if False:\n"
|
||||
" raise RuntimeError("),
|
||||
|
||||
("the RpEntityName gate removed from desired-state validation",
|
||||
"accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
|
||||
"passwordless policy ONLY when that field is non-empty, so every realm PUT would "
|
||||
"be accepted with 204 and change nothing: reported success, zero convergence",
|
||||
" if not policy.get(RP_ENTITY_NAME):\n"
|
||||
" raise ValueError(",
|
||||
" if False:\n raise ValueError("),
|
||||
|
||||
("the RpEntityName gate removed from the write path",
|
||||
"drop the second, defence-in-depth guard -- it is unreachable while the "
|
||||
"desired-state validation holds, which is exactly why deleting it would otherwise "
|
||||
"be an invisible regression",
|
||||
" if not body.get(RP_ENTITY_NAME):\n"
|
||||
" raise RuntimeError(",
|
||||
" if False:\n raise RuntimeError("),
|
||||
|
||||
("no readback verification on the policy write",
|
||||
"trust the 204 -- a server that accepts the PUT and does not apply it is reported "
|
||||
"as REPAIRED, which is 'reports success while the realm never converged' again, "
|
||||
"and it is precisely what the RpEntityName gate does when it is missed",
|
||||
" still = policy_drift(after, wanted)",
|
||||
" still = {}"),
|
||||
|
||||
("AAGUID comparison made order-sensitive",
|
||||
"compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
|
||||
"is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
|
||||
"realm on every run forever while every run still reports success",
|
||||
" if stored != sorted(declared_value):\n"
|
||||
" drift[key] = (stored, sorted(declared_value))",
|
||||
" if stored != declared_value:\n"
|
||||
" drift[key] = (stored, declared_value)"),
|
||||
|
||||
("signature-algorithm order treated as insignificant",
|
||||
"sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
|
||||
"to the authenticator, so ES256-then-RS256 is not the same declaration as "
|
||||
"RS256-then-ES256, and a reorder in the realm would never be corrected",
|
||||
" if stored != list(declared_value):",
|
||||
" if sorted(stored) != sorted(declared_value):"),
|
||||
|
||||
("unmanaged policy keys forced to a value",
|
||||
"iterate every writable key instead of only the declared ones -- a "
|
||||
"webAuthnPolicyPasswordless field the realm holds and git does not declare would "
|
||||
"be overwritten, turning 'not declared' into 'must equal my default'",
|
||||
" for key in sorted(wanted):\n declared_value = wanted[key]",
|
||||
" for key in POLICY_FIELDS:\n declared_value = wanted.get(key, \"\")"),
|
||||
|
||||
# ------------------------------------------------------------------ required actions
|
||||
("required-action PUT sends only the two drifted fields",
|
||||
"a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
|
||||
"the representation, so name, priority and config are all blanked: the enrolment "
|
||||
"action keeps working but loses its priority and display name",
|
||||
" body = dict(rep)\n for key in drift:\n body[key] = wanted[key]",
|
||||
" body = {}\n for key in drift:\n body[key] = wanted[key]"),
|
||||
|
||||
("the guarded default required actions are no longer audited",
|
||||
"stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
|
||||
"exist -- their disappearance is the signature of a requiredActions block in the "
|
||||
"realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
|
||||
"lock-out of every platform-admin",
|
||||
" missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
|
||||
" missing = []"),
|
||||
|
||||
# ----------------------------------------------------------------- inputs / coupling
|
||||
("the three passes are coupled",
|
||||
"abort the run at the first failing pass -- a policy write that fails for an "
|
||||
"unrelated reason would then suppress the browser-flow pass, which is the most "
|
||||
"safety-critical thing here",
|
||||
" problems.extend(found)\n if found:\n rc = 1",
|
||||
" problems.extend(found)\n if found:\n rc = 1\n"
|
||||
" break"),
|
||||
|
||||
("policy-key validation removed from the desired state",
|
||||
"accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
|
||||
"writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
|
||||
"bruteForceProtected or any other top-level realm field on the live IdP",
|
||||
" if key not in POLICY_FIELDS:",
|
||||
" if False:"),
|
||||
|
||||
("fail-open on an unreadable desired state",
|
||||
"treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
|
||||
"reconciling nothing, which is precisely the fail-open of the realm import this "
|
||||
"whole component exists to replace",
|
||||
" if not isinstance(raw, dict) or not raw:\n"
|
||||
" raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
|
||||
" if not isinstance(raw, dict) or not raw:\n return {}"),
|
||||
|
||||
("an undeclared authenticatorConfig reference is tolerated",
|
||||
"stop validating that every configAlias a flow references is actually declared -- "
|
||||
"the build then fails at the config-binding step, AFTER flows and executions have "
|
||||
"already been created, leaving leftovers on every run",
|
||||
" if ca and ca not in configs:\n"
|
||||
" raise ValueError(",
|
||||
" if False:\n raise ValueError("),
|
||||
|
||||
("the flow-cycle guard is removed",
|
||||
"drop the `seen` check -- a cyclic flow declaration then recurses until the "
|
||||
"interpreter dies, in a CronJob that retries",
|
||||
" if alias in seen:\n"
|
||||
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
|
||||
" if False:\n"
|
||||
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) != 2:
|
||||
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
|
||||
return 2
|
||||
src = Path(sys.argv[1]).read_text()
|
||||
|
||||
problems: list[str] = []
|
||||
killed = 0
|
||||
for name, why, old, new in MUTATIONS:
|
||||
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
|
||||
# moved and this mutation silently stopped proving anything.
|
||||
n = src.count(old)
|
||||
if n != 1:
|
||||
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
|
||||
"code moved and this mutation no longer proves anything. "
|
||||
"Re-anchor it against the current source." % (name, n))
|
||||
continue
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
|
||||
fh.write(src.replace(old, new, 1))
|
||||
path = fh.name
|
||||
try:
|
||||
proc = subprocess.run([sys.executable, path, "--selftest"],
|
||||
capture_output=True, text=True, timeout=300)
|
||||
finally:
|
||||
Path(path).unlink(missing_ok=True)
|
||||
if proc.returncode == 0:
|
||||
problems.append("%s SURVIVED -- --selftest still passed with this regression "
|
||||
"applied, so nothing in the suite actually checks it. The "
|
||||
"mistake it models: %s" % (name, why))
|
||||
continue
|
||||
killed += 1
|
||||
out = proc.stdout + proc.stderr
|
||||
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
|
||||
how = ("%s assertion(s) caught it" % m.group(1) if m
|
||||
else "the selftest aborted (caught, less cleanly)")
|
||||
print(" killed %-54s %s" % (name, how))
|
||||
|
||||
print()
|
||||
if problems:
|
||||
print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
|
||||
% (len(problems), len(MUTATIONS)), file=sys.stderr)
|
||||
for p in problems:
|
||||
print(" FAIL %s" % p, file=sys.stderr)
|
||||
return 1
|
||||
print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
|
||||
"(the suite is falsifiable, not decorative)" % killed)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,334 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: feat/keycloak-passkey-reconciler
|
||||
title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
|
||||
PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
|
||||
when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
|
||||
`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
|
||||
`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
|
||||
exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
|
||||
new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
|
||||
`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
|
||||
of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
|
||||
here.
|
||||
|
||||
**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
|
||||
flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
|
||||
A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
|
||||
so the operator will never re-run it and every realm-level field or flow declared after the first
|
||||
import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
|
||||
changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
|
||||
2026-09-06 client-secret incident, third field over.
|
||||
|
||||
**What now makes it converge.** A **sibling** reconciler, modelled on
|
||||
`platform/keycloak/client-secret-reconciler.yaml`:
|
||||
|
||||
- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
|
||||
signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
|
||||
`secretKeyRef` only. Three passes, **all of which converge**:
|
||||
1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
|
||||
changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
|
||||
2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
|
||||
**literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
|
||||
3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
|
||||
alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
|
||||
- `ci/render-keycloak-passkey-policy.py` + the **generated**
|
||||
`platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
|
||||
import, `--check`ed by the merge gate.
|
||||
- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
|
||||
(`scope: whole-repo`), `mutations.py`.
|
||||
- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
|
||||
realm import.
|
||||
- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
|
||||
live realm" section rewritten: all three declared things now converge, with the blue/green
|
||||
mechanics, the rollback command and the `--no-bind` fallback written out.
|
||||
|
||||
### The flow design was reconsidered, and the first version of it was wrong
|
||||
|
||||
**This is worth recording because the superseded reasoning was not silly, it was just wrongly
|
||||
scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
|
||||
report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
|
||||
action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
|
||||
create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
|
||||
run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
|
||||
*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
|
||||
conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
|
||||
platform-admin.
|
||||
|
||||
**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
|
||||
*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
|
||||
audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
|
||||
forces. So:
|
||||
|
||||
- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
|
||||
hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
|
||||
would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
|
||||
and **zero user-visible effect**. The untransacted-calls problem dissolves.
|
||||
- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
|
||||
requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
|
||||
regression the operator feared is now caught *before* it can bite rather than after.
|
||||
- The flip is **one atomic PUT** of the realm setting `browserFlow`.
|
||||
- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
|
||||
`DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
|
||||
|
||||
**The new safety property, enforced structurally rather than by intention.** Blue/green needs
|
||||
flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
|
||||
cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
|
||||
So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
|
||||
**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
|
||||
against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
|
||||
to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
|
||||
**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
|
||||
is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
|
||||
writes nothing.
|
||||
|
||||
**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
|
||||
would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
|
||||
produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
|
||||
compared: a flow this program built carries a hash suffix, so comparing labels would make it
|
||||
permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
|
||||
half-built one is left alone and the next candidate alias is used, with the candidate count
|
||||
**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
|
||||
|
||||
**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
|
||||
recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
|
||||
pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
|
||||
at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
|
||||
reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
|
||||
annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
|
||||
bargain than the one it would buy.
|
||||
|
||||
So the record is in two places that cost nothing: the **run log** prints
|
||||
`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
|
||||
durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
|
||||
listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
|
||||
even if every Job log has rotated away. The README carries the rollback procedure.
|
||||
|
||||
**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
|
||||
(build and verify, leave the flip as a one-write human act) but it is not the default and was not
|
||||
needed.
|
||||
|
||||
**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
|
||||
`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
|
||||
ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
|
||||
**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
|
||||
false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
|
||||
`--check`, not by a second declaration being trusted.
|
||||
|
||||
**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
|
||||
memory, which caught one I had initially missed. **Three**:
|
||||
|
||||
| Path | Why it is in the policy | What I changed |
|
||||
|---|---|---|
|
||||
| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
|
||||
| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
|
||||
| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
|
||||
|
||||
**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
|
||||
`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
|
||||
|
||||
So this PR **requires an approving review from someone who is not its author**, and
|
||||
`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
|
||||
`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
|
||||
self-approval the stage exists to prevent. The operator needs to either review it or post the
|
||||
override.
|
||||
|
||||
## What I verified
|
||||
|
||||
**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
|
||||
|
||||
- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
|
||||
completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
|
||||
- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
|
||||
- `kustomization/platform-config` `.status.lastAppliedRevision` ==
|
||||
`gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
|
||||
merge. So the inert declaration is genuinely applied and genuinely inert.
|
||||
|
||||
**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
|
||||
established from source at tag `26.0.6`, not from documentation or memory:
|
||||
|
||||
- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
|
||||
information in the representation will be ignored. This will only update top-level attributes of
|
||||
the realm."*
|
||||
- It delegates to `RepresentationToModel.updateRealm` →
|
||||
`DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
|
||||
725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
|
||||
**no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
|
||||
`getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
|
||||
the endpoint, not merely ignored by convention.
|
||||
- Two findings that changed the design, both from the same read:
|
||||
- `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
|
||||
path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
|
||||
refuses the PUT if it would differ.
|
||||
- The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
|
||||
(line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
|
||||
null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
|
||||
one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
|
||||
is a strict read-modify-write of the full live representation.
|
||||
- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
|
||||
**replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
|
||||
config), and 404s when the alias is unregistered. Hence read-modify-write there too.
|
||||
- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
|
||||
`defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
|
||||
restate a default while deleting ten siblings.
|
||||
- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
|
||||
the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
|
||||
them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
|
||||
`webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
|
||||
checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
|
||||
those fields are emitted unconditionally, not only on the export path.
|
||||
|
||||
**The five flow-construction endpoints, read from 26.0.6 source before being used:**
|
||||
|
||||
- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
|
||||
alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
|
||||
body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
|
||||
caught that, not my reading.
|
||||
- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
|
||||
Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
|
||||
- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
|
||||
DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
|
||||
so **insertion order is the resulting order** — which is why the builder inserts depth-first in
|
||||
declared order and then verifies by readback.
|
||||
- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
|
||||
compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
|
||||
deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
|
||||
while only a requirement change was intended. Every requirement write therefore carries the
|
||||
execution's current priority back. There is a mutation for it and a selftest assertion on the
|
||||
resulting order.
|
||||
- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
|
||||
it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
|
||||
the old flow already uses the declared alias.
|
||||
|
||||
**Offline proofs, all green:**
|
||||
|
||||
- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
|
||||
pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
|
||||
all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
|
||||
and flow construction where everything is created DISABLED — so these are tested against the real
|
||||
semantics rather than a forgiving fake.
|
||||
- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
|
||||
and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
|
||||
that no write targeted the bound subtree, and that every live flow survives. This is the assertion
|
||||
that makes blue/green a property rather than a claim.
|
||||
- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
|
||||
requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
|
||||
`condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
|
||||
still live.
|
||||
- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
|
||||
safety argument itself (build into the bound flow; never call the screen; reset the protected sets
|
||||
to empty; bind before verify; report a failed build as success; reuse an existing flow without
|
||||
verifying it).
|
||||
- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
|
||||
mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
|
||||
undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
|
||||
all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
|
||||
**two of my proposed mutations were not real regressions**: the module-level initial value of the
|
||||
protected sets is dead code (`run()` resets them first), and removing the early return after a
|
||||
build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
|
||||
*are* load-bearing rather than pretending the originals had been caught.
|
||||
- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
|
||||
import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
|
||||
and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
|
||||
`defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
|
||||
(previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
|
||||
was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
|
||||
stay off the list (they carry no flow alias, so they could not be screened); and — the
|
||||
load-bearing one — that the **protected-subtree guard exists in the source, is wired into
|
||||
`_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
|
||||
that the policy write still refuses to rebind `browserFlow`**.
|
||||
- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
|
||||
by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
|
||||
it flattens to the exact 11-step tree the realm import declares, with
|
||||
`conditional-otp-platform-admin` bound at level 4.
|
||||
- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
|
||||
`platform-config`, so one substitution pass decides the RP ID.
|
||||
- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
|
||||
manifest.
|
||||
|
||||
**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
|
||||
key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
|
||||
carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
|
||||
`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
|
||||
old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
|
||||
building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
|
||||
the comparison and the undeclared-key report and requires the selftest to notice both. The suite
|
||||
also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
|
||||
a correct-but-unbound flow, and a cyclic declaration.
|
||||
|
||||
**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
|
||||
are recorded there rather than paraphrased here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
|
||||
Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
|
||||
forbids decrypting or printing, so I did not read the realm's current
|
||||
`webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
|
||||
therefore **do not know** how much drift the first real run will report, nor that the live flow is
|
||||
in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
|
||||
the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
|
||||
- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
|
||||
in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
|
||||
matters more for the blue/green flow builder than for anything else in the change, because it is
|
||||
**five endpoints I have never exercised**. Specifically unproven in reality:
|
||||
- That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
|
||||
send and assign priorities in insertion order. If they do not, the build produces a tree that
|
||||
fails verification — which is the safe failure (nothing is bound, login unaffected), but it
|
||||
means **the flow would never converge** and the Job would fail every 15 minutes.
|
||||
- That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
|
||||
*client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
|
||||
sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
|
||||
endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
|
||||
the atomic bind fail closed** — loudly, writing nothing.
|
||||
- The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
|
||||
`CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
|
||||
not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
|
||||
converges — rather than as an error.
|
||||
- That `authenticationConfig` on an execution-info response is the config **id** (I read it from
|
||||
the representation class, but never saw a live response).
|
||||
- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
|
||||
assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
|
||||
`getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
|
||||
likewise fails. If either silently succeeded, the recovery path would behave differently than
|
||||
modelled.
|
||||
- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
|
||||
**nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
|
||||
shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
|
||||
beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
|
||||
that could destroy a rollback target) but it is a cost, not a non-issue.
|
||||
- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
|
||||
sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
|
||||
And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
|
||||
offline subset only; it proves nothing about the live realm.**
|
||||
- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
|
||||
design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
|
||||
add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
|
||||
key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
|
||||
- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
|
||||
this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
|
||||
degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
|
||||
asked for TOTP on the password branch. The README's four-step manual check remains the only thing
|
||||
that would establish it, and it remains undone.
|
||||
- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
|
||||
of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
|
||||
is involved, with no server-side log line. The reconciler will happily converge the realm on a
|
||||
wrong RP ID; it checks that git and the realm agree, never that git is correct.
|
||||
- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
|
||||
next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
|
||||
is deliberate and documented (username screen, then password), but it is the first time it will
|
||||
have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
|
||||
that is the mitigation, and it has not been rehearsed either.
|
||||
- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
|
||||
`ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
|
||||
so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
|
||||
did not and will not post `/approve-override` on my own PR.
|
||||
- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
|
||||
`enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
|
||||
the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
|
||||
§2 is **not re-confirmed** by me.
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: feat/keycloak-passkey-reconcile
|
||||
title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
|
||||
authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
|
||||
review of it. Read that entry for the design; this one records what the review found and what
|
||||
changed.
|
||||
|
||||
**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
|
||||
reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
|
||||
README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
|
||||
verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
|
||||
building and binding an `-r2`.
|
||||
|
||||
- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
|
||||
a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
|
||||
`"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
|
||||
on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
|
||||
/ `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
|
||||
which would have failed the whole `platform-config` Kustomization, realm import included.
|
||||
- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
|
||||
`WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
|
||||
a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
|
||||
If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
|
||||
is built beside it).
|
||||
- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
|
||||
and says not to hotfix the flow while it is `"true"`.
|
||||
- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
|
||||
git's value. The L0 check asserts the CronJob carries no `--` argument.
|
||||
- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
|
||||
order and why it matters, plus how to test the candidate on one client (a per-client
|
||||
`account-console` browser-flow override) before the go-live PR.
|
||||
|
||||
**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
|
||||
26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
|
||||
fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
|
||||
`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
|
||||
where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
|
||||
into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
|
||||
rows, which is why all 216 assertions passed anyway.
|
||||
|
||||
- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
|
||||
`getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
|
||||
through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
|
||||
leaves the flow pass fail-closed.
|
||||
- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
|
||||
`alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
|
||||
provider id) and the config alias as `alias`. Nulls are omitted.
|
||||
- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
|
||||
the protected set, and an execution add or requirement write into them is refused. A missing
|
||||
`displayName` fails closed.
|
||||
|
||||
**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
|
||||
resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
|
||||
`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
|
||||
`authenticationFlow` + `displayName`. Now:
|
||||
|
||||
- `body["id"]` is screened against the protected execution ids.
|
||||
- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
|
||||
created, open only inside `build_flow`.
|
||||
- The config `POST /executions/{id}/config` is held to the same build scope.
|
||||
- The body may carry only `id`, `requirement` and `priority`.
|
||||
- Selftest cases cover each of these.
|
||||
|
||||
**4. Stale docs on sensitive paths** now describe what ships, including the switch:
|
||||
`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
|
||||
renderer template, the renderer's own header and `--check` message, and the README.
|
||||
|
||||
**5.**
|
||||
- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
|
||||
another of the 5 candidate aliases.
|
||||
- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
|
||||
`for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
|
||||
shape as the sibling's. It has promtool tests in
|
||||
`platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
|
||||
"the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
|
||||
ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
|
||||
`runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
|
||||
- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
|
||||
enables the `kubernetes` manager over `platform/**`. The image is pinned to
|
||||
`python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
|
||||
the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
|
||||
reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
|
||||
neither today's index nor today's amd64 manifest.
|
||||
|
||||
**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
|
||||
in the script, where it ships next to the code and its selftest. A short header above the
|
||||
ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
|
||||
into `_finish_build()` so the build scope can be opened and closed around it.
|
||||
|
||||
Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
|
||||
`platform-config/kustomization.yaml` (comment only). This needs the operator's
|
||||
`/approve-override` or a non-author review. I did not post one.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
|
||||
shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
|
||||
pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
|
||||
(including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
|
||||
`alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
|
||||
offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
|
||||
`SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
|
||||
skipped by design (no cluster). rc 3 is **not** green.
|
||||
- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
|
||||
in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
|
||||
promtool 3.14.0.
|
||||
|
||||
- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
|
||||
check.
|
||||
- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
|
||||
one fix:
|
||||
- sub-flows read by `alias` again;
|
||||
- an unnameable sub-flow tolerated;
|
||||
- the requirement PUT screened on its path only;
|
||||
- execution writes not held to the build tree;
|
||||
- extra body keys allowed;
|
||||
- the bind switch ignored in `run()`;
|
||||
- the bind switch ignored in `reconcile_flow()`;
|
||||
- an absent switch reading as true;
|
||||
- an invalid switch guessed at;
|
||||
- `-r2` built beside a hand-edited bound flow.
|
||||
"Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
|
||||
rather than failing an assertion). It is still caught.
|
||||
- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
|
||||
checked this by hand-editing the committed file and running `--check`.
|
||||
- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
|
||||
(lines ~225-231, 658-724 and 747-806 of the file at that tag).
|
||||
- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
|
||||
(2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
|
||||
passkey reconciler is **not** live.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
|
||||
source, but they are still my reading of that source, not a captured response. In particular,
|
||||
I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
|
||||
differs, the failure is fail-closed (no build), not fail-open.
|
||||
- **The candidate flow has never been tested in a browser.** That is now a precondition of the
|
||||
go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
|
||||
I also have not checked that the per-client `account-console` override behaves as described on
|
||||
26.0.6.
|
||||
- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
|
||||
realm.** They set the passwordless policy and enable the enrolment action. That action is
|
||||
enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
|
||||
allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
|
||||
- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
|
||||
back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
|
||||
The window is one HTTP round trip and only exists when the policy has drifted.
|
||||
- **The alert has never seen a real series.** The CronJob has never run, so
|
||||
`kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
|
||||
shape is copied from the sibling's rule, which does have live series.
|
||||
- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
|
||||
`tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
|
||||
run here.
|
||||
- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
|
||||
operator overrides.
|
||||
|
||||
### Follow-ups (out of scope here, deliberately not done)
|
||||
|
||||
- A dedicated least-privilege service-account client for the reconciler, instead of the
|
||||
master-realm break-glass admin it shares with the client-secret reconciler.
|
||||
- A NetworkPolicy for the `keycloak` namespace.
|
||||
- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
|
||||
decision, currently "no TOTP after a passkey".
|
||||
- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
|
||||
Only the failure alert was in scope.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
date: 2026-10-30
|
||||
branch: fix/gitea-admit-buildkit
|
||||
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
|
||||
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
|
||||
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
|
||||
|
||||
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
|
||||
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
|
||||
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
|
||||
open the direct ClusterIP path from build code to Gitea, which nothing uses.
|
||||
|
||||
### Why: the brief's hypothesis was half right
|
||||
|
||||
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
|
||||
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
|
||||
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
|
||||
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
|
||||
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
|
||||
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
|
||||
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
|
||||
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
|
||||
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
|
||||
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
|
||||
namespaces either.
|
||||
|
||||
## What I verified
|
||||
|
||||
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
|
||||
|
||||
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
|
||||
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
|
||||
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
|
||||
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
|
||||
Gateway reach Gitea and get logged. This one never arrived.
|
||||
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
|
||||
(Gateway VIP `192.168.2.20`) gave:
|
||||
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
|
||||
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
|
||||
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
|
||||
|
||||
No token was involved, so this cannot be a package-permission problem. The bot account and the
|
||||
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
|
||||
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
|
||||
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
|
||||
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
|
||||
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
|
||||
the 403 itself in 0 ms.
|
||||
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
|
||||
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
|
||||
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
|
||||
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
|
||||
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
|
||||
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
|
||||
happens on the source's egress, not on the destination's ingress.
|
||||
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
|
||||
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
|
||||
is no Gitea identity.
|
||||
- The edited YAML parses (three documents, names unchanged).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
|
||||
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
|
||||
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
|
||||
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
|
||||
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
|
||||
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
|
||||
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
|
||||
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
|
||||
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
|
||||
same pod spec and the same policy, but it is not the original request.
|
||||
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
|
||||
did not read them.
|
||||
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
|
||||
for a first push that creates a new package. These only become testable after this lands.
|
||||
@@ -127,6 +127,30 @@ resources:
|
||||
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
|
||||
# cannot disagree about the domain.
|
||||
- ../platform/keycloak/client-urls.yaml
|
||||
# PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
|
||||
# browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
|
||||
# merged: the KeycloakRealmImport read Done=True / Started=False with the whole
|
||||
# webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
|
||||
# CronJob converges the policy and the enrolment required action, and builds the
|
||||
# declared browser flow BLUE/GREEN: under a new alias, verified by readback, and
|
||||
# structurally unable to write the flow that is currently bound (a half-rewritten bound
|
||||
# flow is a total SSO outage, and this repo has had one). It BINDS the verified flow
|
||||
# (one atomic PUT of browserFlow) ONLY when passkey-desired-state.yaml sets
|
||||
# bind-browser-flow: "true"; the default "false" leaves the realm's binding untouched.
|
||||
# See the script header in that file for the full rationale.
|
||||
- ../platform/keycloak/passkey-reconciler.yaml
|
||||
# Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
|
||||
# ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
|
||||
# client-urls.yaml above: the reconciler pod has no Kubernetes API access
|
||||
# (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
|
||||
# and its input must be MOUNTED.
|
||||
#
|
||||
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
|
||||
# the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
|
||||
# ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
|
||||
# two different values. An RP ID is baked into every passkey at enrolment and can never
|
||||
# be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
|
||||
- ../platform/keycloak/passkey-desired-state.yaml
|
||||
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
|
||||
- ../platform/keycloak/httproute.yaml
|
||||
- ../platform/keycloak/servicemonitor.yaml
|
||||
|
||||
@@ -20,7 +20,7 @@ privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
||||
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
||||
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
||||
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
||||
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
|
||||
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
|
||||
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
||||
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
||||
|
||||
|
||||
@@ -64,6 +64,20 @@ spec:
|
||||
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
|
||||
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
|
||||
# ingress allow admits.
|
||||
# * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
|
||||
# not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
|
||||
# Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
|
||||
# resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
|
||||
# connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
|
||||
# `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
|
||||
# this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
|
||||
# gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
|
||||
# Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
|
||||
# act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
|
||||
# fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
|
||||
# Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
|
||||
# admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
|
||||
# the ClusterIP path from build code to Gitea, which nothing here uses.
|
||||
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
|
||||
# pull request.
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -86,3 +100,10 @@ spec:
|
||||
- { port: "80", protocol: TCP }
|
||||
- toEntities:
|
||||
- ingress
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: gitea
|
||||
app.kubernetes.io/name: gitea
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "3000", protocol: TCP }
|
||||
+153
-15
@@ -31,6 +31,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
|
||||
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
||||
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
|
||||
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
|
||||
| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one, **bound only when git says so**) | **platform-config** | config |
|
||||
| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state — plus the one hand-edited line, `bind-browser-flow`) | **platform-config** | config |
|
||||
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
||||
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
||||
|
||||
@@ -154,7 +156,7 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
|
||||
|
||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
||||
@@ -170,7 +172,7 @@ comments, not repeated here. The headline choices:
|
||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
||||
|
||||
### This change does NOT reach the live realm. At all.
|
||||
### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
|
||||
|
||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
||||
2026-10-17:
|
||||
@@ -188,24 +190,160 @@ endpoint allow-list makes `/authentication` unreachable **by design** (see the t
|
||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
||||
|
||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
||||
an operator applies it by hand.** Two paths, both operator actions:
|
||||
`passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
|
||||
secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
|
||||
declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
|
||||
reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
|
||||
`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
|
||||
|
||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
||||
Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
|
||||
`ci/render-keycloak-passkey-policy.py` and gated by
|
||||
`conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
|
||||
That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
|
||||
and **can never be re-keyed**, so two declarations free to skew about the domain would be an
|
||||
unrecoverable-credential bug, not a cosmetic one.
|
||||
|
||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
||||
| Declared thing | Who applies it | Why |
|
||||
|---|---|---|
|
||||
| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
|
||||
| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
|
||||
| The `browser-mfa` flow restructure | **The reconciler builds and verifies it. Git decides whether it is bound.** | Blue/green: build under a new alias, verify, and flip `browserFlow` in one atomic PUT **only when `bind-browser-flow: "true"`** — see below |
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
||||
#### The flow converges blue/green — it never mutates the flow you log in through
|
||||
|
||||
Converging a five-flow / ten-execution restructure needs create-subflow,
|
||||
create-execution, set-requirement and bind-config as **separate** admin API calls, with no
|
||||
transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
|
||||
leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
|
||||
PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
|
||||
outage from a realm change, and the conditional-OTP sub-flow — which the declared
|
||||
structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
|
||||
|
||||
That risk is real but it is **not inherent**: it follows only from mutating the *bound*
|
||||
flow. So the reconciler does not.
|
||||
|
||||
1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
|
||||
hash of the desired shape, applied to every flow in the tree. Every multi-call write
|
||||
lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
|
||||
flow and **zero user-visible effect**.
|
||||
2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
|
||||
the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
|
||||
regression that once 403'd every platform-admin is caught **before** it can bite.
|
||||
3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow` — **only if git says
|
||||
so** (next section). Otherwise the run stops here and logs `WOULD BIND browser-mfa-<hash>`.
|
||||
4. **Rollback is a PR plus one write** (below), and nothing is ever deleted, so the old flow
|
||||
stays usable.
|
||||
|
||||
#### The bind switch: `bind-browser-flow` in `passkey-desired-state.yaml`
|
||||
|
||||
That generated file carries exactly one hand-edited line, `bind-browser-flow`, and it is
|
||||
`"false"` by default. The renderer carries it through on `--write`, and `--check` refuses
|
||||
anything but the quoted strings `"true"` / `"false"` (an unquoted `true` is a YAML boolean,
|
||||
which a ConfigMap rejects at apply time and which would fail the whole `platform-config`
|
||||
Kustomization). The CronJob reads it from the same mount as the desired state; no flag can
|
||||
widen it.
|
||||
|
||||
| `bind-browser-flow` | What the reconciler does to the realm's `browserFlow` |
|
||||
|---|---|
|
||||
| `"false"` (default) | **Never writes it.** It builds and verifies the candidate (or reuses an already verified one), logs `WOULD BIND <alias>`, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it). |
|
||||
| `"true"` | Binds the verified candidate in one atomic PUT, **and keeps asserting it**: a binding changed by hand is changed back within 15 minutes, and a hand-edit *inside* the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is `"true"`. |
|
||||
|
||||
**Merging with `"false"` changes nothing about how anyone logs in.** Passes 1 and 2 do write
|
||||
the realm (the passwordless policy and the enrolment action, neither of which any login
|
||||
step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but
|
||||
`browserFlow` is untouched.
|
||||
|
||||
**Going live** is a later one-line PR — `bind-browser-flow: "true"` — made **after** the
|
||||
candidate has been tested in a browser (see "Testing the candidate flow before binding it").
|
||||
|
||||
**Rolling back**, in this order:
|
||||
|
||||
1. A PR setting `bind-browser-flow: "false"`, merged and applied by Flux. **First** — while it
|
||||
is `"true"`, the next run rebinds the candidate over your rollback.
|
||||
2. One write: bind the previous flow again. The Job log of the bind printed it
|
||||
(`ROLLBACK is one write: set browserFlow back to ...`); it is `browser-mfa` for the first
|
||||
go-live. Admin console: *Authentication → Flows → <previous flow> → Action → Bind flow →
|
||||
Browser flow*, or `PUT /admin/realms/homelab` as a read-modify-write of the full realm
|
||||
representation (a partial body drops the passkey policy).
|
||||
|
||||
If login is broken *right now*, do step 2 immediately and step 1 straight after: the worst
|
||||
case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux
|
||||
applies the PR, and you repeat step 2.
|
||||
|
||||
**The safety property is structural, not intentional.** The reconciler is *incapable* of
|
||||
writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
|
||||
whole subtree of sub-flow aliases and execution ids are read first and screened against
|
||||
every write before a socket opens. It is **fail-closed** — until that binding has been
|
||||
read, every flow write is refused, and if the bound flow's executions cannot be read the
|
||||
flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
|
||||
|
||||
**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
|
||||
would churn the login path on a schedule. The bound flow is compared structurally against
|
||||
the declared shape and, when they match, the run issues **zero writes and builds
|
||||
nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
|
||||
reconciler built carries a hash suffix, so comparing labels would make it permanently
|
||||
unrecognisable as converged.
|
||||
|
||||
**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
|
||||
later run reads it: if it matches desired it is a completed-but-unbound build and is
|
||||
simply **bound**; if not, it is left alone and the next candidate alias is used. The
|
||||
candidate count is **bounded**, so a persistently failing build fails loudly naming the
|
||||
flows to clean up rather than creating unbounded sprawl.
|
||||
|
||||
```sh
|
||||
# What the reconciler currently says. A normal run prints the full comparison.
|
||||
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
|
||||
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
|
||||
|
||||
# Which candidate it would bind (bind-browser-flow "false"):
|
||||
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
|
||||
| grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'
|
||||
|
||||
# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
|
||||
# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
|
||||
# override the container command, so --audit / --no-bind need a one-off Job with the flag
|
||||
# appended (--no-bind can only NARROW git's switch, never widen it).
|
||||
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
|
||||
passkey-run-once
|
||||
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
|
||||
|
||||
# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
|
||||
# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
|
||||
# Browser flow). The old flow is never deleted, so this always works.
|
||||
|
||||
# Regenerate the desired state after editing the passkey policy or the flow in the realm
|
||||
# import (the merge gate will demand it).
|
||||
python3 ci/render-keycloak-passkey-policy.py --write
|
||||
```
|
||||
|
||||
### Testing the candidate flow before binding it
|
||||
|
||||
With `bind-browser-flow: "false"` the candidate exists but nothing logs in through it. Test it
|
||||
on **one client** without touching the realm binding, using a per-client flow override:
|
||||
|
||||
1. Find the candidate: the Job log line `WOULD BIND browser-mfa-<hash>`.
|
||||
2. Admin console → *Clients → `account-console` → Advanced → Authentication flow overrides →
|
||||
Browser Flow* = `browser-mfa-<hash>`. This is a temporary, hand-made client change; remove it
|
||||
in step 7.
|
||||
3. **Password fallback.** Private window → `https://keycloak.<base domain>/realms/homelab/account`.
|
||||
Expect a **username** screen, then a choice: passkey, or the password form. As a user with no
|
||||
passkey, choose password and sign in.
|
||||
4. **Platform-admin TOTP on the password branch.** Repeat step 3 as a `platform-admins` member:
|
||||
after the password, a TOTP prompt **must** appear. If it does not — or you get a 403 — the
|
||||
conditional-OTP step is wrong: stop, do not bind.
|
||||
5. **Passkey enrolment.** Signed in to the account console → *Signing in → Passkey → Set up*. A
|
||||
browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and
|
||||
nothing server-side will say so.
|
||||
6. **Passkey login.** Sign out; private window again; username, then choose passkey. Expect to be
|
||||
signed in with **no** TOTP prompt, even as a platform-admin.
|
||||
7. Remove the override from step 2.
|
||||
8. Only now: the one-line PR `bind-browser-flow: "true"`. The next run binds the candidate you
|
||||
tested (same alias — the hash is of the declared shape).
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-29)
|
||||
|
||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
||||
it, in order:
|
||||
it on the BOUND flow, after go-live, in order:
|
||||
|
||||
```sh
|
||||
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
---
|
||||
# =============================================================================
|
||||
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
|
||||
#
|
||||
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
|
||||
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
|
||||
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
|
||||
# (runs this renderer with --check on every merge)
|
||||
#
|
||||
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
|
||||
# the WebAuthn passwordless policy, the enrolment required action, and the declared
|
||||
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
|
||||
# it and (a) converges the policy and the required action onto it, (b) builds the
|
||||
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
|
||||
# writing the flow users are logging in through, and (c) BINDS that verified flow as
|
||||
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
|
||||
#
|
||||
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
|
||||
# operator decision, carried through by the renderer from this committed file (see
|
||||
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
|
||||
#
|
||||
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
|
||||
# once Done=True, so a realm-level field or a flow edited in git never reaches the
|
||||
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
|
||||
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
|
||||
# the level-triggered loop that closes that; this file is what it reconciles
|
||||
# TOWARDS.
|
||||
#
|
||||
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
|
||||
# user-verification requirement and the flow tree would be a second source of truth,
|
||||
# free to disagree with the realm import — the same "two declarations that can
|
||||
# disagree" shape as the bug being closed. It is rendered from the import and the
|
||||
# render is gated, so the two cannot drift.
|
||||
#
|
||||
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
|
||||
# substitution token exactly as the realm import writes it. Both files are rendered
|
||||
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
|
||||
# same value in both and the domain is decided in exactly one place.
|
||||
#
|
||||
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
|
||||
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
|
||||
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
|
||||
# nine more (a hard lock-out of every platform-admin). It is derived instead from
|
||||
# the presence of a webauthn-authenticator-passwordless execution in the flow:
|
||||
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
|
||||
# false, so enrolment stays opt-in.
|
||||
#
|
||||
# A field NOT declared on the realm in the import is simply ABSENT here, and the
|
||||
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: keycloak-passkey-desired-state
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-passkey-desired-state
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
# ---------------------------------------------------------------------------
|
||||
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
|
||||
# "false", QUOTED.
|
||||
#
|
||||
# "false" the reconciler builds and verifies the candidate browser flow and
|
||||
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
|
||||
# Whatever the realm is bound to -- including a rollback -- is left alone.
|
||||
# "true" it binds the verified candidate in one atomic PUT, and keeps the
|
||||
# realm bound to it: a binding changed by hand is changed back.
|
||||
#
|
||||
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
|
||||
# "Passkeys"), then a PR setting this to "true".
|
||||
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
|
||||
# one write setting browserFlow back to the previous alias from the Job log.
|
||||
# ---------------------------------------------------------------------------
|
||||
bind-browser-flow: "false"
|
||||
passkey-desired-state.json: |
|
||||
{
|
||||
"authenticatorConfig": {
|
||||
"conditional-otp-platform-admin": {
|
||||
"condUserRole": "platform-admin",
|
||||
"negate": "false"
|
||||
}
|
||||
},
|
||||
"browserFlow": "browser-mfa",
|
||||
"flows": {
|
||||
"browser-mfa": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-cookie",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "identity-provider-redirector",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-forms",
|
||||
"requirement": "ALTERNATIVE"
|
||||
}
|
||||
],
|
||||
"topLevel": true
|
||||
},
|
||||
"browser-mfa-conditional-otp": {
|
||||
"executions": [
|
||||
{
|
||||
"configAlias": "conditional-otp-platform-admin",
|
||||
"kind": "authenticator",
|
||||
"name": "conditional-user-role",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-otp-form",
|
||||
"requirement": "REQUIRED"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-forms": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-username-form",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-passkey-or-password",
|
||||
"requirement": "REQUIRED"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-passkey-or-password": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "webauthn-authenticator-passwordless",
|
||||
"requirement": "ALTERNATIVE"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-password-and-otp",
|
||||
"requirement": "ALTERNATIVE"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
},
|
||||
"browser-mfa-password-and-otp": {
|
||||
"executions": [
|
||||
{
|
||||
"kind": "authenticator",
|
||||
"name": "auth-password-form",
|
||||
"requirement": "REQUIRED"
|
||||
},
|
||||
{
|
||||
"kind": "flow",
|
||||
"name": "browser-mfa-conditional-otp",
|
||||
"requirement": "CONDITIONAL"
|
||||
}
|
||||
],
|
||||
"topLevel": false
|
||||
}
|
||||
},
|
||||
"realm": "homelab",
|
||||
"requiredAction": {
|
||||
"alias": "webauthn-register-passwordless",
|
||||
"defaultAction": false,
|
||||
"enabled": true
|
||||
},
|
||||
"webAuthnPolicyPasswordless": {
|
||||
"webAuthnPolicyPasswordlessAcceptableAaguids": [],
|
||||
"webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
|
||||
"webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
|
||||
"webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
|
||||
"webAuthnPolicyPasswordlessCreateTimeout": 300,
|
||||
"webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
|
||||
"webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
|
||||
"webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
|
||||
"webAuthnPolicyPasswordlessSignatureAlgorithms": [
|
||||
"ES256",
|
||||
"RS256"
|
||||
],
|
||||
"webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1445,6 +1445,36 @@ spec:
|
||||
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
|
||||
that would catch the healer being quietly absent during a domain migration.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
|
||||
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
|
||||
# enrolment required action, and the browser login flow (blue/green; bound only when
|
||||
# git's bind-browser-flow is "true"). There is no failure that is "expected by
|
||||
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
|
||||
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
|
||||
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
|
||||
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
|
||||
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
|
||||
# single transient failure while still firing within about two ticks.
|
||||
# A separate rule from the client reconciler's on purpose: different CronJob,
|
||||
# different job_name, different runbook — one cause, one alert.
|
||||
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
|
||||
# did not exist yet. The metric and label are the ones the sibling rule above uses.
|
||||
- alert: KeycloakPasskeyReconcilerFailing
|
||||
expr: |
|
||||
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak passkey reconciler failing"
|
||||
description: >-
|
||||
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
|
||||
policy, the passkey enrolment action or the browser login flow may differ from
|
||||
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
|
||||
never writes the flow users are logging in through, so a failure here does NOT
|
||||
by itself mean login is broken. Read the Job log: every PROBLEM line names the
|
||||
pass and the exact divergence.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
|
||||
|
||||
# =====================================================================================
|
||||
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
|
||||
# group platform-health.identity).
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
|
||||
# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
|
||||
# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
|
||||
# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
|
||||
# flow, so a failed Job is always news. What can go wrong with the rule:
|
||||
# * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
|
||||
# * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
|
||||
# failures already have their own alert and runbook; one cause, one alert;
|
||||
# * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
|
||||
# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
|
||||
# that forgot the `> 0` would fire here.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: passkey reconciler succeeding - quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
|
||||
values: '0+0x60'
|
||||
- series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: passkey reconciler Job failed - fires after 15m
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
- eval_time: 20m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Keycloak passkey reconciler failing"
|
||||
description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
|
||||
# same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: only the client-secret reconciler failing - passkey alert quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
|
||||
values: '0+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
|
||||
# series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: single transient failure cleared within 15m - quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
|
||||
values: '1+0x5'
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
|
||||
values: '_x15 0+0x45'
|
||||
alert_rule_test:
|
||||
- eval_time: 30m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
|
||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||
every green signal missed.
|
||||
- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
|
||||
failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
|
||||
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
|
||||
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
|
||||
rather than a 401, the browser follows it silently against a live SSO session, and the call
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
|
||||
|
||||
**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
|
||||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
|
||||
**Expression:**
|
||||
```
|
||||
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
|
||||
```
|
||||
|
||||
## What this means here
|
||||
|
||||
`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
|
||||
converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
|
||||
at `Done=True`:
|
||||
|
||||
1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
|
||||
2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
|
||||
3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
|
||||
(`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
|
||||
`bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
|
||||
default) a run that builds and verifies a candidate but does not bind it is a **success**.
|
||||
|
||||
So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
|
||||
is structurally unable to write the flow users are currently logging in through, so **a failing Job
|
||||
does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
|
||||
`CONFIGURE_TOTP` required action (see below).
|
||||
|
||||
## First commands
|
||||
|
||||
Every command runs in the pinned toolbox — kubectl is not on the workstation:
|
||||
|
||||
```sh
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
|
||||
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
|
||||
# What git currently asks for (the bind switch):
|
||||
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
|
||||
flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
|
||||
design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
|
||||
(Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
|
||||
2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
|
||||
but the readback differs (the log names the step). Login is unaffected. If it names the
|
||||
conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
|
||||
platform-admin, caught before it went live.
|
||||
3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
|
||||
`requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
|
||||
lock-out for platform-admins: re-register the action in the admin console
|
||||
(Authentication → Required actions). The reconciler cannot repair it.
|
||||
4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
|
||||
`keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
|
||||
(`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
|
||||
5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
|
||||
transient; if persistent, the realm is bound to a flow that no longer exists.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
|
||||
`"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
|
||||
first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
|
||||
- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
|
||||
roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
|
||||
alias printed in the Job log).
|
||||
- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.
|
||||
@@ -77,6 +77,7 @@ configMapGenerator:
|
||||
- image-automation=image-automation.md
|
||||
- image-policy-stalled=image-policy-stalled.md
|
||||
- keycloak-client-secrets=keycloak-client-secrets.md
|
||||
- keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
|
||||
- keycloak-db=keycloak-db.md
|
||||
- keycloak-down=keycloak-down.md
|
||||
- kubevirt-control-plane=kubevirt-control-plane.md
|
||||
|
||||
Reference in new issue
Block a user