home-network-k3s-impl/runbooks/keycloak-passkey-reconciler.md

keycloak-passkey-reconciler — the passkey reconciler's Job is failing

Alert: KeycloakPasskeyReconcilerFailing (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.identity Expression:

max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0

What this means here

platform/keycloak/passkey-reconciler.yaml is a CronJob (every 15 minutes, backoffLimit: 0) that converges three things the one-shot KeycloakRealmImport can never carry to the realm once it rests at Done=True:

1. the realm's WebAuthn passwordless policy (webAuthnPolicyPasswordless*);
2. the webauthn-register-passwordless required action (enabled, not a default action);
3. the browser login flow, blue/green: it builds the declared tree under a new alias (browser-mfa-<hash>), verifies it by readback, and binds it only if git says so — bind-browser-flow: "true" in platform/keycloak/passkey-desired-state.yaml. With "false" (the default) a run that builds and verifies a candidate but does not bind it is a success.

So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler is structurally unable to write the flow users are currently logging in through, so a failing Job does not by itself mean login is broken — but one cause it reports does lock people out: a missing CONFIGURE_TOTP required action (see below).

First commands

Every command runs in the pinned toolbox — kubectl is not on the workstation:

./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
# What git currently asks for (the bind switch):
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml

Likely causes, most likely first

1. could not find an unused alias ... after 5 attempt(s) — earlier builds left half-built flows under every candidate alias (browser-mfa-<hash>, -r2 … -r5). Nothing here deletes, by design. Delete the unbound browser-mfa-<hash>* flows named in the log in the admin console (Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
2. was built but does NOT match git, so it was NOT BOUND — the admin API accepted the build but the readback differs (the log names the step). Login is unaffected. If it names the conditional-OTP config (condUserRole), that is the regression that once 403'd every platform-admin, caught before it went live.
3. MISSING the default required action(s) ... CONFIGURE_TOTP — someone declared a requiredActions block in the realm import (26.0.6 treats it as REPLACE). This one IS a live lock-out for platform-admins: re-register the action in the admin console (Authentication → Required actions). The reconciler cannot repair it.
4. Keycloak admin API not usable / FATAL — Keycloak is down (check KeycloakDown first), the keycloak-admin-credentials Secret changed, or the desired-state ConfigMap is malformed (bind-browser-flow must be the quoted string "true" or "false").
5. FAIL-CLOSED — the bound flow's executions could not be read, so nothing was built. Usually transient; if persistent, the realm is bound to a flow that no longer exists.

When to escalate or stop

• Never fix this by editing the bound flow in the admin console while bind-browser-flow is "true": the next run will build a fresh copy and rebind it. Flip the switch to "false" in a PR first; see platform/keycloak/README.md, "Passkeys", for the rollback procedure.
• If people cannot log in, this alert is not the cause to chase — start at keycloak-down and roll the browser flow back (PR setting bind-browser-flow: "false", then rebind the previous alias printed in the Job log).
• Do not kubectl edit the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.