feat(monitoring): alert when the keycloak passkey reconciler's Job fails

KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

runbooks/README.md

@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
+  failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
   `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
   rather than a 401, the browser follows it silently against a live SSO session, and the call

runbooks/keycloak-passkey-reconciler.md

@@ -0,0 +1,69 @@
+# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
+
+**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expression:**
+```
+max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
+```
+
+## What this means here
+
+`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
+converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
+at `Done=True`:
+
+1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
+2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
+3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
+   (`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
+   `bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
+   default) a run that builds and verifies a candidate but does not bind it is a **success**.
+
+So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
+is structurally unable to write the flow users are currently logging in through, so **a failing Job
+does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
+`CONFIGURE_TOTP` required action (see below).
+
+## First commands
+
+Every command runs in the pinned toolbox — kubectl is not on the workstation:
+
+```sh
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
+# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
+# What git currently asks for (the bind switch):
+grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
+```
+
+## Likely causes, most likely first
+
+1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
+   flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
+   design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
+   (Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
+2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
+   but the readback differs (the log names the step). Login is unaffected. If it names the
+   conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
+   platform-admin, caught before it went live.
+3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
+   `requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
+   lock-out for platform-admins: re-register the action in the admin console
+   (Authentication → Required actions). The reconciler cannot repair it.
+4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
+   `keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
+   (`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
+5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
+   transient; if persistent, the realm is bound to a flow that no longer exists.
+
+## When to escalate or stop
+
+- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
+  `"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
+  first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
+- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
+  roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
+  alias printed in the Job log).
+- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.

runbooks/kustomization.yaml

@@ -77,6 +77,7 @@ configMapGenerator:
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
       - keycloak-client-secrets=keycloak-client-secrets.md
+      - keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
       - keycloak-db=keycloak-db.md
       - keycloak-down=keycloak-down.md
       - kubevirt-control-plane=kubevirt-control-plane.md