# keycloak-passkey-reconciler — the passkey reconciler's Job is failing

**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
```

## What this means here

`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
at `Done=True`:

1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
   (`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
   `bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
   default) a run that builds and verifies a candidate but does not bind it is a **success**.

So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
is structurally unable to write the flow users are currently logging in through, so **a failing Job
does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
`CONFIGURE_TOTP` required action (see below).

## First commands

Every command runs in the pinned toolbox — kubectl is not on the workstation:

```sh
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
# What git currently asks for (the bind switch):
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
```

## Likely causes, most likely first

1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
   flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
   design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
   (Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
   but the readback differs (the log names the step). Login is unaffected. If it names the
   conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
   platform-admin, caught before it went live.
3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
   `requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
   lock-out for platform-admins: re-register the action in the admin console
   (Authentication → Required actions). The reconciler cannot repair it.
4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
   `keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
   (`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
   transient; if persistent, the realm is bound to a flow that no longer exists.

## When to escalate or stop

- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
  `"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
  first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
  roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
  alias printed in the Job log).
- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.