jakman85andClaude Opus 5.5
7c4eb439b2
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
70 lines
4.3 KiB
Markdown
70 lines
4.3 KiB
Markdown
# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
|
|
|
|
**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
|
|
**Expression:**
|
|
```
|
|
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
|
|
```
|
|
|
|
## What this means here
|
|
|
|
`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
|
|
converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
|
|
at `Done=True`:
|
|
|
|
1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
|
|
2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
|
|
3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
|
|
(`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
|
|
`bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
|
|
default) a run that builds and verifies a candidate but does not bind it is a **success**.
|
|
|
|
So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
|
|
is structurally unable to write the flow users are currently logging in through, so **a failing Job
|
|
does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
|
|
`CONFIGURE_TOTP` required action (see below).
|
|
|
|
## First commands
|
|
|
|
Every command runs in the pinned toolbox — kubectl is not on the workstation:
|
|
|
|
```sh
|
|
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
|
|
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
|
|
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
|
|
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
|
|
# What git currently asks for (the bind switch):
|
|
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
|
|
flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
|
|
design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
|
|
(Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
|
|
2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
|
|
but the readback differs (the log names the step). Login is unaffected. If it names the
|
|
conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
|
|
platform-admin, caught before it went live.
|
|
3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
|
|
`requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
|
|
lock-out for platform-admins: re-register the action in the admin console
|
|
(Authentication → Required actions). The reconciler cannot repair it.
|
|
4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
|
|
`keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
|
|
(`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
|
|
5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
|
|
transient; if persistent, the realm is bound to a flow that no longer exists.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
|
|
`"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
|
|
first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
|
|
- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
|
|
roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
|
|
alias printed in the Job log).
|
|
- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.
|