feat(monitoring): alert when the keycloak passkey reconciler's Job fails

KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-29 21:41:15 -05:00
1 parent 17eed6e8ae
commit 7c4eb439b2
5 files changed
+192

No files matched your search

+30
View File
@@ -1445,6 +1445,36 @@ spec:
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
that would catch the healer being quietly absent during a domain migration.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
# enrolment required action, and the browser login flow (blue/green; bound only when
# git's bind-browser-flow is "true"). There is no failure that is "expected by
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
# single transient failure while still firing within about two ticks.
# A separate rule from the client reconciler's on purpose: different CronJob,
# different job_name, different runbook — one cause, one alert.
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
# did not exist yet. The metric and label are the ones the sibling rule above uses.
- alert: KeycloakPasskeyReconcilerFailing
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak passkey reconciler failing"
description: >-
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
policy, the passkey enrolment action or the browser login flow may differ from
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
never writes the flow users are logging in through, so a failure here does NOT
by itself mean login is broken. Read the Job log: every PROBLEM line names the
pass and the exact divergence.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)