feat(monitoring): alert when the keycloak passkey reconciler's Job fails
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
17eed6e8ae
commit
7c4eb439b2
5 files changed
+192
No files matched your search
@@ -1445,6 +1445,36 @@ spec:
|
||||
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
|
||||
that would catch the healer being quietly absent during a domain migration.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
|
||||
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
|
||||
# enrolment required action, and the browser login flow (blue/green; bound only when
|
||||
# git's bind-browser-flow is "true"). There is no failure that is "expected by
|
||||
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
|
||||
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
|
||||
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
|
||||
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
|
||||
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
|
||||
# single transient failure while still firing within about two ticks.
|
||||
# A separate rule from the client reconciler's on purpose: different CronJob,
|
||||
# different job_name, different runbook — one cause, one alert.
|
||||
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
|
||||
# did not exist yet. The metric and label are the ones the sibling rule above uses.
|
||||
- alert: KeycloakPasskeyReconcilerFailing
|
||||
expr: |
|
||||
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak passkey reconciler failing"
|
||||
description: >-
|
||||
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
|
||||
policy, the passkey enrolment action or the browser login flow may differ from
|
||||
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
|
||||
never writes the flow users are logging in through, so a failure here does NOT
|
||||
by itself mean login is broken. Read the Job log: every PROBLEM line names the
|
||||
pass and the exact divergence.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
|
||||
|
||||
# =====================================================================================
|
||||
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
|
||||
# group platform-health.identity).
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
|
||||
# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
|
||||
# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
|
||||
# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
|
||||
# flow, so a failed Job is always news. What can go wrong with the rule:
|
||||
# * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
|
||||
# * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
|
||||
# failures already have their own alert and runbook; one cause, one alert;
|
||||
# * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
|
||||
# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
|
||||
# that forgot the `> 0` would fire here.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: passkey reconciler succeeding - quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
|
||||
values: '0+0x60'
|
||||
- series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: passkey reconciler Job failed - fires after 15m
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
- eval_time: 20m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Keycloak passkey reconciler failing"
|
||||
description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
|
||||
# same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: only the client-secret reconciler failing - passkey alert quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
|
||||
values: '0+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
|
||||
# series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: single transient failure cleared within 15m - quiet
|
||||
input_series:
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
|
||||
values: '1+0x5'
|
||||
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
|
||||
values: '_x15 0+0x45'
|
||||
alert_rule_test:
|
||||
- eval_time: 30m
|
||||
alertname: KeycloakPasskeyReconcilerFailing
|
||||
exp_alerts: []
|
||||
Reference in new issue
Block a user