test(conformance): gate the URL desired state, and prove the selftest can fail
L0 (212 assertions, was 175):
* runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
what the realm import renders. A hand edit that makes the two disagree is
unmergeable -- this is what keeps the realm import the single source of truth for
hostnames rather than just the place they were first typed.
* asserts client-urls.yaml and the realm import are rendered by the SAME Flux
Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
a hostname the HTTPRoutes do not serve.
* asserts the desired state uses no substitution token the realm definition lacks,
names only clients the import declares, and declares no field outside the
five-field write surface; that the CronJob mounts it readOnly at the path the
script defaults to; and that the script, renderer and check agree on which fields
are writable.
* NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
requires --selftest to catch every one. A passing suite proves nothing if it cannot
fail, and "reported success while the realm was broken" is precisely what this
component exists to end -- so the same standard applies to its own test. ~20s.
L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.
Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
23ea72d01e
commit
6202d92c27
4 files changed
+465
-20
No files matched your search
@@ -431,7 +431,7 @@ spec:
|
||||
summary: "Keycloak database unavailable"
|
||||
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
|
||||
# --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
||||
# --- The client reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
||||
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
|
||||
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
|
||||
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
|
||||
@@ -440,6 +440,14 @@ spec:
|
||||
# the healer; these two rules make its SILENCE observable, so the class cannot sit
|
||||
# undetected again. They are the runtime counterpart of the L2 audit test.
|
||||
#
|
||||
# THE CronJob NOW RECONCILES TWO THINGS — client secrets, and the environment-derived
|
||||
# client URLs (redirectUris, webOrigins, rootUrl, baseUrl, adminUrl) that a BASE_DOMAIN
|
||||
# migration re-renders in git and the one-shot import never carries to the realm. The
|
||||
# pair below is EXTENDED to say so rather than DUPLICATED: one CronJob gets one
|
||||
# Failing/Stale pair, because a second pair keyed to the same job_name regex would fire
|
||||
# twice for a single cause and train the operator to ignore both. The Job log names which
|
||||
# pass failed, which client, and for a URL the exact before/after.
|
||||
#
|
||||
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
|
||||
# these series had no samples to check against. `kube_job_status_failed` /
|
||||
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
|
||||
@@ -457,12 +465,15 @@ spec:
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak client-secret reconciler failing"
|
||||
summary: "Keycloak client reconciler failing"
|
||||
description: >-
|
||||
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
|
||||
secret may differ from git (unresolved import placeholder, or an unrepairable
|
||||
drift) and SSO logins for that client will fail. Read the Job log: it names the
|
||||
client and never prints a secret.
|
||||
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client's
|
||||
stored SECRET or its REDIRECT URIs may differ from git (an unresolved import
|
||||
placeholder, a pre-migration hostname the one-shot realm import never updated, or
|
||||
an unrepairable drift), and SSO logins for that client will fail — a bad
|
||||
redirectUri fails with a redirect_uri mismatch at the IdP. Read the Job log: it
|
||||
names the pass, the client and the exact URL before/after, and never prints a
|
||||
secret.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
- alert: KeycloakClientSecretReconcilerStale
|
||||
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
|
||||
@@ -475,11 +486,12 @@ spec:
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak client-secret reconciler has not succeeded"
|
||||
summary: "Keycloak client reconciler has not succeeded"
|
||||
description: >-
|
||||
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
|
||||
The CronJob may be suspended, unapplied, or unable to schedule — client-secret
|
||||
drift would go uncorrected and unnoticed (docs/15).
|
||||
The CronJob may be suspended, unapplied, or unable to schedule — client-secret AND
|
||||
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
|
||||
that would catch the healer being quietly absent during a domain migration.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
Reference in new issue
Block a user