docs(keycloak): record the URL-reconciliation decisions
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched, public clients' URLs managed, one alert pair not two, and the mutation harness. Also records a process note: my first pass reported three false findings (reconciler absent from main, import not templated, BASE_DOMAIN dead) because I diffed against origin/main -- origin is dev's Gitea, stale at4809199, while the live remote is prod-gitea atfa4deac. All three were wrong. The rule for this repo is written down so the next agent does not repeat it: origin is not the source of truth. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
6202d92c27
commit
f7d4648851
2 files changed
+149
-7
No files matched your search
@@ -29,7 +29,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
|
||||
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
|
||||
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
|
||||
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
||||
| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
|
||||
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
|
||||
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
|
||||
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
||||
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
||||
|
||||
@@ -85,8 +86,8 @@ Three layers now cover it:
|
||||
| Layer | Where | What it does |
|
||||
|---|---|---|
|
||||
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
|
||||
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
|
||||
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
|
||||
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m, two passes | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ; a second, independent pass does the same for the client URLs against the generated `client-urls.yaml` (see below). Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
|
||||
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch, literal placeholder **or drifted client URL**; the alerts make the reconciler's silence observable in prod. |
|
||||
|
||||
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
|
||||
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
|
||||
@@ -99,10 +100,59 @@ kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler
|
||||
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
|
||||
```
|
||||
|
||||
**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
|
||||
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
|
||||
its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
|
||||
landed.
|
||||
## Client URLs: the same one-shot failure, one field over
|
||||
|
||||
Moving an environment's `BASE_DOMAIN` re-renders every declared `redirectUris` / `webOrigins` /
|
||||
`rootUrl` / `baseUrl` / `adminUrl` in git — and reaches the realm **never**, because a completed
|
||||
import does not re-run. Measured on prod: stored redirect URIs still on the pre-migration hostname
|
||||
while git declared the new one. Every OIDC login then fails with a `redirect_uri` mismatch, the
|
||||
`KeycloakRealmImport` still reads `Done=True`, and nothing alerts.
|
||||
|
||||
**Pass 2 of the same CronJob** closes it. It was made a second pass of the existing reconciler rather
|
||||
than a sibling CronJob for one decisive reason: **a URL write needs no new endpoint.** It is the same
|
||||
minimal `PUT /admin/realms/<realm>/clients/<uuid>`, and the current URLs already come back in the
|
||||
allow-listed client lookup — so the hard allow-list that makes `/users`, `/groups`, `/roles` and
|
||||
`/authentication` unreachable is widened by *nothing*, and is preserved rather than re-declared in a
|
||||
second copy. (The ConfigMap also carries exactly one script key by L0 contract, so a sibling would
|
||||
have had to fork ~300 lines of safety-critical code the header explicitly forbids forking.)
|
||||
|
||||
**The desired state is generated, not authored.** `client-urls.yaml` is rendered from
|
||||
`keycloak-realm-import.yaml` by `ci/render-keycloak-client-urls.py`, and the L0 gate runs that
|
||||
renderer with `--check` on every merge, so the committed file cannot disagree with the import. URL
|
||||
strings are copied verbatim — a `${BASE_DOMAIN}` token passes through unchanged, and both files are
|
||||
rendered by the **same** Flux Kustomization, so both are substituted in one pass with one value. The
|
||||
renderer never parses a hostname and never reads `cluster-vars`: there is no second place the domain
|
||||
can be decided.
|
||||
|
||||
```sh
|
||||
# regenerate after editing a client URL in the realm import (the gate will demand it)
|
||||
python3 ci/render-keycloak-client-urls.py --write
|
||||
```
|
||||
|
||||
Properties, each proved by `--selftest` and each shown to be *falsifiable* by
|
||||
`conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`:
|
||||
|
||||
* **Converged ⇒ zero writes.** List fields compare order-insensitively (`sorted`, not `set` —
|
||||
duplicates are still drift), because Keycloak does not preserve the authored order of
|
||||
`redirectUris`; an order-sensitive compare would rewrite every client forever while reporting
|
||||
success.
|
||||
* **Gaining and losing a URI both work.** The declared list is exact, so a URI removed from git is
|
||||
removed from the realm — a stale wildcard left behind quietly widens what the realm will redirect
|
||||
to.
|
||||
* **Undeclared ⇒ unmanaged.** A field the desired state omits is left exactly as it is. Absence never
|
||||
means "force to empty".
|
||||
* **Public clients' URLs *are* managed** (their secrets are not): `kubernetes` is PKCE-public and its
|
||||
localhost callbacks are what `kubectl` logins land on.
|
||||
* **The two passes are independent.** Each reads its own desired state and fails closed for its own
|
||||
fields only. A malformed URL ConfigMap cannot stop client-*secret* repair, and an unmounted Secret
|
||||
cannot stop URL repair.
|
||||
* URLs are logged in full (they are public hostnames, and the before/after is what makes a migration
|
||||
auditable); secrets are still never logged.
|
||||
|
||||
**What the reconciler does NOT fix:** anything outside client secrets and those five URL fields. A
|
||||
NEW client, a new protocol mapper or a changed scope still needs the realm import to actually re-run
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||
|
||||
|
||||
Reference in new issue
Block a user