--- # ============================================================================= # Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04) # # CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only # after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients # Secret and the Keycloak Operator, and alongside the realm import itself. # # THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06) # ------------------------------------------------------------------- # keycloak-realm-import.yaml declares 12 confidential clients whose secrets are # import placeholders resolved from the keycloak-realm-clients Secret via # spec.placeholders. When the operator cannot resolve a placeholder it stores the # LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS # SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and # NEVER RE-RUNS, so nothing ever corrects it: # # * prod: all 12 clients held the literal placeholder. Every OIDC login on the # cluster failed with "Invalid client or Invalid client credentials". # * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added # AFTER dev's realm import had already completed, and a completed import is # never re-run, so the new key was never picked up. # # Both were repaired live via the admin API on 2026-09-06 and verified. This # CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is # replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same # posture Flux gives every other resource in this repo (R-GITOPS-03 drift # correction, applied to the one piece of state Flux itself cannot reach -- # Keycloak's own database). # # ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph # already guarantees the Secret is applied before the realm import (the Secret is # rendered by ./platform; the import by ./platform-config, which dependsOn # platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of # the Secret lands before revision N of the import). Ordering still cannot help, # because the operator will not re-run a completed import when a LATER revision # adds a key -- which is precisely how dev broke. Only a reconciler closes that. # # WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state # lives in Keycloak's Postgres database, not in a Kubernetes object, so no # admission or mutation policy can see it. The admin REST API is the only # interface to it. # # WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token, # client lookup, client-secret read, minimal client PUT). Users, groups, roles, # authentication flows and realm settings are structurally unreachable from it -- # prod's `homelab` realm carries a real user and the groups media-users / # platform-admins / viewers, and they must survive every run. It never logs a # secret, and it writes only where the stored value differs from git. # # PROOF: the script's own `--selftest` drives these real code paths against an # in-process stub Keycloak and asserts compare-then-write, idempotence (a second # run over a converged realm issues ZERO writes), no-secret-logging, the endpoint # allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script # from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships: # conformance/tests/l0-static/keycloak-client-secret-reconciler/. # ============================================================================= apiVersion: v1 kind: ConfigMap metadata: name: keycloak-client-secret-reconciler namespace: keycloak labels: app.kubernetes.io/name: keycloak-client-secret-reconciler app.kubernetes.io/part-of: keycloak data: # SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy: # the L0 gate extracts THIS block and executes its --selftest, so a second copy # would be untested. Contains no dollar sign at all, on purpose -- this file is # rendered by a Flux Kustomization with postBuild.substituteFrom, so an # unescaped dollar-brace expansion here would be rewritten before any container # ran (the trap documented at length in apps/wave-1/nextcloud.yaml). reconcile-client-secrets.py: | #!/usr/bin/env python3 # ============================================================================= # reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret # equal the value git declares for it, idempotently, without touching anything # else in the realm. # # WHY THIS EXISTS (the incident it prevents) # ------------------------------------------ # keycloak-realm-import.yaml declares each confidential client's secret as an # `(env:_CLIENT_SECRET)` import placeholder and maps to a key of the # SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the # Keycloak Operator cannot resolve a placeholder at import time it stores the # LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS # SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and # never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12 # clients with secrets held the literal placeholder and every OIDC login failed # with "Invalid client or Invalid client credentials"; on dev exactly one # (oauth2-proxy) was broken, because that Secret key was added after dev's import # had already completed. # # So the import is a ONE-SHOT that fails OPEN. This is the level-triggered # reconciler that closes it. # # CONTRACT # -------- # Desired state files in SECRETS_DIR named `-client-secret`, each # holding exactly the secret for that clientId (this is the # `keycloak-realm-clients` Secret, mounted). # Actual state GET /admin/realms//clients//client-secret # Action only where they differ: PUT /admin/realms//clients/ # with the MINIMAL representation {"clientId":..,"secret":..}. # # WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a # two-field PUT rewrites the secret and nothing else. PUTting the full # round-tripped representation back has a far larger blast radius and is in fact # rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the # JSON"} (measured during the live repair). The stub in --selftest models that # rejection, so a regression to the fat PUT fails in CI, not on a live realm. # # SAFETY PROPERTIES, each enforced in code and proved by --selftest: # 1. compare-then-write: a client already correct is SKIPPED, no PUT issued. # 2. idempotent: a second run over a converged realm issues zero writes. # 3. no secret is ever printed, in any mode, on any path. # 4. the ONLY endpoints this program may touch are the master-realm token # endpoint, the client LOOKUP, the client-secret READ and the minimal client # PUT. Every other URL is refused by _http() before a socket is opened, so # users, groups, roles, authentication flows and realm settings are # structurally unreachable from here -- prod's `homelab` realm carries a # real user and the groups media-users / platform-admins / viewers, and they # MUST survive. # 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped. # # MODES # (default) reconcile: compare, and write only what differs. # --audit READ-ONLY. Report drift and exit 1 if any client's stored secret # differs from the declared one. Never writes. This is what the L2 # conformance test runs. # --selftest the offline behavioural proof, against an in-process stub # Keycloak. No cluster; loopback only. This is what the L0 static # gate runs, so the code the gate proves is the code that ships. # # NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor # (hence re.fullmatch throughout), and not even inside these comments. It ships # inside a ConfigMap that the `platform-config` Flux Kustomization renders with # postBuild.substituteFrom, so Flux envsubst rewrites this text before any # container runs: an unescaped dollar-brace expansion would be silently emptied at # reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's # OIDC guard dead code for months -- read the comment block there. Python needs no # dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and # the character is banned outright so no future edit has to reason about escaping. # Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh. # ============================================================================= from __future__ import annotations import hmac import json import os import re import sys import time import urllib.error import urllib.parse import urllib.request SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients") KEY_SUFFIX = "-client-secret" TIMEOUT = 20 READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24")) READY_SLEEP = float(os.environ.get("READY_SLEEP", "5")) # Endpoint allow-list (safety property 4). fullmatch-ed against the path of every # URL _http() is asked to open; anything else raises before a socket exists. _ALLOWED = ( r"/realms/master/protocol/openid-connect/token", r"/admin/realms/[^/]+/clients", r"/admin/realms/[^/]+/clients/[^/]+", r"/admin/realms/[^/]+/clients/[^/]+/client-secret", ) class Refused(RuntimeError): """A URL outside the allow-list was requested. Never expected in a normal run.""" def _http(method, base, path, query=None, headers=None, form=None, body=None): """Return (status, text). Refuses any path outside the allow-list.""" if not any(re.fullmatch(p, path) for p in _ALLOWED): raise Refused("endpoint not on the allow-list: %s %s" % (method, path)) url = base.rstrip("/") + path if query: url = url + "?" + urllib.parse.urlencode(query) hdrs = dict(headers or {}) data = None if form is not None: data = urllib.parse.urlencode(form).encode() hdrs["Content-Type"] = "application/x-www-form-urlencoded" elif body is not None: data = json.dumps(body).encode() hdrs["Content-Type"] = "application/json" req = urllib.request.Request(url, data=data, headers=hdrs, method=method) try: with urllib.request.urlopen(req, timeout=TIMEOUT) as resp: return resp.status, resp.read().decode("utf-8", "replace") except urllib.error.HTTPError as e: return e.code, e.read().decode("utf-8", "replace") def declared(secrets_dir): """{clientId: secret} from the mounted Secret. Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A key must be named `-client-secret`: that naming contract is what lets this reconciler derive the client set from git without a second declaration, and the L0 static check enforces it against keycloak-realm-import.yaml so the two can never drift apart. """ out = {} for name in sorted(os.listdir(secrets_dir)): if name.startswith(".") or not name.endswith(KEY_SUFFIX): continue path = os.path.join(secrets_dir, name) if not os.path.isfile(path): continue with open(path, "r", encoding="utf-8") as fh: value = fh.read() # A trailing newline in the SOPS source would otherwise become part of the # secret and mismatch forever. Keycloak secrets never contain whitespace. out[name[: -len(KEY_SUFFIX)]] = value.strip() return out def admin_token(base, user, password, log): """Bearer token for admin-cli in the master realm, with a bounded readiness wait.""" last = "" for attempt in range(1, READY_ATTEMPTS + 1): try: st, text = _http( "POST", base, "/realms/master/protocol/openid-connect/token", form={"client_id": "admin-cli", "grant_type": "password", "username": user, "password": password}) except OSError as e: st, text = 0, "connection error (%s)" % e.__class__.__name__ if st == 200: token = json.loads(text).get("access_token") if token: return token last = "http 200 but no access_token in the response" else: # Deliberately do NOT echo the response body: it is harmless today, but # the request that produced it carried the admin password, and keeping # that class of mistake impossible by habit is cheaper than reviewing it. last = "http %s" % st log("waiting for the Keycloak admin API (attempt %d/%d): %s" % (attempt, READY_ATTEMPTS, last)) if attempt < READY_ATTEMPTS: time.sleep(READY_SLEEP) raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s" % (READY_ATTEMPTS, last)) def reconcile(base, realm, token, want, audit, log): """Return (changed, problems). Writes only when audit is False.""" hdrs = {"Authorization": "Bearer " + token} changed = 0 problems = [] for cid in sorted(want): st, text = _http("GET", base, "/admin/realms/%s/clients" % realm, query={"clientId": cid}, headers=hdrs) if st != 200: problems.append("%s: client lookup failed (http %s)" % (cid, st)) continue found = json.loads(text) if not isinstance(found, list) or not found: problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it " "but the realm has no such client (the operator runs the " "realm import exactly once)" % (cid, realm)) continue client = found[0] uuid = client.get("id") if client.get("publicClient"): log("%-14s skipped (public client, holds no secret)" % cid) continue st, text = _http("GET", base, "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid), headers=hdrs) if st != 200: problems.append("%s: could not read the stored secret (http %s)" % (cid, st)) continue stored = json.loads(text).get("value") or "" if hmac.compare_digest(stored, want[cid]): log("%-14s ok (already correct, no write)" % cid) continue # Describe the drift WITHOUT quoting either value. The literal-placeholder # case is named because it is the failure this program exists for, and # saying so reveals nothing secret. kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER" if "env:" in stored else "stored value differs from the declared one") if audit: problems.append("%s: DRIFT -- %s" % (cid, kind)) continue log("%-14s DRIFT (%s) -- repairing" % (cid, kind)) # MINIMAL representation only -- see the header. Keycloak merges non-null # fields, so protocol mappers, redirect URIs, flows, groups, roles and users # are all untouched by this write. st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid), headers=hdrs, body={"clientId": client.get("clientId"), "secret": want[cid]}) if not 200 <= st < 300: problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160])) continue st, text = _http("GET", base, "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid), headers=hdrs) readback = (json.loads(text).get("value") or "") if st == 200 else "" if not hmac.compare_digest(readback, want[cid]): problems.append("%s: PUT was accepted but the readback still does not match" % cid) continue changed += 1 log("%-14s REPAIRED and verified by readback" % cid) return changed, problems def run(argv, log): audit = "--audit" in argv base = os.environ.get("KEYCLOAK_BASE_URL", "").strip() realm = os.environ.get("KEYCLOAK_REALM", "").strip() user = os.environ.get("KEYCLOAK_ADMIN_USER", "") password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "") for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm), ("KEYCLOAK_ADMIN_USER", user), ("KEYCLOAK_ADMIN_PASSWORD", password)): if not value: log("FATAL: %s is empty" % name) return 1 log("mode=%s realm=%s base=%s" % ("audit (read-only)" if audit else "reconcile", realm, base)) try: want = declared(SECRETS_DIR) except OSError as e: log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__)) return 1 if not want: log("FATAL: no `%s` keys found in %s -- the keycloak-realm-clients " "Secret is missing or empty, and reporting success here would repeat the " "very fail-open the realm import already has" % (KEY_SUFFIX, SECRETS_DIR)) return 1 log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want)))) try: token = admin_token(base, user, password, log) changed, problems = reconcile(base, realm, token, want, audit, log) except Refused as e: log("FATAL: %s" % e) return 1 except (RuntimeError, OSError, ValueError) as e: log("FATAL: %s: %s" % (e.__class__.__name__, e)) return 1 if problems: log("") for p in problems: log("PROBLEM %s" % p) log("") if audit: log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems)) else: log("RECONCILE FAIL: %d client secret(s) could not be reconciled" % len(problems)) return 1 if audit: log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match " "git" % len(want)) else: log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired" % (len(want), changed)) return 0 # ============================================================================= # --selftest -- the offline behavioural proof. It drives the REAL code paths above # against an in-process stub Keycloak, so what the gate proves is what ships. # ============================================================================= def _refused(path): """True iff _http() refuses `path` outright (allow-list, safety property 4).""" try: _http("PUT", "http://127.0.0.1:1", path, body={}) except Refused: return True except OSError: return False return False def _selftest(): import contextlib import http.server import io import tempfile import threading global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" SECRET_C = "cccccccccccccccccccccccccccccccc" STALE = "0ld-r0tated-value" ADMIN_PW = "stub-admin-password" # The exact value the broken clusters stored. chr(36) is the dollar sign, # spelled this way so the file stays free of the character envsubst reacts to. LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)" state = {"clients": {}, "writes": [], "touched": []} class Stub(http.server.BaseHTTPRequestHandler): def log_message(self, *a): pass def _send(self, code, obj): payload = json.dumps(obj).encode() self.send_response(code) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(payload))) self.end_headers() self.wfile.write(payload) def _route(self, method): parsed = urllib.parse.urlparse(self.path) path, qs = parsed.path, urllib.parse.parse_qs(parsed.query) state["touched"].append("%s %s" % (method, path)) if path == "/realms/master/protocol/openid-connect/token": return self._send(200, {"access_token": "stub-token"}) if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path): cid = (qs.get("clientId") or [""])[0] c = state["clients"].get(cid) return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)]) m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret", path) if m and method == "GET": c = state["clients"].get(m.group(1)) if c is None: return self._send(404, {"error": "not found"}) return self._send(200, {"type": "secret", "value": c.get("secret", "")}) m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path) if m and method == "PUT": raw = self.rfile.read(int(self.headers.get("Content-Length") or 0)) rep = json.loads(raw or b"{}") # The live server rejects a full round-tripped representation with # 400 "Cannot parse the JSON"; modelled so a regression to the fat # PUT fails here rather than against a real realm. if set(rep) != {"clientId", "secret"}: return self._send(400, { "error": "unknown_error", "error_description": "Cannot parse the JSON"}) state["writes"].append(m.group(1)) state["clients"][m.group(1)]["secret"] = rep["secret"] return self._send(204, {}) return self._send(500, {"error": "an endpoint the reconciler must not touch"}) def do_GET(self): self._route("GET") def do_PUT(self): self._route("PUT") def do_POST(self): self._route("POST") srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub) threading.Thread(target=srv.serve_forever, daemon=True).start() base = "http://127.0.0.1:%d" % srv.server_address[1] tmp = tempfile.mkdtemp() for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B), ("kubernetes", SECRET_C), ("gitea", SECRET_A)): with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh: fh.write(secret + "\n") # trailing newline: must be tolerated # Kubernetes' own projection entries must be ignored, not read as clients. os.mkdir(os.path.join(tmp, "..data")) SECRETS_DIR = tmp READY_ATTEMPTS, READY_SLEEP = 2, 0.01 os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab", "KEYCLOAK_ADMIN_USER": "admin", "KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW}) failures = [] checks = [0] def expect(cond, what): checks[0] += 1 if not cond: failures.append(what) def invoke(*argv): state["writes"] = [] state["touched"] = [] buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(list(argv), print) return rc, buf.getvalue() def reset(): state["clients"] = { # already correct "grafana": {"clientId": "grafana", "publicClient": False, "secret": SECRET_A}, # THE INCIDENT: the literal, unresolved import placeholder "pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL}, # public client: no secret, must be skipped "kubernetes": {"clientId": "kubernetes", "publicClient": True}, # ordinary drift (rotated in git, stale in the realm) "gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE}, } def leaked(text): return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW) if s in text] # -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING reset() rc, out = invoke("--audit") expect(rc == 1, "audit must exit 1 when a stored secret differs from git") expect(state["writes"] == [], "audit mode must never write (writes=%r)" % state["writes"]) expect("UNRESOLVED IMPORT PLACEHOLDER" in out, "audit must name the unresolved-placeholder case explicitly") expect("pihole: DRIFT" in out and "gitea: DRIFT" in out, "audit must report BOTH drifted clients, not stop at the first") expect(re.search(r"grafana\s+ok \(already correct", out), "an already-correct client must be reported as such") expect(re.search(r"kubernetes\s+skipped \(public client", out), "a public client must be skipped, not treated as drift") expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out)) # -- 2. reconcile: repairs exactly the drifted clients, verified by readback reset() rc, out = invoke() expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:]) expect(sorted(state["writes"]) == ["gitea", "pihole"], "reconcile must write exactly the drifted clients, got %r" % state["writes"]) expect("grafana" not in state["writes"], "compare-then-write: an already-correct client must not be written") expect("kubernetes" not in state["writes"], "a public client must not be written") expect("REPAIRED and verified by readback" in out, "every repair must be readback-verified") expect("2 repaired" in out, "the summary must count the repairs") expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out)) # -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing rc, out = invoke() expect(rc == 0, "the second reconcile must exit 0") expect(state["writes"] == [], "IDEMPOTENCE: a converged realm must take ZERO writes, got %r" % state["writes"]) expect("0 repaired" in out, "the second run must report 0 repaired") rc, out = invoke("--audit") expect(rc == 0, "audit must pass once the realm has converged") expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out, "a clean audit must emit the sentinel the L2 conformance test greps for") # -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched reset() invoke() for touched in state["touched"]: expect(re.fullmatch( r"POST /realms/master/protocol/openid-connect/token" r"|GET /admin/realms/homelab/clients" r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?", touched), "the reconciler touched an off-list endpoint: %s" % touched) expect(not [t for t in state["touched"] if "/users" in t or "/groups" in t or "/roles" in t or "/authentication" in t], "the reconciler must never touch users/groups/roles/authentication") expect(_refused("/admin/realms/homelab/users"), "the allow-list must refuse a users endpoint before opening a socket") expect(_refused("/admin/realms/homelab/groups"), "the allow-list must refuse a groups endpoint before opening a socket") expect(_refused("/admin/realms/homelab"), "the allow-list must refuse a whole-realm update endpoint") # -- 5. a declared client the realm never got is LOUD, not a silent pass reset() del state["clients"]["gitea"] rc, out = invoke("--audit") expect(rc == 1, "a declared client missing from the realm must fail the audit") expect("NOT FOUND" in out, "the missing client must be named in the output") # -- 6. an empty / unmounted Secret must fail closed, never report success SECRETS_DIR = tempfile.mkdtemp() rc, out = invoke() expect(rc == 1, "an empty secrets dir must fail closed") expect("keys found in" in out, "the empty-secret failure must say what is missing") SECRETS_DIR = tmp srv.shutdown() print("") if failures: print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0])) for f in failures: print(" FAIL %s" % f) return 1 print("SELFTEST OK: %d assertions (compare-then-write, idempotence, " "no-secret-logging, endpoint allow-list, fail-closed)" % checks[0]) return 0 def main(): argv = sys.argv[1:] if "--selftest" in argv: return _selftest() return run(argv, lambda m: print(m, flush=True)) if __name__ == "__main__": sys.exit(main()) --- # CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key # added after the realm import already completed) converges within one interval # instead of waiting for a human to notice a failed login. Runs are cheap: a # converged realm performs 12 GETs and ZERO writes. apiVersion: batch/v1 kind: CronJob metadata: name: keycloak-client-secret-reconciler namespace: keycloak labels: app.kubernetes.io/name: keycloak-client-secret-reconciler app.kubernetes.io/part-of: keycloak spec: schedule: "*/15 * * * *" # Forbid: a run that is still waiting for Keycloak must not be lapped by the next # tick. Two concurrent reconcilers would be harmless (both converge to the same # value) but would double the write attempts during an outage recovery. concurrencyPolicy: Forbid startingDeadlineSeconds: 300 successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 3 jobTemplate: spec: backoffLimit: 2 activeDeadlineSeconds: 600 ttlSecondsAfterFinished: 3600 template: metadata: labels: app.kubernetes.io/name: keycloak-client-secret-reconciler app.kubernetes.io/part-of: keycloak spec: restartPolicy: Never # The reconciler talks ONLY to the Keycloak admin API. It needs no # Kubernetes API access at all — its two inputs are mounted Secrets — so # it gets no ServiceAccount token (R-SEC-02, least privilege). automountServiceAccountToken: false securityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 # fsGroup makes the projected Secret volume GROUP-owned by 65532. Without it the # files are owned by root and mode 0400 (owner-only) is unreadable to runAsUser # 65532 — the container died with "cannot read /etc/keycloak-realm-clients # (PermissionError)" on every run. fsGroup + 0440 keeps the secret unreadable to # other users rather than relaxing it to world-readable 0444. fsGroup: 65532 seccompProfile: type: RuntimeDefault containers: - name: reconciler # Stock upstream python; the reconciler is pure standard library, so # nothing is installed at runtime. Pinned exactly, like busybox / # curlimages elsewhere in the repo. Renovate does not manage plain # `image:` tags (renovate.json disables image-tag updates so it never # fights Flux image-automation), so this is bumped by hand. image: python:3.13.15-alpine3.23 imagePullPolicy: IfNotPresent command: - python3 - /opt/reconciler/reconcile-client-secrets.py env: - name: KEYCLOAK_BASE_URL value: http://keycloak-service.keycloak.svc.cluster.local:8080 - name: KEYCLOAK_REALM value: homelab # readOnlyRootFilesystem is on; keep CPython from trying to write # bytecode caches beside the read-only ConfigMap mount. - name: PYTHONDONTWRITEBYTECODE value: "1" - name: PYTHONUNBUFFERED value: "1" # Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to # bootstrap the admin; keys admin-user / admin-password. - name: KEYCLOAK_ADMIN_USER valueFrom: secretKeyRef: name: keycloak-admin-credentials key: admin-user - name: KEYCLOAK_ADMIN_PASSWORD valueFrom: secretKeyRef: name: keycloak-admin-credentials key: admin-password volumeMounts: - name: script mountPath: /opt/reconciler readOnly: true # The DESIRED state: one file per declared client, named # `-client-secret`. Mounting the Secret (rather than # reading it through the API) is what lets this pod run with no # ServiceAccount token at all. - name: client-secrets mountPath: /etc/keycloak-realm-clients readOnly: true securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: cpu: 10m memory: 48Mi limits: memory: 128Mi volumes: - name: script configMap: name: keycloak-client-secret-reconciler defaultMode: 0444 - name: client-secrets secret: secretName: keycloak-realm-clients # 0440 (not 0400): readable by fsGroup 65532, which the pod securityContext # sets. 0400 is owner-only and the owner is root, not the 65532 the container # runs as. See the fsGroup comment above. defaultMode: 0440