jakman85andClaude Opus 5
bc9c93d2ab
Shipped in the previous delivery and failing on every run since: FATAL: cannot read /etc/keycloak-realm-clients (PermissionError) BackoffLimitExceeded The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532. The sibling script volume worked only because it is 0444. So the healer that exists to correct client-secret drift was itself inert. It failed CLOSED (no writes attempted, no drift introduced), and the two alerts shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler Failing fires on the failed Jobs. The observability caught its own subject. Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable 0444: fsGroup makes the projected volume group-owned by the running user, so the secret stays unreadable to any other uid in the container. Co-Authored-By: Claude Opus 5 <[email protected]>
730 lines
36 KiB
YAML
730 lines
36 KiB
YAML
---
|
|
# =============================================================================
|
|
# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04)
|
|
#
|
|
# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only
|
|
# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients
|
|
# Secret and the Keycloak Operator, and alongside the realm import itself.
|
|
#
|
|
# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06)
|
|
# -------------------------------------------------------------------
|
|
# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are
|
|
# import placeholders resolved from the keycloak-realm-clients Secret via
|
|
# spec.placeholders. When the operator cannot resolve a placeholder it stores the
|
|
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
|
|
# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and
|
|
# NEVER RE-RUNS, so nothing ever corrects it:
|
|
#
|
|
# * prod: all 12 clients held the literal placeholder. Every OIDC login on the
|
|
# cluster failed with "Invalid client or Invalid client credentials".
|
|
# * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added
|
|
# AFTER dev's realm import had already completed, and a completed import is
|
|
# never re-run, so the new key was never picked up.
|
|
#
|
|
# Both were repaired live via the admin API on 2026-09-06 and verified. This
|
|
# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is
|
|
# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same
|
|
# posture Flux gives every other resource in this repo (R-GITOPS-03 drift
|
|
# correction, applied to the one piece of state Flux itself cannot reach --
|
|
# Keycloak's own database).
|
|
#
|
|
# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph
|
|
# already guarantees the Secret is applied before the realm import (the Secret is
|
|
# rendered by ./platform; the import by ./platform-config, which dependsOn
|
|
# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of
|
|
# the Secret lands before revision N of the import). Ordering still cannot help,
|
|
# because the operator will not re-run a completed import when a LATER revision
|
|
# adds a key -- which is precisely how dev broke. Only a reconciler closes that.
|
|
#
|
|
# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state
|
|
# lives in Keycloak's Postgres database, not in a Kubernetes object, so no
|
|
# admission or mutation policy can see it. The admin REST API is the only
|
|
# interface to it.
|
|
#
|
|
# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token,
|
|
# client lookup, client-secret read, minimal client PUT). Users, groups, roles,
|
|
# authentication flows and realm settings are structurally unreachable from it --
|
|
# prod's `homelab` realm carries a real user and the groups media-users /
|
|
# platform-admins / viewers, and they must survive every run. It never logs a
|
|
# secret, and it writes only where the stored value differs from git.
|
|
#
|
|
# PROOF: the script's own `--selftest` drives these real code paths against an
|
|
# in-process stub Keycloak and asserts compare-then-write, idempotence (a second
|
|
# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint
|
|
# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script
|
|
# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships:
|
|
# conformance/tests/l0-static/keycloak-client-secret-reconciler/.
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: keycloak-client-secret-reconciler
|
|
namespace: keycloak
|
|
labels:
|
|
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
|
app.kubernetes.io/part-of: keycloak
|
|
data:
|
|
# SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy:
|
|
# the L0 gate extracts THIS block and executes its --selftest, so a second copy
|
|
# would be untested. Contains no dollar sign at all, on purpose -- this file is
|
|
# rendered by a Flux Kustomization with postBuild.substituteFrom, so an
|
|
# unescaped dollar-brace expansion here would be rewritten before any container
|
|
# ran (the trap documented at length in apps/wave-1/nextcloud.yaml).
|
|
reconcile-client-secrets.py: |
|
|
#!/usr/bin/env python3
|
|
# =============================================================================
|
|
# reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret
|
|
# equal the value git declares for it, idempotently, without touching anything
|
|
# else in the realm.
|
|
#
|
|
# WHY THIS EXISTS (the incident it prevents)
|
|
# ------------------------------------------
|
|
# keycloak-realm-import.yaml declares each confidential client's secret as an
|
|
# `(env:<NAME>_CLIENT_SECRET)` import placeholder and maps <NAME> to a key of the
|
|
# SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the
|
|
# Keycloak Operator cannot resolve a placeholder at import time it stores the
|
|
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
|
|
# SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and
|
|
# never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12
|
|
# clients with secrets held the literal placeholder and every OIDC login failed
|
|
# with "Invalid client or Invalid client credentials"; on dev exactly one
|
|
# (oauth2-proxy) was broken, because that Secret key was added after dev's import
|
|
# had already completed.
|
|
#
|
|
# So the import is a ONE-SHOT that fails OPEN. This is the level-triggered
|
|
# reconciler that closes it.
|
|
#
|
|
# CONTRACT
|
|
# --------
|
|
# Desired state files in SECRETS_DIR named `<clientId>-client-secret`, each
|
|
# holding exactly the secret for that clientId (this is the
|
|
# `keycloak-realm-clients` Secret, mounted).
|
|
# Actual state GET /admin/realms/<realm>/clients/<uuid>/client-secret
|
|
# Action only where they differ: PUT /admin/realms/<realm>/clients/<uuid>
|
|
# with the MINIMAL representation {"clientId":..,"secret":..}.
|
|
#
|
|
# WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a
|
|
# two-field PUT rewrites the secret and nothing else. PUTting the full
|
|
# round-tripped representation back has a far larger blast radius and is in fact
|
|
# rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the
|
|
# JSON"} (measured during the live repair). The stub in --selftest models that
|
|
# rejection, so a regression to the fat PUT fails in CI, not on a live realm.
|
|
#
|
|
# SAFETY PROPERTIES, each enforced in code and proved by --selftest:
|
|
# 1. compare-then-write: a client already correct is SKIPPED, no PUT issued.
|
|
# 2. idempotent: a second run over a converged realm issues zero writes.
|
|
# 3. no secret is ever printed, in any mode, on any path.
|
|
# 4. the ONLY endpoints this program may touch are the master-realm token
|
|
# endpoint, the client LOOKUP, the client-secret READ and the minimal client
|
|
# PUT. Every other URL is refused by _http() before a socket is opened, so
|
|
# users, groups, roles, authentication flows and realm settings are
|
|
# structurally unreachable from here -- prod's `homelab` realm carries a
|
|
# real user and the groups media-users / platform-admins / viewers, and they
|
|
# MUST survive.
|
|
# 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped.
|
|
#
|
|
# MODES
|
|
# (default) reconcile: compare, and write only what differs.
|
|
# --audit READ-ONLY. Report drift and exit 1 if any client's stored secret
|
|
# differs from the declared one. Never writes. This is what the L2
|
|
# conformance test runs.
|
|
# --selftest the offline behavioural proof, against an in-process stub
|
|
# Keycloak. No cluster; loopback only. This is what the L0 static
|
|
# gate runs, so the code the gate proves is the code that ships.
|
|
#
|
|
# NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor
|
|
# (hence re.fullmatch throughout), and not even inside these comments. It ships
|
|
# inside a ConfigMap that the `platform-config` Flux Kustomization renders with
|
|
# postBuild.substituteFrom, so Flux envsubst rewrites this text before any
|
|
# container runs: an unescaped dollar-brace expansion would be silently emptied at
|
|
# reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's
|
|
# OIDC guard dead code for months -- read the comment block there. Python needs no
|
|
# dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and
|
|
# the character is banned outright so no future edit has to reason about escaping.
|
|
# Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh.
|
|
# =============================================================================
|
|
from __future__ import annotations
|
|
|
|
import hmac
|
|
import json
|
|
import os
|
|
import re
|
|
import sys
|
|
import time
|
|
import urllib.error
|
|
import urllib.parse
|
|
import urllib.request
|
|
|
|
SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients")
|
|
KEY_SUFFIX = "-client-secret"
|
|
TIMEOUT = 20
|
|
READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24"))
|
|
READY_SLEEP = float(os.environ.get("READY_SLEEP", "5"))
|
|
|
|
# Endpoint allow-list (safety property 4). fullmatch-ed against the path of every
|
|
# URL _http() is asked to open; anything else raises before a socket exists.
|
|
_ALLOWED = (
|
|
r"/realms/master/protocol/openid-connect/token",
|
|
r"/admin/realms/[^/]+/clients",
|
|
r"/admin/realms/[^/]+/clients/[^/]+",
|
|
r"/admin/realms/[^/]+/clients/[^/]+/client-secret",
|
|
)
|
|
|
|
|
|
class Refused(RuntimeError):
|
|
"""A URL outside the allow-list was requested. Never expected in a normal run."""
|
|
|
|
|
|
def _http(method, base, path, query=None, headers=None, form=None, body=None):
|
|
"""Return (status, text). Refuses any path outside the allow-list."""
|
|
if not any(re.fullmatch(p, path) for p in _ALLOWED):
|
|
raise Refused("endpoint not on the allow-list: %s %s" % (method, path))
|
|
url = base.rstrip("/") + path
|
|
if query:
|
|
url = url + "?" + urllib.parse.urlencode(query)
|
|
hdrs = dict(headers or {})
|
|
data = None
|
|
if form is not None:
|
|
data = urllib.parse.urlencode(form).encode()
|
|
hdrs["Content-Type"] = "application/x-www-form-urlencoded"
|
|
elif body is not None:
|
|
data = json.dumps(body).encode()
|
|
hdrs["Content-Type"] = "application/json"
|
|
req = urllib.request.Request(url, data=data, headers=hdrs, method=method)
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
|
|
return resp.status, resp.read().decode("utf-8", "replace")
|
|
except urllib.error.HTTPError as e:
|
|
return e.code, e.read().decode("utf-8", "replace")
|
|
|
|
|
|
def declared(secrets_dir):
|
|
"""{clientId: secret} from the mounted Secret.
|
|
|
|
Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal
|
|
entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A
|
|
key must be named `<clientId>-client-secret`: that naming contract is what
|
|
lets this reconciler derive the client set from git without a second
|
|
declaration, and the L0 static check enforces it against
|
|
keycloak-realm-import.yaml so the two can never drift apart.
|
|
"""
|
|
out = {}
|
|
for name in sorted(os.listdir(secrets_dir)):
|
|
if name.startswith(".") or not name.endswith(KEY_SUFFIX):
|
|
continue
|
|
path = os.path.join(secrets_dir, name)
|
|
if not os.path.isfile(path):
|
|
continue
|
|
with open(path, "r", encoding="utf-8") as fh:
|
|
value = fh.read()
|
|
# A trailing newline in the SOPS source would otherwise become part of the
|
|
# secret and mismatch forever. Keycloak secrets never contain whitespace.
|
|
out[name[: -len(KEY_SUFFIX)]] = value.strip()
|
|
return out
|
|
|
|
|
|
def admin_token(base, user, password, log):
|
|
"""Bearer token for admin-cli in the master realm, with a bounded readiness wait."""
|
|
last = ""
|
|
for attempt in range(1, READY_ATTEMPTS + 1):
|
|
try:
|
|
st, text = _http(
|
|
"POST", base, "/realms/master/protocol/openid-connect/token",
|
|
form={"client_id": "admin-cli", "grant_type": "password",
|
|
"username": user, "password": password})
|
|
except OSError as e:
|
|
st, text = 0, "connection error (%s)" % e.__class__.__name__
|
|
if st == 200:
|
|
token = json.loads(text).get("access_token")
|
|
if token:
|
|
return token
|
|
last = "http 200 but no access_token in the response"
|
|
else:
|
|
# Deliberately do NOT echo the response body: it is harmless today, but
|
|
# the request that produced it carried the admin password, and keeping
|
|
# that class of mistake impossible by habit is cheaper than reviewing it.
|
|
last = "http %s" % st
|
|
log("waiting for the Keycloak admin API (attempt %d/%d): %s"
|
|
% (attempt, READY_ATTEMPTS, last))
|
|
if attempt < READY_ATTEMPTS:
|
|
time.sleep(READY_SLEEP)
|
|
raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s"
|
|
% (READY_ATTEMPTS, last))
|
|
|
|
|
|
def reconcile(base, realm, token, want, audit, log):
|
|
"""Return (changed, problems). Writes only when audit is False."""
|
|
hdrs = {"Authorization": "Bearer " + token}
|
|
changed = 0
|
|
problems = []
|
|
for cid in sorted(want):
|
|
st, text = _http("GET", base, "/admin/realms/%s/clients" % realm,
|
|
query={"clientId": cid}, headers=hdrs)
|
|
if st != 200:
|
|
problems.append("%s: client lookup failed (http %s)" % (cid, st))
|
|
continue
|
|
found = json.loads(text)
|
|
if not isinstance(found, list) or not found:
|
|
problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it "
|
|
"but the realm has no such client (the operator runs the "
|
|
"realm import exactly once)" % (cid, realm))
|
|
continue
|
|
client = found[0]
|
|
uuid = client.get("id")
|
|
if client.get("publicClient"):
|
|
log("%-14s skipped (public client, holds no secret)" % cid)
|
|
continue
|
|
st, text = _http("GET", base,
|
|
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
|
|
headers=hdrs)
|
|
if st != 200:
|
|
problems.append("%s: could not read the stored secret (http %s)" % (cid, st))
|
|
continue
|
|
stored = json.loads(text).get("value") or ""
|
|
if hmac.compare_digest(stored, want[cid]):
|
|
log("%-14s ok (already correct, no write)" % cid)
|
|
continue
|
|
# Describe the drift WITHOUT quoting either value. The literal-placeholder
|
|
# case is named because it is the failure this program exists for, and
|
|
# saying so reveals nothing secret.
|
|
kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER"
|
|
if "env:" in stored else "stored value differs from the declared one")
|
|
if audit:
|
|
problems.append("%s: DRIFT -- %s" % (cid, kind))
|
|
continue
|
|
log("%-14s DRIFT (%s) -- repairing" % (cid, kind))
|
|
# MINIMAL representation only -- see the header. Keycloak merges non-null
|
|
# fields, so protocol mappers, redirect URIs, flows, groups, roles and users
|
|
# are all untouched by this write.
|
|
st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid),
|
|
headers=hdrs, body={"clientId": client.get("clientId"),
|
|
"secret": want[cid]})
|
|
if not 200 <= st < 300:
|
|
problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160]))
|
|
continue
|
|
st, text = _http("GET", base,
|
|
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
|
|
headers=hdrs)
|
|
readback = (json.loads(text).get("value") or "") if st == 200 else ""
|
|
if not hmac.compare_digest(readback, want[cid]):
|
|
problems.append("%s: PUT was accepted but the readback still does not match"
|
|
% cid)
|
|
continue
|
|
changed += 1
|
|
log("%-14s REPAIRED and verified by readback" % cid)
|
|
return changed, problems
|
|
|
|
|
|
def run(argv, log):
|
|
audit = "--audit" in argv
|
|
base = os.environ.get("KEYCLOAK_BASE_URL", "").strip()
|
|
realm = os.environ.get("KEYCLOAK_REALM", "").strip()
|
|
user = os.environ.get("KEYCLOAK_ADMIN_USER", "")
|
|
password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")
|
|
for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm),
|
|
("KEYCLOAK_ADMIN_USER", user),
|
|
("KEYCLOAK_ADMIN_PASSWORD", password)):
|
|
if not value:
|
|
log("FATAL: %s is empty" % name)
|
|
return 1
|
|
log("mode=%s realm=%s base=%s"
|
|
% ("audit (read-only)" if audit else "reconcile", realm, base))
|
|
try:
|
|
want = declared(SECRETS_DIR)
|
|
except OSError as e:
|
|
log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__))
|
|
return 1
|
|
if not want:
|
|
log("FATAL: no `<clientId>%s` keys found in %s -- the keycloak-realm-clients "
|
|
"Secret is missing or empty, and reporting success here would repeat the "
|
|
"very fail-open the realm import already has"
|
|
% (KEY_SUFFIX, SECRETS_DIR))
|
|
return 1
|
|
log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want))))
|
|
try:
|
|
token = admin_token(base, user, password, log)
|
|
changed, problems = reconcile(base, realm, token, want, audit, log)
|
|
except Refused as e:
|
|
log("FATAL: %s" % e)
|
|
return 1
|
|
except (RuntimeError, OSError, ValueError) as e:
|
|
log("FATAL: %s: %s" % (e.__class__.__name__, e))
|
|
return 1
|
|
if problems:
|
|
log("")
|
|
for p in problems:
|
|
log("PROBLEM %s" % p)
|
|
log("")
|
|
if audit:
|
|
log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems))
|
|
else:
|
|
log("RECONCILE FAIL: %d client secret(s) could not be reconciled"
|
|
% len(problems))
|
|
return 1
|
|
if audit:
|
|
log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match "
|
|
"git" % len(want))
|
|
else:
|
|
log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired"
|
|
% (len(want), changed))
|
|
return 0
|
|
|
|
|
|
# =============================================================================
|
|
# --selftest -- the offline behavioural proof. It drives the REAL code paths above
|
|
# against an in-process stub Keycloak, so what the gate proves is what ships.
|
|
# =============================================================================
|
|
def _refused(path):
|
|
"""True iff _http() refuses `path` outright (allow-list, safety property 4)."""
|
|
try:
|
|
_http("PUT", "http://127.0.0.1:1", path, body={})
|
|
except Refused:
|
|
return True
|
|
except OSError:
|
|
return False
|
|
return False
|
|
|
|
|
|
def _selftest():
|
|
import contextlib
|
|
import http.server
|
|
import io
|
|
import tempfile
|
|
import threading
|
|
|
|
global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP
|
|
|
|
SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
|
SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
|
SECRET_C = "cccccccccccccccccccccccccccccccc"
|
|
STALE = "0ld-r0tated-value"
|
|
ADMIN_PW = "stub-admin-password"
|
|
# The exact value the broken clusters stored. chr(36) is the dollar sign,
|
|
# spelled this way so the file stays free of the character envsubst reacts to.
|
|
LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)"
|
|
|
|
state = {"clients": {}, "writes": [], "touched": []}
|
|
|
|
class Stub(http.server.BaseHTTPRequestHandler):
|
|
def log_message(self, *a):
|
|
pass
|
|
|
|
def _send(self, code, obj):
|
|
payload = json.dumps(obj).encode()
|
|
self.send_response(code)
|
|
self.send_header("Content-Type", "application/json")
|
|
self.send_header("Content-Length", str(len(payload)))
|
|
self.end_headers()
|
|
self.wfile.write(payload)
|
|
|
|
def _route(self, method):
|
|
parsed = urllib.parse.urlparse(self.path)
|
|
path, qs = parsed.path, urllib.parse.parse_qs(parsed.query)
|
|
state["touched"].append("%s %s" % (method, path))
|
|
if path == "/realms/master/protocol/openid-connect/token":
|
|
return self._send(200, {"access_token": "stub-token"})
|
|
if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path):
|
|
cid = (qs.get("clientId") or [""])[0]
|
|
c = state["clients"].get(cid)
|
|
return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)])
|
|
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret",
|
|
path)
|
|
if m and method == "GET":
|
|
c = state["clients"].get(m.group(1))
|
|
if c is None:
|
|
return self._send(404, {"error": "not found"})
|
|
return self._send(200, {"type": "secret", "value": c.get("secret", "")})
|
|
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path)
|
|
if m and method == "PUT":
|
|
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
|
|
rep = json.loads(raw or b"{}")
|
|
# The live server rejects a full round-tripped representation with
|
|
# 400 "Cannot parse the JSON"; modelled so a regression to the fat
|
|
# PUT fails here rather than against a real realm.
|
|
if set(rep) != {"clientId", "secret"}:
|
|
return self._send(400, {
|
|
"error": "unknown_error",
|
|
"error_description": "Cannot parse the JSON"})
|
|
state["writes"].append(m.group(1))
|
|
state["clients"][m.group(1)]["secret"] = rep["secret"]
|
|
return self._send(204, {})
|
|
return self._send(500, {"error": "an endpoint the reconciler must not touch"})
|
|
|
|
def do_GET(self):
|
|
self._route("GET")
|
|
|
|
def do_PUT(self):
|
|
self._route("PUT")
|
|
|
|
def do_POST(self):
|
|
self._route("POST")
|
|
|
|
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub)
|
|
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
|
base = "http://127.0.0.1:%d" % srv.server_address[1]
|
|
|
|
tmp = tempfile.mkdtemp()
|
|
for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B),
|
|
("kubernetes", SECRET_C), ("gitea", SECRET_A)):
|
|
with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh:
|
|
fh.write(secret + "\n") # trailing newline: must be tolerated
|
|
# Kubernetes' own projection entries must be ignored, not read as clients.
|
|
os.mkdir(os.path.join(tmp, "..data"))
|
|
|
|
SECRETS_DIR = tmp
|
|
READY_ATTEMPTS, READY_SLEEP = 2, 0.01
|
|
os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab",
|
|
"KEYCLOAK_ADMIN_USER": "admin",
|
|
"KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW})
|
|
|
|
failures = []
|
|
checks = [0]
|
|
|
|
def expect(cond, what):
|
|
checks[0] += 1
|
|
if not cond:
|
|
failures.append(what)
|
|
|
|
def invoke(*argv):
|
|
state["writes"] = []
|
|
state["touched"] = []
|
|
buf = io.StringIO()
|
|
with contextlib.redirect_stdout(buf):
|
|
rc = run(list(argv), print)
|
|
return rc, buf.getvalue()
|
|
|
|
def reset():
|
|
state["clients"] = {
|
|
# already correct
|
|
"grafana": {"clientId": "grafana", "publicClient": False,
|
|
"secret": SECRET_A},
|
|
# THE INCIDENT: the literal, unresolved import placeholder
|
|
"pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL},
|
|
# public client: no secret, must be skipped
|
|
"kubernetes": {"clientId": "kubernetes", "publicClient": True},
|
|
# ordinary drift (rotated in git, stale in the realm)
|
|
"gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE},
|
|
}
|
|
|
|
def leaked(text):
|
|
return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW)
|
|
if s in text]
|
|
|
|
# -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING
|
|
reset()
|
|
rc, out = invoke("--audit")
|
|
expect(rc == 1, "audit must exit 1 when a stored secret differs from git")
|
|
expect(state["writes"] == [],
|
|
"audit mode must never write (writes=%r)" % state["writes"])
|
|
expect("UNRESOLVED IMPORT PLACEHOLDER" in out,
|
|
"audit must name the unresolved-placeholder case explicitly")
|
|
expect("pihole: DRIFT" in out and "gitea: DRIFT" in out,
|
|
"audit must report BOTH drifted clients, not stop at the first")
|
|
expect(re.search(r"grafana\s+ok \(already correct", out),
|
|
"an already-correct client must be reported as such")
|
|
expect(re.search(r"kubernetes\s+skipped \(public client", out),
|
|
"a public client must be skipped, not treated as drift")
|
|
expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out))
|
|
|
|
# -- 2. reconcile: repairs exactly the drifted clients, verified by readback
|
|
reset()
|
|
rc, out = invoke()
|
|
expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:])
|
|
expect(sorted(state["writes"]) == ["gitea", "pihole"],
|
|
"reconcile must write exactly the drifted clients, got %r" % state["writes"])
|
|
expect("grafana" not in state["writes"],
|
|
"compare-then-write: an already-correct client must not be written")
|
|
expect("kubernetes" not in state["writes"], "a public client must not be written")
|
|
expect("REPAIRED and verified by readback" in out,
|
|
"every repair must be readback-verified")
|
|
expect("2 repaired" in out, "the summary must count the repairs")
|
|
expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out))
|
|
|
|
# -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing
|
|
rc, out = invoke()
|
|
expect(rc == 0, "the second reconcile must exit 0")
|
|
expect(state["writes"] == [],
|
|
"IDEMPOTENCE: a converged realm must take ZERO writes, got %r"
|
|
% state["writes"])
|
|
expect("0 repaired" in out, "the second run must report 0 repaired")
|
|
rc, out = invoke("--audit")
|
|
expect(rc == 0, "audit must pass once the realm has converged")
|
|
expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out,
|
|
"a clean audit must emit the sentinel the L2 conformance test greps for")
|
|
|
|
# -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched
|
|
reset()
|
|
invoke()
|
|
for touched in state["touched"]:
|
|
expect(re.fullmatch(
|
|
r"POST /realms/master/protocol/openid-connect/token"
|
|
r"|GET /admin/realms/homelab/clients"
|
|
r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?",
|
|
touched), "the reconciler touched an off-list endpoint: %s" % touched)
|
|
expect(not [t for t in state["touched"]
|
|
if "/users" in t or "/groups" in t or "/roles" in t
|
|
or "/authentication" in t],
|
|
"the reconciler must never touch users/groups/roles/authentication")
|
|
expect(_refused("/admin/realms/homelab/users"),
|
|
"the allow-list must refuse a users endpoint before opening a socket")
|
|
expect(_refused("/admin/realms/homelab/groups"),
|
|
"the allow-list must refuse a groups endpoint before opening a socket")
|
|
expect(_refused("/admin/realms/homelab"),
|
|
"the allow-list must refuse a whole-realm update endpoint")
|
|
|
|
# -- 5. a declared client the realm never got is LOUD, not a silent pass
|
|
reset()
|
|
del state["clients"]["gitea"]
|
|
rc, out = invoke("--audit")
|
|
expect(rc == 1, "a declared client missing from the realm must fail the audit")
|
|
expect("NOT FOUND" in out, "the missing client must be named in the output")
|
|
|
|
# -- 6. an empty / unmounted Secret must fail closed, never report success
|
|
SECRETS_DIR = tempfile.mkdtemp()
|
|
rc, out = invoke()
|
|
expect(rc == 1, "an empty secrets dir must fail closed")
|
|
expect("keys found in" in out, "the empty-secret failure must say what is missing")
|
|
SECRETS_DIR = tmp
|
|
|
|
srv.shutdown()
|
|
print("")
|
|
if failures:
|
|
print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0]))
|
|
for f in failures:
|
|
print(" FAIL %s" % f)
|
|
return 1
|
|
print("SELFTEST OK: %d assertions (compare-then-write, idempotence, "
|
|
"no-secret-logging, endpoint allow-list, fail-closed)" % checks[0])
|
|
return 0
|
|
|
|
|
|
def main():
|
|
argv = sys.argv[1:]
|
|
if "--selftest" in argv:
|
|
return _selftest()
|
|
return run(argv, lambda m: print(m, flush=True))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|
|
---
|
|
# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key
|
|
# added after the realm import already completed) converges within one interval
|
|
# instead of waiting for a human to notice a failed login. Runs are cheap: a
|
|
# converged realm performs 12 GETs and ZERO writes.
|
|
apiVersion: batch/v1
|
|
kind: CronJob
|
|
metadata:
|
|
name: keycloak-client-secret-reconciler
|
|
namespace: keycloak
|
|
labels:
|
|
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
|
app.kubernetes.io/part-of: keycloak
|
|
spec:
|
|
schedule: "*/15 * * * *"
|
|
# Forbid: a run that is still waiting for Keycloak must not be lapped by the next
|
|
# tick. Two concurrent reconcilers would be harmless (both converge to the same
|
|
# value) but would double the write attempts during an outage recovery.
|
|
concurrencyPolicy: Forbid
|
|
startingDeadlineSeconds: 300
|
|
successfulJobsHistoryLimit: 1
|
|
failedJobsHistoryLimit: 3
|
|
jobTemplate:
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 600
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
|
app.kubernetes.io/part-of: keycloak
|
|
spec:
|
|
restartPolicy: Never
|
|
# The reconciler talks ONLY to the Keycloak admin API. It needs no
|
|
# Kubernetes API access at all — its two inputs are mounted Secrets — so
|
|
# it gets no ServiceAccount token (R-SEC-02, least privilege).
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 65532
|
|
runAsGroup: 65532
|
|
# fsGroup makes the projected Secret volume GROUP-owned by 65532. Without it the
|
|
# files are owned by root and mode 0400 (owner-only) is unreadable to runAsUser
|
|
# 65532 — the container died with "cannot read /etc/keycloak-realm-clients
|
|
# (PermissionError)" on every run. fsGroup + 0440 keeps the secret unreadable to
|
|
# other users rather than relaxing it to world-readable 0444.
|
|
fsGroup: 65532
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: reconciler
|
|
# Stock upstream python; the reconciler is pure standard library, so
|
|
# nothing is installed at runtime. Pinned exactly, like busybox /
|
|
# curlimages elsewhere in the repo. Renovate does not manage plain
|
|
# `image:` tags (renovate.json disables image-tag updates so it never
|
|
# fights Flux image-automation), so this is bumped by hand.
|
|
image: python:3.13.15-alpine3.23
|
|
imagePullPolicy: IfNotPresent
|
|
command:
|
|
- python3
|
|
- /opt/reconciler/reconcile-client-secrets.py
|
|
env:
|
|
- name: KEYCLOAK_BASE_URL
|
|
value: http://keycloak-service.keycloak.svc.cluster.local:8080
|
|
- name: KEYCLOAK_REALM
|
|
value: homelab
|
|
# readOnlyRootFilesystem is on; keep CPython from trying to write
|
|
# bytecode caches beside the read-only ConfigMap mount.
|
|
- name: PYTHONDONTWRITEBYTECODE
|
|
value: "1"
|
|
- name: PYTHONUNBUFFERED
|
|
value: "1"
|
|
# Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to
|
|
# bootstrap the admin; keys admin-user / admin-password.
|
|
- name: KEYCLOAK_ADMIN_USER
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: keycloak-admin-credentials
|
|
key: admin-user
|
|
- name: KEYCLOAK_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: keycloak-admin-credentials
|
|
key: admin-password
|
|
volumeMounts:
|
|
- name: script
|
|
mountPath: /opt/reconciler
|
|
readOnly: true
|
|
# The DESIRED state: one file per declared client, named
|
|
# `<clientId>-client-secret`. Mounting the Secret (rather than
|
|
# reading it through the API) is what lets this pod run with no
|
|
# ServiceAccount token at all.
|
|
- name: client-secrets
|
|
mountPath: /etc/keycloak-realm-clients
|
|
readOnly: true
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 48Mi
|
|
limits:
|
|
memory: 128Mi
|
|
volumes:
|
|
- name: script
|
|
configMap:
|
|
name: keycloak-client-secret-reconciler
|
|
defaultMode: 0444
|
|
- name: client-secrets
|
|
secret:
|
|
secretName: keycloak-realm-clients
|
|
# 0440 (not 0400): readable by fsGroup 65532, which the pod securityContext
|
|
# sets. 0400 is owner-only and the owner is root, not the 65532 the container
|
|
# runs as. See the fsGroup comment above.
|
|
defaultMode: 0440
|