Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Browse Source 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

platform/keycloak/README.md
+48 -4
View File
@@ -29,6 +29,7 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
@@ -47,10 +48,14 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
## Operator hand-offs (fill before / during deploy)
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. Two client
secrets (`grafana-client-secret`, `oauth2-proxy-client-secret`), `openssl rand -hex 32` each. The
realm import injects them into the Keycloak clients via `spec.placeholders`. Add to
`kustomization.yaml` once filled. The realm import stays PENDING until this exists.
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
`<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
import injects them into the Keycloak clients via `spec.placeholders`. Add to
`kustomization.yaml` once filled.
**The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
from it, and CI rejects any other name.
*(This step used to read "the realm import stays PENDING until this exists". It does not — see
the next section. That false belief is what allowed the 2026-09-06 outage.)*
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
`client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
@@ -60,6 +65,45 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
exact keys + steps. No plaintext secret is ever committed.
## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
**The realm import is a one-shot that fails open.** Measured on both clusters:
* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
error condition, and `HasErrors` stays `False`.
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
added later is never picked up.
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
signal was green throughout, including the conformance test that asserts `Done=True`.
Three layers now cover it:
| Layer | Where | What it does |
|---|---|---|
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
**Manual repair / one-off run** (the CronJob normally does this):
```sh
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
```
**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
landed.
## Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
platform/keycloak/client-secret-reconciler.yaml
+720
View File
@@ -0,0 +1,720 @@
---
# =============================================================================
# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04)
#
# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only
# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients
# Secret and the Keycloak Operator, and alongside the realm import itself.
#
# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06)
# -------------------------------------------------------------------
# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are
# import placeholders resolved from the keycloak-realm-clients Secret via
# spec.placeholders. When the operator cannot resolve a placeholder it stores the
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and
# NEVER RE-RUNS, so nothing ever corrects it:
#
# * prod: all 12 clients held the literal placeholder. Every OIDC login on the
# cluster failed with "Invalid client or Invalid client credentials".
# * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added
# AFTER dev's realm import had already completed, and a completed import is
# never re-run, so the new key was never picked up.
#
# Both were repaired live via the admin API on 2026-09-06 and verified. This
# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is
# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same
# posture Flux gives every other resource in this repo (R-GITOPS-03 drift
# correction, applied to the one piece of state Flux itself cannot reach --
# Keycloak's own database).
#
# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph
# already guarantees the Secret is applied before the realm import (the Secret is
# rendered by ./platform; the import by ./platform-config, which dependsOn
# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of
# the Secret lands before revision N of the import). Ordering still cannot help,
# because the operator will not re-run a completed import when a LATER revision
# adds a key -- which is precisely how dev broke. Only a reconciler closes that.
#
# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state
# lives in Keycloak's Postgres database, not in a Kubernetes object, so no
# admission or mutation policy can see it. The admin REST API is the only
# interface to it.
#
# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token,
# client lookup, client-secret read, minimal client PUT). Users, groups, roles,
# authentication flows and realm settings are structurally unreachable from it --
# prod's `homelab` realm carries a real user and the groups media-users /
# platform-admins / viewers, and they must survive every run. It never logs a
# secret, and it writes only where the stored value differs from git.
#
# PROOF: the script's own `--selftest` drives these real code paths against an
# in-process stub Keycloak and asserts compare-then-write, idempotence (a second
# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint
# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script
# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships:
# conformance/tests/l0-static/keycloak-client-secret-reconciler/.
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
data:
# SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy:
# the L0 gate extracts THIS block and executes its --selftest, so a second copy
# would be untested. Contains no dollar sign at all, on purpose -- this file is
# rendered by a Flux Kustomization with postBuild.substituteFrom, so an
# unescaped dollar-brace expansion here would be rewritten before any container
# ran (the trap documented at length in apps/wave-1/nextcloud.yaml).
reconcile-client-secrets.py: |
#!/usr/bin/env python3
# =============================================================================
# reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret
# equal the value git declares for it, idempotently, without touching anything
# else in the realm.
#
# WHY THIS EXISTS (the incident it prevents)
# ------------------------------------------
# keycloak-realm-import.yaml declares each confidential client's secret as an
# `(env:<NAME>_CLIENT_SECRET)` import placeholder and maps <NAME> to a key of the
# SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the
# Keycloak Operator cannot resolve a placeholder at import time it stores the
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
# SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and
# never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12
# clients with secrets held the literal placeholder and every OIDC login failed
# with "Invalid client or Invalid client credentials"; on dev exactly one
# (oauth2-proxy) was broken, because that Secret key was added after dev's import
# had already completed.
#
# So the import is a ONE-SHOT that fails OPEN. This is the level-triggered
# reconciler that closes it.
#
# CONTRACT
# --------
# Desired state files in SECRETS_DIR named `<clientId>-client-secret`, each
# holding exactly the secret for that clientId (this is the
# `keycloak-realm-clients` Secret, mounted).
# Actual state GET /admin/realms/<realm>/clients/<uuid>/client-secret
# Action only where they differ: PUT /admin/realms/<realm>/clients/<uuid>
# with the MINIMAL representation {"clientId":..,"secret":..}.
#
# WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a
# two-field PUT rewrites the secret and nothing else. PUTting the full
# round-tripped representation back has a far larger blast radius and is in fact
# rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the
# JSON"} (measured during the live repair). The stub in --selftest models that
# rejection, so a regression to the fat PUT fails in CI, not on a live realm.
#
# SAFETY PROPERTIES, each enforced in code and proved by --selftest:
# 1. compare-then-write: a client already correct is SKIPPED, no PUT issued.
# 2. idempotent: a second run over a converged realm issues zero writes.
# 3. no secret is ever printed, in any mode, on any path.
# 4. the ONLY endpoints this program may touch are the master-realm token
# endpoint, the client LOOKUP, the client-secret READ and the minimal client
# PUT. Every other URL is refused by _http() before a socket is opened, so
# users, groups, roles, authentication flows and realm settings are
# structurally unreachable from here -- prod's `homelab` realm carries a
# real user and the groups media-users / platform-admins / viewers, and they
# MUST survive.
# 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped.
#
# MODES
# (default) reconcile: compare, and write only what differs.
# --audit READ-ONLY. Report drift and exit 1 if any client's stored secret
# differs from the declared one. Never writes. This is what the L2
# conformance test runs.
# --selftest the offline behavioural proof, against an in-process stub
# Keycloak. No cluster; loopback only. This is what the L0 static
# gate runs, so the code the gate proves is the code that ships.
#
# NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor
# (hence re.fullmatch throughout), and not even inside these comments. It ships
# inside a ConfigMap that the `platform-config` Flux Kustomization renders with
# postBuild.substituteFrom, so Flux envsubst rewrites this text before any
# container runs: an unescaped dollar-brace expansion would be silently emptied at
# reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's
# OIDC guard dead code for months -- read the comment block there. Python needs no
# dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and
# the character is banned outright so no future edit has to reason about escaping.
# Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh.
# =============================================================================
from __future__ import annotations
import hmac
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients")
KEY_SUFFIX = "-client-secret"
TIMEOUT = 20
READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24"))
READY_SLEEP = float(os.environ.get("READY_SLEEP", "5"))
# Endpoint allow-list (safety property 4). fullmatch-ed against the path of every
# URL _http() is asked to open; anything else raises before a socket exists.
_ALLOWED = (
r"/realms/master/protocol/openid-connect/token",
r"/admin/realms/[^/]+/clients",
r"/admin/realms/[^/]+/clients/[^/]+",
r"/admin/realms/[^/]+/clients/[^/]+/client-secret",
)
class Refused(RuntimeError):
"""A URL outside the allow-list was requested. Never expected in a normal run."""
def _http(method, base, path, query=None, headers=None, form=None, body=None):
"""Return (status, text). Refuses any path outside the allow-list."""
if not any(re.fullmatch(p, path) for p in _ALLOWED):
raise Refused("endpoint not on the allow-list: %s %s" % (method, path))
url = base.rstrip("/") + path
if query:
url = url + "?" + urllib.parse.urlencode(query)
hdrs = dict(headers or {})
data = None
if form is not None:
data = urllib.parse.urlencode(form).encode()
hdrs["Content-Type"] = "application/x-www-form-urlencoded"
elif body is not None:
data = json.dumps(body).encode()
hdrs["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=hdrs, method=method)
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
return resp.status, resp.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, e.read().decode("utf-8", "replace")
def declared(secrets_dir):
"""{clientId: secret} from the mounted Secret.
Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal
entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A
key must be named `<clientId>-client-secret`: that naming contract is what
lets this reconciler derive the client set from git without a second
declaration, and the L0 static check enforces it against
keycloak-realm-import.yaml so the two can never drift apart.
"""
out = {}
for name in sorted(os.listdir(secrets_dir)):
if name.startswith(".") or not name.endswith(KEY_SUFFIX):
continue
path = os.path.join(secrets_dir, name)
if not os.path.isfile(path):
continue
with open(path, "r", encoding="utf-8") as fh:
value = fh.read()
# A trailing newline in the SOPS source would otherwise become part of the
# secret and mismatch forever. Keycloak secrets never contain whitespace.
out[name[: -len(KEY_SUFFIX)]] = value.strip()
return out
def admin_token(base, user, password, log):
"""Bearer token for admin-cli in the master realm, with a bounded readiness wait."""
last = ""
for attempt in range(1, READY_ATTEMPTS + 1):
try:
st, text = _http(
"POST", base, "/realms/master/protocol/openid-connect/token",
form={"client_id": "admin-cli", "grant_type": "password",
"username": user, "password": password})
except OSError as e:
st, text = 0, "connection error (%s)" % e.__class__.__name__
if st == 200:
token = json.loads(text).get("access_token")
if token:
return token
last = "http 200 but no access_token in the response"
else:
# Deliberately do NOT echo the response body: it is harmless today, but
# the request that produced it carried the admin password, and keeping
# that class of mistake impossible by habit is cheaper than reviewing it.
last = "http %s" % st
log("waiting for the Keycloak admin API (attempt %d/%d): %s"
% (attempt, READY_ATTEMPTS, last))
if attempt < READY_ATTEMPTS:
time.sleep(READY_SLEEP)
raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s"
% (READY_ATTEMPTS, last))
def reconcile(base, realm, token, want, audit, log):
"""Return (changed, problems). Writes only when audit is False."""
hdrs = {"Authorization": "Bearer " + token}
changed = 0
problems = []
for cid in sorted(want):
st, text = _http("GET", base, "/admin/realms/%s/clients" % realm,
query={"clientId": cid}, headers=hdrs)
if st != 200:
problems.append("%s: client lookup failed (http %s)" % (cid, st))
continue
found = json.loads(text)
if not isinstance(found, list) or not found:
problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it "
"but the realm has no such client (the operator runs the "
"realm import exactly once)" % (cid, realm))
continue
client = found[0]
uuid = client.get("id")
if client.get("publicClient"):
log("%-14s skipped (public client, holds no secret)" % cid)
continue
st, text = _http("GET", base,
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
headers=hdrs)
if st != 200:
problems.append("%s: could not read the stored secret (http %s)" % (cid, st))
continue
stored = json.loads(text).get("value") or ""
if hmac.compare_digest(stored, want[cid]):
log("%-14s ok (already correct, no write)" % cid)
continue
# Describe the drift WITHOUT quoting either value. The literal-placeholder
# case is named because it is the failure this program exists for, and
# saying so reveals nothing secret.
kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER"
if "env:" in stored else "stored value differs from the declared one")
if audit:
problems.append("%s: DRIFT -- %s" % (cid, kind))
continue
log("%-14s DRIFT (%s) -- repairing" % (cid, kind))
# MINIMAL representation only -- see the header. Keycloak merges non-null
# fields, so protocol mappers, redirect URIs, flows, groups, roles and users
# are all untouched by this write.
st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid),
headers=hdrs, body={"clientId": client.get("clientId"),
"secret": want[cid]})
if not 200 <= st < 300:
problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160]))
continue
st, text = _http("GET", base,
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
headers=hdrs)
readback = (json.loads(text).get("value") or "") if st == 200 else ""
if not hmac.compare_digest(readback, want[cid]):
problems.append("%s: PUT was accepted but the readback still does not match"
% cid)
continue
changed += 1
log("%-14s REPAIRED and verified by readback" % cid)
return changed, problems
def run(argv, log):
audit = "--audit" in argv
base = os.environ.get("KEYCLOAK_BASE_URL", "").strip()
realm = os.environ.get("KEYCLOAK_REALM", "").strip()
user = os.environ.get("KEYCLOAK_ADMIN_USER", "")
password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")
for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm),
("KEYCLOAK_ADMIN_USER", user),
("KEYCLOAK_ADMIN_PASSWORD", password)):
if not value:
log("FATAL: %s is empty" % name)
return 1
log("mode=%s realm=%s base=%s"
% ("audit (read-only)" if audit else "reconcile", realm, base))
try:
want = declared(SECRETS_DIR)
except OSError as e:
log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__))
return 1
if not want:
log("FATAL: no `<clientId>%s` keys found in %s -- the keycloak-realm-clients "
"Secret is missing or empty, and reporting success here would repeat the "
"very fail-open the realm import already has"
% (KEY_SUFFIX, SECRETS_DIR))
return 1
log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want))))
try:
token = admin_token(base, user, password, log)
changed, problems = reconcile(base, realm, token, want, audit, log)
except Refused as e:
log("FATAL: %s" % e)
return 1
except (RuntimeError, OSError, ValueError) as e:
log("FATAL: %s: %s" % (e.__class__.__name__, e))
return 1
if problems:
log("")
for p in problems:
log("PROBLEM %s" % p)
log("")
if audit:
log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems))
else:
log("RECONCILE FAIL: %d client secret(s) could not be reconciled"
% len(problems))
return 1
if audit:
log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match "
"git" % len(want))
else:
log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired"
% (len(want), changed))
return 0
# =============================================================================
# --selftest -- the offline behavioural proof. It drives the REAL code paths above
# against an in-process stub Keycloak, so what the gate proves is what ships.
# =============================================================================
def _refused(path):
"""True iff _http() refuses `path` outright (allow-list, safety property 4)."""
try:
_http("PUT", "http://127.0.0.1:1", path, body={})
except Refused:
return True
except OSError:
return False
return False
def _selftest():
import contextlib
import http.server
import io
import tempfile
import threading
global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP
SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
SECRET_C = "cccccccccccccccccccccccccccccccc"
STALE = "0ld-r0tated-value"
ADMIN_PW = "stub-admin-password"
# The exact value the broken clusters stored. chr(36) is the dollar sign,
# spelled this way so the file stays free of the character envsubst reacts to.
LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)"
state = {"clients": {}, "writes": [], "touched": []}
class Stub(http.server.BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def _send(self, code, obj):
payload = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def _route(self, method):
parsed = urllib.parse.urlparse(self.path)
path, qs = parsed.path, urllib.parse.parse_qs(parsed.query)
state["touched"].append("%s %s" % (method, path))
if path == "/realms/master/protocol/openid-connect/token":
return self._send(200, {"access_token": "stub-token"})
if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path):
cid = (qs.get("clientId") or [""])[0]
c = state["clients"].get(cid)
return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)])
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret",
path)
if m and method == "GET":
c = state["clients"].get(m.group(1))
if c is None:
return self._send(404, {"error": "not found"})
return self._send(200, {"type": "secret", "value": c.get("secret", "")})
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path)
if m and method == "PUT":
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
rep = json.loads(raw or b"{}")
# The live server rejects a full round-tripped representation with
# 400 "Cannot parse the JSON"; modelled so a regression to the fat
# PUT fails here rather than against a real realm.
if set(rep) != {"clientId", "secret"}:
return self._send(400, {
"error": "unknown_error",
"error_description": "Cannot parse the JSON"})
state["writes"].append(m.group(1))
state["clients"][m.group(1)]["secret"] = rep["secret"]
return self._send(204, {})
return self._send(500, {"error": "an endpoint the reconciler must not touch"})
def do_GET(self):
self._route("GET")
def do_PUT(self):
self._route("PUT")
def do_POST(self):
self._route("POST")
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub)
threading.Thread(target=srv.serve_forever, daemon=True).start()
base = "http://127.0.0.1:%d" % srv.server_address[1]
tmp = tempfile.mkdtemp()
for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B),
("kubernetes", SECRET_C), ("gitea", SECRET_A)):
with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh:
fh.write(secret + "\n") # trailing newline: must be tolerated
# Kubernetes' own projection entries must be ignored, not read as clients.
os.mkdir(os.path.join(tmp, "..data"))
SECRETS_DIR = tmp
READY_ATTEMPTS, READY_SLEEP = 2, 0.01
os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab",
"KEYCLOAK_ADMIN_USER": "admin",
"KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW})
failures = []
checks = [0]
def expect(cond, what):
checks[0] += 1
if not cond:
failures.append(what)
def invoke(*argv):
state["writes"] = []
state["touched"] = []
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = run(list(argv), print)
return rc, buf.getvalue()
def reset():
state["clients"] = {
# already correct
"grafana": {"clientId": "grafana", "publicClient": False,
"secret": SECRET_A},
# THE INCIDENT: the literal, unresolved import placeholder
"pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL},
# public client: no secret, must be skipped
"kubernetes": {"clientId": "kubernetes", "publicClient": True},
# ordinary drift (rotated in git, stale in the realm)
"gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE},
}
def leaked(text):
return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW)
if s in text]
# -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING
reset()
rc, out = invoke("--audit")
expect(rc == 1, "audit must exit 1 when a stored secret differs from git")
expect(state["writes"] == [],
"audit mode must never write (writes=%r)" % state["writes"])
expect("UNRESOLVED IMPORT PLACEHOLDER" in out,
"audit must name the unresolved-placeholder case explicitly")
expect("pihole: DRIFT" in out and "gitea: DRIFT" in out,
"audit must report BOTH drifted clients, not stop at the first")
expect(re.search(r"grafana\s+ok \(already correct", out),
"an already-correct client must be reported as such")
expect(re.search(r"kubernetes\s+skipped \(public client", out),
"a public client must be skipped, not treated as drift")
expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out))
# -- 2. reconcile: repairs exactly the drifted clients, verified by readback
reset()
rc, out = invoke()
expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:])
expect(sorted(state["writes"]) == ["gitea", "pihole"],
"reconcile must write exactly the drifted clients, got %r" % state["writes"])
expect("grafana" not in state["writes"],
"compare-then-write: an already-correct client must not be written")
expect("kubernetes" not in state["writes"], "a public client must not be written")
expect("REPAIRED and verified by readback" in out,
"every repair must be readback-verified")
expect("2 repaired" in out, "the summary must count the repairs")
expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out))
# -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing
rc, out = invoke()
expect(rc == 0, "the second reconcile must exit 0")
expect(state["writes"] == [],
"IDEMPOTENCE: a converged realm must take ZERO writes, got %r"
% state["writes"])
expect("0 repaired" in out, "the second run must report 0 repaired")
rc, out = invoke("--audit")
expect(rc == 0, "audit must pass once the realm has converged")
expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out,
"a clean audit must emit the sentinel the L2 conformance test greps for")
# -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched
reset()
invoke()
for touched in state["touched"]:
expect(re.fullmatch(
r"POST /realms/master/protocol/openid-connect/token"
r"|GET /admin/realms/homelab/clients"
r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?",
touched), "the reconciler touched an off-list endpoint: %s" % touched)
expect(not [t for t in state["touched"]
if "/users" in t or "/groups" in t or "/roles" in t
or "/authentication" in t],
"the reconciler must never touch users/groups/roles/authentication")
expect(_refused("/admin/realms/homelab/users"),
"the allow-list must refuse a users endpoint before opening a socket")
expect(_refused("/admin/realms/homelab/groups"),
"the allow-list must refuse a groups endpoint before opening a socket")
expect(_refused("/admin/realms/homelab"),
"the allow-list must refuse a whole-realm update endpoint")
# -- 5. a declared client the realm never got is LOUD, not a silent pass
reset()
del state["clients"]["gitea"]
rc, out = invoke("--audit")
expect(rc == 1, "a declared client missing from the realm must fail the audit")
expect("NOT FOUND" in out, "the missing client must be named in the output")
# -- 6. an empty / unmounted Secret must fail closed, never report success
SECRETS_DIR = tempfile.mkdtemp()
rc, out = invoke()
expect(rc == 1, "an empty secrets dir must fail closed")
expect("keys found in" in out, "the empty-secret failure must say what is missing")
SECRETS_DIR = tmp
srv.shutdown()
print("")
if failures:
print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0]))
for f in failures:
print(" FAIL %s" % f)
return 1
print("SELFTEST OK: %d assertions (compare-then-write, idempotence, "
"no-secret-logging, endpoint allow-list, fail-closed)" % checks[0])
return 0
def main():
argv = sys.argv[1:]
if "--selftest" in argv:
return _selftest()
return run(argv, lambda m: print(m, flush=True))
if __name__ == "__main__":
sys.exit(main())
---
# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key
# added after the realm import already completed) converges within one interval
# instead of waiting for a human to notice a failed login. Runs are cheap: a
# converged realm performs 12 GETs and ZERO writes.
apiVersion: batch/v1
kind: CronJob
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
spec:
schedule: "*/15 * * * *"
# Forbid: a run that is still waiting for Keycloak must not be lapped by the next
# tick. Two concurrent reconcilers would be harmless (both converge to the same
# value) but would double the write attempts during an outage recovery.
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
ttlSecondsAfterFinished: 3600
template:
metadata:
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
spec:
restartPolicy: Never
# The reconciler talks ONLY to the Keycloak admin API. It needs no
# Kubernetes API access at all — its two inputs are mounted Secrets — so
# it gets no ServiceAccount token (R-SEC-02, least privilege).
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: reconciler
# Stock upstream python; the reconciler is pure standard library, so
# nothing is installed at runtime. Pinned exactly, like busybox /
# curlimages elsewhere in the repo. Renovate does not manage plain
# `image:` tags (renovate.json disables image-tag updates so it never
# fights Flux image-automation), so this is bumped by hand.
image: python:3.13.15-alpine3.23
imagePullPolicy: IfNotPresent
command:
- python3
- /opt/reconciler/reconcile-client-secrets.py
env:
- name: KEYCLOAK_BASE_URL
value: http://keycloak-service.keycloak.svc.cluster.local:8080
- name: KEYCLOAK_REALM
value: homelab
# readOnlyRootFilesystem is on; keep CPython from trying to write
# bytecode caches beside the read-only ConfigMap mount.
- name: PYTHONDONTWRITEBYTECODE
value: "1"
- name: PYTHONUNBUFFERED
value: "1"
# Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to
# bootstrap the admin; keys admin-user / admin-password.
- name: KEYCLOAK_ADMIN_USER
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-user
- name: KEYCLOAK_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-password
volumeMounts:
- name: script
mountPath: /opt/reconciler
readOnly: true
# The DESIRED state: one file per declared client, named
# `<clientId>-client-secret`. Mounting the Secret (rather than
# reading it through the API) is what lets this pod run with no
# ServiceAccount token at all.
- name: client-secrets
mountPath: /etc/keycloak-realm-clients
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
volumes:
- name: script
configMap:
name: keycloak-client-secret-reconciler
defaultMode: 0444
- name: client-secrets
secret:
secretName: keycloak-realm-clients
defaultMode: 0400