jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
530 lines
26 KiB
Bash
Executable File
530 lines
26 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — keycloak-client-secret-reconciler
|
|
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
|
|
# — spec docs/15-identity-and-access.md)
|
|
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
|
|
# — spec docs/15-identity-and-access.md)
|
|
#
|
|
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
|
|
# ---------------------------------------------------------------
|
|
# platform/keycloak/keycloak-realm-import.yaml declares 12 confidential OIDC clients
|
|
# whose secrets are `(env:NAME)` import placeholders, mapped by spec.placeholders to
|
|
# keys of the SOPS-encrypted keycloak-realm-clients Secret. Measured on both clusters
|
|
# on 2026-09-06:
|
|
#
|
|
# * when the Keycloak Operator cannot resolve a placeholder it stores the LITERAL
|
|
# placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS;
|
|
# * the KeycloakRealmImport then rests at Done=True / Started=False and NEVER
|
|
# re-runs, so it never self-heals.
|
|
#
|
|
# Consequence: on prod all 12 clients held the literal placeholder and every OIDC
|
|
# login on the cluster failed ("Invalid client or Invalid client credentials"); on dev
|
|
# exactly one (oauth2-proxy) was broken, because that Secret key was added AFTER dev's
|
|
# import had already completed and a completed import is never re-run.
|
|
#
|
|
# Nothing in the repo asserted the declaration was self-consistent, and nothing
|
|
# corrected the realm afterwards. This check is the static half of the fix (the
|
|
# reconciler CronJob is the runtime half, and this check proves that too).
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# THEN (1) DECLARATION IS COMPLETE: every confidential client in the realm import
|
|
# declares a secret placeholder; every placeholder it uses is defined in
|
|
# spec.placeholders; every placeholder resolves to the keycloak-realm-clients
|
|
# Secret; and every referenced key EXISTS in the SOPS file. A client whose
|
|
# key is missing is exactly the dev failure, and it is now unmergeable.
|
|
# (2) THE NAMING CONTRACT HOLDS: each key is `<clientId>-client-secret`. That is
|
|
# what lets the reconciler derive the client set from the mounted Secret with
|
|
# no second declaration — so it must be enforced, not assumed. No orphan
|
|
# placeholder, and no orphan Secret key (an orphan key would make every
|
|
# reconciler run fail with NOT FOUND).
|
|
# (3) PUBLIC CLIENTS DECLARE NO SECRET (the `kubernetes` client is PKCE-public;
|
|
# giving it a secret would be a silent downgrade).
|
|
# (4) ORDERING IS A REAL FLUX EDGE, NOT A COMMENT: the Flux Kustomization that
|
|
# renders the Secret is a transitive `dependsOn` ancestor — with `wait: true`
|
|
# — of the one that renders the realm import and the reconciler. The repo's
|
|
# hard-won rule: a dependency that is only DOCUMENTED is not a dependency.
|
|
# (5) THE RECONCILER IS WIRED AND SAFE: it is reconciled from BOTH cluster
|
|
# entrypoints, mounts exactly the two Secrets it needs at the paths its code
|
|
# defaults to, carries no ServiceAccount token, and contains no dollar sign
|
|
# (it ships inside a substituted Flux path, where an unescaped dollar-brace
|
|
# expansion is rewritten before any container runs).
|
|
# (6) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
|
|
# script from the ConfigMap and executes its `--selftest`, which drives the
|
|
# real code paths against an in-process stub Keycloak and asserts
|
|
# compare-then-write, idempotence (zero writes on a converged realm),
|
|
# no-secret-logging, the endpoint allow-list (users/groups/roles are
|
|
# unreachable) and fail-closed behaviour. What CI proves is byte-for-byte
|
|
# what ships.
|
|
# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
|
|
# blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
|
|
# platform/keycloak/ to these requirement IDs, and an L2 test exists that
|
|
# audits the LIVE realm. An unreferenced test is precisely the "looks applied
|
|
# and isn't" failure this repo keeps hitting.
|
|
#
|
|
# NOT proven here (named honestly, in the style of the sibling L0 checks):
|
|
# - that the LIVE realm's stored secrets match git. That is cluster state; it is
|
|
# conformance/tests/l2-behavioral/keycloak-client-secrets-match-git.
|
|
# - that the SOPS ciphertext decrypts to a usable secret. Key PRESENCE is checkable
|
|
# offline; the plaintext is not, and must not be.
|
|
# - that the CronJob's image can be pulled or its schedule fires.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
|
|
SOPS_FILE="platform/keycloak/keycloak-realm-clients.sops.yaml"
|
|
RECON_FILE="platform/keycloak/client-secret-reconciler.yaml"
|
|
|
|
# Honest SKIP (never green) if the thing under test is absent entirely.
|
|
for f in "$IMPORT_FILE" "$SOPS_FILE" "$RECON_FILE"; do
|
|
if [ ! -f "${ROOT}/${f}" ]; then
|
|
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
|
|
exit 3
|
|
fi
|
|
done
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
|
"""Static audit of the Keycloak client-secret declaration, its Flux ordering, and the
|
|
reconciler that heals it. PyYAML (the same dependency tests/l0-static/no-plaintext-secrets
|
|
and least-privilege-rbac already take) parses the manifests; lib/status.py is IMPORTED for
|
|
the Flux graph walk rather than forked, so there is one definition of "what a Flux path
|
|
renders" in the harness."""
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
try:
|
|
import yaml
|
|
except ImportError: # pragma: no cover
|
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
|
file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
|
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
|
import status as st # noqa: E402 — reuse the walk, don't fork it
|
|
|
|
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
|
|
SOPS_FILE = "platform/keycloak/keycloak-realm-clients.sops.yaml"
|
|
RECON_FILE = "platform/keycloak/client-secret-reconciler.yaml"
|
|
SECRET_NAME = "keycloak-realm-clients"
|
|
ADMIN_SECRET = "keycloak-admin-credentials"
|
|
KEY_SUFFIX = "-client-secret"
|
|
SECRETS_MOUNT = "/etc/keycloak-realm-clients"
|
|
DOLLAR = chr(36)
|
|
|
|
FAILURES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
def load(rel):
|
|
return list(yaml.safe_load_all((ROOT / rel).read_text()))
|
|
|
|
|
|
# ================================================================= (1)-(3) declaration
|
|
realm_import = next((d for d in load(IMPORT_FILE)
|
|
if d and d.get("kind") == "KeycloakRealmImport"), None)
|
|
check(realm_import is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document found")
|
|
if realm_import is None:
|
|
print("CHECK FAIL: cannot parse the realm import", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
placeholders = realm_import["spec"].get("placeholders") or {}
|
|
clients = realm_import["spec"]["realm"].get("clients") or []
|
|
check(len(clients) > 0, f"{IMPORT_FILE}: parsed zero clients — the file shape changed and "
|
|
f"this check would silently audit nothing")
|
|
|
|
# The operator's placeholder syntax, spelled without a literal dollar so this file is
|
|
# itself safe to grep for one. Matches `<dollar>(env:NAME)`.
|
|
PLACEHOLDER_RE = re.compile(re.escape(DOLLAR) + r"\(env:([A-Za-z_][A-Za-z0-9_]*)\)")
|
|
|
|
used_placeholders: set[str] = set()
|
|
declared_pairs: dict[str, str] = {} # clientId -> secret key in the Secret
|
|
|
|
for c in clients:
|
|
cid = c.get("clientId", "<unnamed>")
|
|
secret = c.get("secret")
|
|
public = bool(c.get("publicClient"))
|
|
if public:
|
|
# (3) a PKCE public client must not carry a secret at all.
|
|
check(secret is None,
|
|
f"{IMPORT_FILE}: client '{cid}' is publicClient but declares a secret — a "
|
|
f"public client holds no secret; declaring one is a silent downgrade")
|
|
continue
|
|
# (1) every confidential client must declare one.
|
|
check(isinstance(secret, str) and secret,
|
|
f"{IMPORT_FILE}: confidential client '{cid}' declares NO secret. The operator "
|
|
f"would generate a random one that exists nowhere in git, so nothing could "
|
|
f"ever reconcile it and the consuming app's mirrored credential would be wrong")
|
|
if not isinstance(secret, str):
|
|
continue
|
|
m = PLACEHOLDER_RE.fullmatch(secret.strip())
|
|
check(m is not None,
|
|
f"{IMPORT_FILE}: client '{cid}' has a secret that is not an (env:NAME) "
|
|
f"placeholder. A literal secret in git violates R-IDP-04/R-SEC-01; the only "
|
|
f"permitted form is a placeholder resolved from the {SECRET_NAME} Secret")
|
|
if m is None:
|
|
continue
|
|
name = m.group(1)
|
|
used_placeholders.add(name)
|
|
# (1) the placeholder must be DEFINED. An undefined one is stored literally by the
|
|
# operator, with a successful-looking import — the whole incident.
|
|
ph = placeholders.get(name)
|
|
check(ph is not None,
|
|
f"{IMPORT_FILE}: client '{cid}' uses placeholder '{name}' but spec.placeholders "
|
|
f"does not define it. The operator stores the LITERAL placeholder as the client "
|
|
f"secret and STILL reports the import successful (measured 2026-09-06)")
|
|
if ph is None:
|
|
continue
|
|
ref = (ph or {}).get("secret") or {}
|
|
check(ref.get("name") == SECRET_NAME,
|
|
f"{IMPORT_FILE}: placeholder '{name}' reads Secret '{ref.get('name')}', not "
|
|
f"'{SECRET_NAME}'. The reconciler mounts only {SECRET_NAME}, so a second source "
|
|
f"would be declared but never healed")
|
|
key = ref.get("key")
|
|
check(isinstance(key, str) and key,
|
|
f"{IMPORT_FILE}: placeholder '{name}' names no Secret key")
|
|
if not isinstance(key, str):
|
|
continue
|
|
# (2) the naming contract the reconciler derives its client set from.
|
|
check(key == cid + KEY_SUFFIX,
|
|
f"{IMPORT_FILE}: client '{cid}' reads Secret key '{key}', but the reconciler "
|
|
f"derives the clientId from the key name, so it MUST be "
|
|
f"'{cid}{KEY_SUFFIX}'. Rename the key (and the SOPS entry) or the reconciler "
|
|
f"would look up a client that does not exist")
|
|
declared_pairs[cid] = key
|
|
|
|
# (2) no orphan placeholder: a defined-but-unused placeholder is dead config that will
|
|
# quietly rot out of sync with the Secret.
|
|
for name in sorted(set(placeholders) - used_placeholders):
|
|
check(False,
|
|
f"{IMPORT_FILE}: spec.placeholders defines '{name}' but no client uses it — "
|
|
f"dead declaration; remove it or wire it to the client that needs it")
|
|
|
|
# ---- the SOPS Secret: keys are cleartext in a SOPS-encrypted file; values are not.
|
|
sops_doc = next((d for d in load(SOPS_FILE) if d and d.get("kind") == "Secret"), None)
|
|
check(sops_doc is not None, f"{SOPS_FILE}: no Secret document found")
|
|
sops_keys = set((sops_doc or {}).get("stringData") or {})
|
|
check(len(sops_keys) > 0,
|
|
f"{SOPS_FILE}: parsed zero stringData keys — the file shape changed and this check "
|
|
f"would silently audit nothing")
|
|
check((sops_doc or {}).get("metadata", {}).get("name") == SECRET_NAME,
|
|
f"{SOPS_FILE}: the Secret is not named {SECRET_NAME}")
|
|
|
|
for cid, key in sorted(declared_pairs.items()):
|
|
# (1) THE DEV FAILURE, made unmergeable: a client declared in the import whose key
|
|
# is absent from the Secret imports "successfully" with a literal placeholder.
|
|
check(key in sops_keys,
|
|
f"{SOPS_FILE}: client '{cid}' needs key '{key}' and it is NOT in the Secret. "
|
|
f"The realm import would still report SUCCESS and store the literal placeholder "
|
|
f"as that client's secret — this is exactly how dev's oauth2-proxy broke")
|
|
|
|
# (2) no orphan Secret key: the reconciler treats every `<x>-client-secret` key as a
|
|
# client that MUST exist, and fails loudly if it does not. Catch it here instead.
|
|
for key in sorted(k for k in sops_keys if k.endswith(KEY_SUFFIX)):
|
|
cid = key[: -len(KEY_SUFFIX)]
|
|
check(cid in declared_pairs,
|
|
f"{SOPS_FILE}: key '{key}' has no matching client in {IMPORT_FILE}. The "
|
|
f"reconciler would look up client '{cid}', not find it, and fail every run")
|
|
|
|
# ===================================================== (4) ordering is a real Flux edge
|
|
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`; a doc
|
|
# is an entrypoint only if it carries the kustomize.toolkit apiVersion, `kind:
|
|
# Kustomization` and a spec-level `path:`. Overlay `patch: |-` bodies carry Kustomizations
|
|
# WITHOUT a path and are correctly skipped (they are patches, not entrypoints).
|
|
_DEPENDSON_RE = re.compile(r"^(\s*)dependsOn:\s*$", re.M)
|
|
_DEP_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
|
|
_WAIT_RE = re.compile(r"^\s{2,6}wait:\s*true\s*(?:#.*)?$", re.M)
|
|
|
|
|
|
def depends_on(doc: str) -> list[str]:
|
|
m = _DEPENDSON_RE.search(doc)
|
|
if not m:
|
|
return []
|
|
indent = len(m.group(1))
|
|
out: list[str] = []
|
|
for raw in doc[m.end():].splitlines():
|
|
if raw.strip() == "" or raw.lstrip().startswith("#"):
|
|
continue
|
|
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
|
|
break # dedented out of the list
|
|
nm = _DEP_NAME_RE.match(raw)
|
|
if nm:
|
|
out.append(nm.group(1))
|
|
return out
|
|
|
|
|
|
kustomizations: dict[str, dict] = {}
|
|
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
|
|
text = yfile.read_text()
|
|
if st.FLUX_API not in text:
|
|
continue
|
|
for doc in st._DOC_SPLIT_RE.split(text):
|
|
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
|
|
continue
|
|
pm = st._PATH_RE.search(doc)
|
|
if not pm:
|
|
continue
|
|
nm = st._NAME_RE.search(doc)
|
|
name = nm.group(1) if nm else "?"
|
|
kustomizations[name] = {
|
|
"file": str(yfile.relative_to(ROOT)),
|
|
"path": pm.group(1),
|
|
"dependsOn": depends_on(doc),
|
|
"wait": bool(_WAIT_RE.search(doc)),
|
|
}
|
|
|
|
check(len(kustomizations) > 0,
|
|
"no Flux Kustomization with a spec.path found under clusters/ — the parser or the "
|
|
"repo layout changed; this check would silently audit nothing")
|
|
|
|
|
|
def direct_render(path_spec: str) -> set[str]:
|
|
"""Files a Flux `spec.path` renders ITSELF — a kustomize walk that deliberately does
|
|
NOT expand nested Flux Kustomizations.
|
|
|
|
lib/status.py's reconciled_files() follows every `spec.path` it discovers, which is
|
|
right for "is this file applied to the cluster at all" but wrong for OWNERSHIP: the
|
|
root Kustomization's path is clusters/<env>, so a transitive walk makes the root the
|
|
owner of every file in the repo and the ordering assertion below becomes vacuous
|
|
(measured: it reported the root as an owner of all three keycloak files). `resources:`
|
|
parsing itself is st.kustomize_resources, not forked.
|
|
"""
|
|
rel = path_spec.lstrip("./") or "."
|
|
out: set[str] = set()
|
|
seen: set[str] = set()
|
|
|
|
def walk(d: str) -> None:
|
|
if d in seen:
|
|
return
|
|
seen.add(d)
|
|
kfile = ROOT / d / "kustomization.yaml"
|
|
if not kfile.is_file():
|
|
return
|
|
out.add(str(kfile.relative_to(ROOT)))
|
|
for item in st.kustomize_resources(kfile):
|
|
if "://" in item:
|
|
continue # remote base: not repo state
|
|
target = (ROOT / d / item).resolve()
|
|
try:
|
|
trel = target.relative_to(ROOT).as_posix()
|
|
except ValueError:
|
|
continue # outside the repo
|
|
if target.is_dir():
|
|
walk(trel)
|
|
elif target.is_file():
|
|
out.add(trel)
|
|
|
|
if (ROOT / rel).is_dir():
|
|
walk(rel)
|
|
return out
|
|
|
|
|
|
rendered_by = {n: direct_render(k["path"]) for n, k in kustomizations.items()}
|
|
|
|
|
|
def owners(rel_file: str) -> list[str]:
|
|
return sorted(n for n, files in rendered_by.items() if rel_file in files)
|
|
|
|
|
|
def ancestors(name: str, seen=None) -> set[str]:
|
|
"""Transitive dependsOn closure of `name` (its dependencies, direct and indirect)."""
|
|
seen = seen if seen is not None else set()
|
|
for dep in kustomizations.get(name, {}).get("dependsOn", []):
|
|
if dep in seen:
|
|
continue
|
|
seen.add(dep)
|
|
ancestors(dep, seen)
|
|
return seen
|
|
|
|
|
|
secret_owners = owners(SOPS_FILE)
|
|
import_owners = owners(IMPORT_FILE)
|
|
recon_owners = owners(RECON_FILE)
|
|
|
|
check(bool(secret_owners),
|
|
f"{SOPS_FILE} is rendered by NO Flux Kustomization — the client secrets are in git "
|
|
f"but never applied")
|
|
check(bool(import_owners),
|
|
f"{IMPORT_FILE} is rendered by NO Flux Kustomization")
|
|
# (5) the reconciler must actually be reconciled, or it is decoration.
|
|
check(bool(recon_owners),
|
|
f"{RECON_FILE} is rendered by NO Flux Kustomization — the healer would sit in git "
|
|
f"and never run, which is the 'looks applied and isn't' failure it exists to end")
|
|
|
|
for consumer_file, consumer_owners in ((IMPORT_FILE, import_owners),
|
|
(RECON_FILE, recon_owners)):
|
|
for co in consumer_owners:
|
|
for so in secret_owners:
|
|
if so == co:
|
|
continue # same Kustomization: one atomic apply, ordering moot
|
|
anc = ancestors(co)
|
|
check(so in anc,
|
|
f"ORDERING IS ONLY DOCUMENTED: '{co}' renders {consumer_file} but does "
|
|
f"not (transitively) dependsOn '{so}', which renders the "
|
|
f"{SECRET_NAME} Secret. Without that edge the consumer can reconcile "
|
|
f"before the secret exists, and the Keycloak Operator's response to a "
|
|
f"missing placeholder is to store it LITERALLY and report success")
|
|
if so in anc:
|
|
check(kustomizations[so]["wait"],
|
|
f"'{so}' (which renders the {SECRET_NAME} Secret) does not set "
|
|
f"wait: true, so '{co}' can proceed before it is applied")
|
|
|
|
# ------ both environments must reconcile the reconciler, not just dev.
|
|
for env in ("dev", "prod"):
|
|
entry = f"clusters/{env}"
|
|
if not (ROOT / entry).is_dir():
|
|
continue
|
|
applied, _ = st.reconciled_files(ROOT, entry)
|
|
check(RECON_FILE in applied,
|
|
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the healer would "
|
|
f"exist on one cluster only, and prod is where all 12 clients were broken")
|
|
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
|
|
|
|
# ============================================ (5) the reconciler manifest is safe + wired
|
|
recon_docs = [d for d in load(RECON_FILE) if d]
|
|
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
|
|
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
|
|
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
|
|
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
|
|
|
|
script = ""
|
|
if cm is not None:
|
|
data = cm.get("data") or {}
|
|
check(len(data) == 1,
|
|
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
|
|
f"(found {sorted(data)}) — a second copy would be untested by this gate")
|
|
script = next(iter(data.values()), "")
|
|
check(len(script.splitlines()) > 50,
|
|
f"{RECON_FILE}: the extracted script is implausibly short "
|
|
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
|
|
f"selftest below would prove nothing")
|
|
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
|
|
# postBuild.substituteFrom, so envsubst rewrites it before any container runs. The
|
|
# script is written with NO dollar sign at all, so the hazard is structural rather
|
|
# than escaped and no future edit has to reason about escaping.
|
|
check(DOLLAR not in script,
|
|
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside a "
|
|
f"SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
|
|
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
|
|
f"never needs one; chr(36) spells it where a literal is genuinely required")
|
|
|
|
if cj is not None:
|
|
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
|
|
ctr = pod["containers"][0]
|
|
vols = {v["name"]: v for v in pod.get("volumes", [])}
|
|
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
|
|
secret_vols = [v for v in vols.values()
|
|
if (v.get("secret") or {}).get("secretName") == SECRET_NAME]
|
|
check(len(secret_vols) == 1,
|
|
f"{RECON_FILE}: the CronJob must mount the {SECRET_NAME} Secret — that mount IS "
|
|
f"the desired state it reconciles towards")
|
|
if secret_vols:
|
|
mp = mounts.get(secret_vols[0]["name"], {}).get("mountPath")
|
|
check(mp == SECRETS_MOUNT,
|
|
f"{RECON_FILE}: {SECRET_NAME} is mounted at {mp}, but the script defaults "
|
|
f"to {SECRETS_MOUNT} and no env var overrides it")
|
|
cm_vols = [v for v in vols.values()
|
|
if (v.get("configMap") or {}).get("name") == (cm or {}).get(
|
|
"metadata", {}).get("name")]
|
|
check(len(cm_vols) == 1,
|
|
f"{RECON_FILE}: the CronJob does not mount the reconciler ConfigMap, so it "
|
|
f"would run nothing")
|
|
env_srcs = {e["name"]: e for e in ctr.get("env", [])}
|
|
for var in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
|
|
ref = (env_srcs.get(var, {}).get("valueFrom") or {}).get("secretKeyRef") or {}
|
|
check(ref.get("name") == ADMIN_SECRET,
|
|
f"{RECON_FILE}: {var} must come from the {ADMIN_SECRET} Secret "
|
|
f"(got {ref.get('name')!r}) — the admin credential is never inlined")
|
|
for var in ("KEYCLOAK_BASE_URL", "KEYCLOAK_REALM"):
|
|
check(env_srcs.get(var, {}).get("value"),
|
|
f"{RECON_FILE}: {var} is not set; the script fails closed without it, but "
|
|
f"the CronJob should not ship broken")
|
|
# R-SEC-02: it talks only to Keycloak's REST API and needs no Kubernetes API rights.
|
|
check(pod.get("automountServiceAccountToken") is False,
|
|
f"{RECON_FILE}: the reconciler needs NO Kubernetes API access (its inputs are "
|
|
f"mounted Secrets) — set automountServiceAccountToken: false")
|
|
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
|
|
f"{RECON_FILE}: set concurrencyPolicy: Forbid so a run stalled on an "
|
|
f"unavailable Keycloak is not lapped by the next tick")
|
|
img = ctr.get("image", "")
|
|
check(re.search(r":[0-9]", img),
|
|
f"{RECON_FILE}: container image {img!r} is not pinned to an explicit version")
|
|
|
|
# ============================== (6) the reconciler's BEHAVIOUR, proven by its own selftest
|
|
if script:
|
|
with tempfile.TemporaryDirectory() as td:
|
|
p = Path(td) / "reconcile.py"
|
|
p.write_text(script)
|
|
proc = subprocess.run([sys.executable, str(p), "--selftest"],
|
|
capture_output=True, text=True, timeout=180)
|
|
tail = (proc.stdout + proc.stderr).strip().splitlines()
|
|
check(proc.returncode == 0,
|
|
"the reconciler's --selftest FAILED (compare-then-write / idempotence / "
|
|
"no-secret-logging / endpoint allow-list / fail-closed):\n "
|
|
+ "\n ".join(tail[-12:]))
|
|
if proc.returncode == 0:
|
|
print(" reconciler --selftest: " + (tail[-1] if tail else "(no output)"))
|
|
|
|
# ======================================================= (7) this check is load-bearing
|
|
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
|
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
|
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
|
|
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
|
|
check("keycloak-client-secret-reconciler/check.sh" in L0,
|
|
"ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
|
|
"requirement-scoped conformance selection can filter this check out of a gate run, "
|
|
"so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
|
|
"it. An unreferenced test is the failure this whole change exists to end")
|
|
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
|
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
|
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
|
|
"ci/touched-requirements.sh has no platform/keycloak/* entry in its path->requirement "
|
|
"map, so editing the realm import or this reconciler attributes NO requirement and "
|
|
"the L2 keycloak tests are filtered out of the merge gate's touched subset")
|
|
l2 = ROOT / "conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml"
|
|
check(l2.is_file(),
|
|
"the LIVE-realm counterpart (conformance/tests/l2-behavioral/"
|
|
"keycloak-client-secrets-match-git) is missing — this static check cannot see a "
|
|
"realm that drifted after apply, which is the whole failure mode")
|
|
if l2.is_file():
|
|
check("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in l2.read_text(),
|
|
"the L2 test does not assert the reconciler's audit sentinel, so it could pass "
|
|
"on a Job that never ran the audit")
|
|
|
|
# ------------------------------------------------------------------------------ report
|
|
print(f"realm import: {len(clients)} client(s), {len(declared_pairs)} confidential with a "
|
|
f"declared secret, {len(placeholders)} placeholder(s)")
|
|
print(f"{SECRET_NAME} Secret: {len(sops_keys)} key(s) in git (names only; values stay "
|
|
f"SOPS-encrypted)")
|
|
print("flux ordering: " + ", ".join(
|
|
f"{f.split('/')[-1]} <- {owners(f)}" for f in (SOPS_FILE, IMPORT_FILE, RECON_FILE)))
|
|
|
|
print()
|
|
if not FAILURES:
|
|
print(f"check OK: keycloak-client-secret-reconciler — {CHECKS} assertions "
|
|
f"(R-IDP-03, R-IDP-04)")
|
|
raise SystemExit(0)
|
|
print(f"CHECK FAIL (keycloak-client-secret-reconciler): {len(FAILURES)} failed, "
|
|
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" FAIL {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
PY
|