Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Browse Source 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

ci/l0.sh
+38
View File
@@ -137,6 +137,44 @@ else
esac
fi
# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ----
# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
# Invalid client credentials") behind an entirely green cluster; dev had one. The check
# asserts the declaration is self-consistent (every placeholder defined, every Secret key
# present, names matching the reconciler's contract), that the Flux ordering is a REAL
# dependsOn edge rather than a comment, and — by extracting the shipped script from its
# ConfigMap and running its --selftest — that the reconciler is compare-then-write,
# idempotent, non-logging and blast-radius-bounded.
#
# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance
# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/,
# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04
# and the requirement-scoped selection would filter it out of exactly the change that
# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no
# side effects, so running it twice when 3b also selects it is not worth avoiding.
KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}"
log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})"
if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then
skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green"
summary "- keycloak-client-secrets: **SKIPPED** (check missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
kc_rc=0
bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$?
case "$kc_rc" in
0) ok "Keycloak client-secret declaration + reconciler passed"
summary "- keycloak-client-secrets: **passed**" ;;
3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass"
summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)"
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
*) err "Keycloak client-secret declaration + reconciler FAILED"
summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;;
esac
fi
case "$rc" in
0) ok "L0 stage complete" ;;
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
ci/touched-requirements.sh
+32
View File
@@ -180,9 +180,41 @@ path_requirements() {
platform/externaldns/*) echo "R-DNS-02" ;;
platform/velero/*) echo "R-STATE-03" ;;
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
# when a diff does carry the tags.
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
# the client-secret incident, in which editing platform/keycloak/ attributed NO
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
# subset — the change that could break every OIDC login was gated by nothing.
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
# its off-cluster backup credential, not only the realm.
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
# enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
# already-mapped path scoped the L0 run to that other path's requirements and silently
# excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
# precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
# stopped matching when the value it guards changed, in a different file) is exactly the
# class that omission would hide.
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
*) : ;;
esac
conformance/incluster/rbac.yaml
+7
View File
@@ -57,6 +57,13 @@ rules:
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
verbs: [create, delete, get, list, watch, patch]
- apiGroups: [""]
resources: [pods/exec]
conformance/lib/status.py
+213 -19
View File
@@ -297,6 +297,109 @@ _EXECUTABLE_RE = re.compile(
_BLOCKED_RE = re.compile(
r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
# ---------------------------------------------------------- scaffold-marker detection
# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
# ---------------------------------------------------
# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
# is placeholder detection has to name the idioms it hunts for:
# `l0-static/placeholder-guard-coupling/check.sh` carries
# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
# A whole-file substring test read that as the test declaring itself unfinished, classified
# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
#
# The rule below is derived from the corpus (43 tests), not invented. Every genuine
# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
# two structural positions, and no executable test does either:
#
# 1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
# repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
# `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
# `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
# 2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
# bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
# message is a note about work that is missing. A marker that is the ENTIRE content of
# a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
# marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
#
# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
# carrying any trailing text all classify as scaffold.
#
# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
# several idioms — is still called a scaffold. That is the conservative direction, and the
# author's remedy is the same structural discipline the corpus already uses: keep each
# idiom a standalone literal in a list.
# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
# `placeholder.*`), and none of them is used here to mark a placeholder body.
# Case-insensitive: no English word contains either token, and the corpus carries no
# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
_STATUS_SCAFFOLD_RE = re.compile(
r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
# A quoted literal whose entire content is one marker word: the marker as data.
_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
def _split_doc_header(text: str) -> tuple[str, str]:
"""Split a test file into (documentation header, body).
Both file kinds in this tree open with a banner comment block that is PROSE about the
test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
placeholder idioms when the test hunts them, so an incidental mention there is
documentation, not a placeholder body.
The body starts at the first line that is neither blank nor a `#` comment:
*.sh -> `set -euo pipefail` (the shebang is a comment)
*.yaml -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
# TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
"""
lines = text.splitlines(keepends=True)
for i, ln in enumerate(lines):
s = ln.strip()
if s and not s.startswith("#"):
return "".join(lines[:i]), "".join(lines[i:])
return text, "" # no executable line at all: it is all header, body is empty
def scaffold_marker(text: str) -> str:
"""Why `text` declares itself a placeholder body, or "" if it does not.
Returns a short human-readable reason (used verbatim in the hygiene report) so the
defect is actionable rather than a bare path.
"""
head, body = _split_doc_header(text)
m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
if m:
return m.group(0).strip().lstrip("#").strip()[:80]
# Blank out markers held as pure data before looking for a marker that has a message.
m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
if m2:
line = body[:m2.start()].count("\n")
snippet = body.splitlines()[line].strip()
return f"placeholder marker in body: {snippet[:80]}"
return ""
def classify_test(t: dict) -> dict:
"""Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
@@ -304,14 +407,18 @@ def classify_test(t: dict) -> dict:
A tagged test is only worth counting if it can actually assert something. Rules, in
order, chosen so ambiguity NEVER upgrades a test:
scaffold the body contains TODO markers (a placeholder), *even if* it
is annotated executable — an annotation must never outrank the
body. That mismatch is also reported as a hygiene defect.
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
executable-untagged no TODOs and a real runnable body, but the annotation is
missing (true of the L1 render tests, which execute a
render-check.sh + golden diff, of the L0 static checks, and
of the META self-check).
scaffold the test DECLARES itself a placeholder — a `# STATUS: scaffold`
header line, or a TODO/FIXME marker carrying a message in the
body (see `scaffold_marker` for the rule and the corpus it was
derived from) — *even if* it is annotated executable. An
annotation must never outrank the body. That mismatch is also
reported as a hygiene defect.
executable annotated `conformance.home.k3s/executable: "true"` and no
scaffold declaration.
executable-untagged no scaffold declaration and a real runnable body, but the
annotation is missing (true of the L1 render tests, which
execute a render-check.sh + golden diff, of the L0 static
checks, and of the META self-check).
Counted as executable, and flagged as an annotation gap.
scaffold anything else.
"""
@@ -322,16 +429,26 @@ def classify_test(t: dict) -> dict:
# downgrades the test to a scaffold — an annotation must never outrank the body,
# whichever file the body lives in.
extra = ""
scripts: list[tuple[str, str]] = []
for script in ("render-check.sh", "check.sh"):
sp = p.parent / script
if sp.is_file():
extra += sp.read_text()
body = sp.read_text()
extra += body
scripts.append((script, body))
declared = bool(_EXECUTABLE_RE.search(text))
has_todo = "TODO" in text or "TODO" in extra
# Each file is classified on ITS OWN structure — the yaml and the script have different
# header blocks, so they must not be concatenated before the header/body split.
reason = scaffold_marker(text)
for script, body in scripts:
if reason:
break
r = scaffold_marker(body)
reason = f"{script}: {r}" if r else ""
bm = _BLOCKED_RE.search(text)
has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
if has_todo:
if reason:
state = "scaffold"
elif declared:
state = "executable"
@@ -348,7 +465,8 @@ def classify_test(t: dict) -> dict:
"name": t["name"], "path": rel, "layer": t["layer"],
"state": state,
"declared_executable": declared,
"annotation_conflict": bool(declared and has_todo),
"annotation_conflict": bool(declared and reason),
"scaffold_reason": reason,
"blocked_by": bm.group(1).strip().strip('"') if bm else "",
}
@@ -640,6 +758,10 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
# --- hygiene defects this tool owns -------------------------------------------
hygiene = {
"annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
# Same set, with WHY each one was called a placeholder body, so the fix is obvious
# from the report instead of requiring the reader to re-derive the rule.
"annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
for t in classified if t["annotation_conflict"]},
"executable_untagged": [t["path"] for t in classified
if t["state"] == "executable-untagged"],
"registry_not_in_spec": [],
@@ -910,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
else:
o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
"this is a skip, not a pass)")
o.append(f"- tests annotated executable but containing TODO bodies: "
f"{h['annotation_conflicts'] or 'none'}")
o.append("- tests annotated executable but declaring a placeholder body: "
+ (", ".join(f"`{k}` ({v})"
for k, v in sorted(h["annotation_conflict_reasons"].items()))
if h["annotation_conflicts"] else "none"))
o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
f"{h['executable_untagged'] or 'none'}")
@@ -930,9 +1054,12 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
"reconciled |")
o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
"(the authoritative tag, same source as `coverage.md`) | exact |")
o.append("| executable vs scaffold | TODO markers in the test body outrank the "
"`executable` annotation; a real body without the annotation still counts "
"and is flagged | conservative — ambiguity never upgrades |")
o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
"TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
"quoted datum — a placeholder detector's own vocabulary — is not one); "
"either outranks the `executable` annotation, and a real body without the "
"annotation still counts and is flagged | conservative — ambiguity never "
"upgrades |")
o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
"every verifying test: JUnit result **plus** that test's own pass sentinel "
"token grepped from the run log | exact where a run was recorded, refused "
@@ -1127,6 +1254,70 @@ def _selftest() -> int:
classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
== "scaffold")
# === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
# A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
# coupling` — a 32-assertion executable check whose SUBJECT is placeholder
# detection, so it must name the idioms it hunts — as a scaffold, and through it
# reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
# assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
# real placeholder body must still be caught. Both halves are load-bearing; the
# second is what stops this fix from degrading into "always executable".
yml = ("metadata:\n annotations:\n"
" conformance.home.k3s/executable: \"true\"\n"
"spec:\n steps:\n - script:\n"
" content: bash ./check.sh\n")
f.write_text(yml)
detector = td2 / "check.sh"
# Shaped exactly like the real check: a prose header that discusses the idioms,
# then executable code holding them as a vocabulary list.
detector.write_text(
"#!/usr/bin/env bash\n"
"# L0 static check — placeholder vocabulary\n"
"# * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
"# STATUS: EXECUTABLE. A pure offline assertion script.\n"
"set -euo pipefail\n"
"python3 - <<'EOF'\n"
"SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
"assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
"EOF\n")
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
check("a marker in HEADER PROSE does not downgrade an executable check",
c["state"] == "executable" and not c["annotation_conflict"])
check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
c["scaffold_reason"] == "")
# ... and the protection is intact: a real placeholder body is still caught, in
# each of the three shapes the corpus actually uses.
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"echo \"TODO: assert the PVC is bound\"\n")
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
check("a TODO carrying a message in check.sh IS still a scaffold",
c["state"] == "scaffold" and c["annotation_conflict"])
check("the scaffold reason names the file and the offending line",
c["scaffold_reason"].startswith("check.sh:")
and "TODO: assert the PVC is bound" in c["scaffold_reason"])
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
== "scaffold")
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
"echo \"FIXME the extraction before trusting this\"\n")
check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
== "scaffold")
# The dedicated declaration slot outranks a body that would otherwise look real —
# this is how all three genuine scaffolds in the tree declare themselves.
detector.unlink()
f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
"# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
c["state"] == "scaffold" and c["annotation_conflict"])
check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
"assertion script; covers no scaffold.\nset -e\n") == "")
# ---------------------------------------------------------------- PROOF / ledger
# These are the assertions that stand between this report and a fake green.
check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
@@ -1341,8 +1532,11 @@ def main() -> int:
f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
rc = 1
if h["annotation_conflicts"]:
sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
f"bodies: {h['annotation_conflicts']}\n")
sys.stderr.write(
"[status] STRICT: tests annotated executable that declare a placeholder "
"body: " + "; ".join(f"{k} ({v})" for k, v in
sorted(h["annotation_conflict_reasons"].items()))
+ "\n")
rc = 1
if rep["safety_critical_without_executable_l3"]:
sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "
conformance/tests/README.md
+47 -4
View File
@@ -93,9 +93,52 @@ and coverage extraction exact and robust.
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
the PM — do not relax the assertion or the requirement.
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
binding over the same ConfigMap Flux substitutes from:
```yaml
spec:
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
```
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
the Service was assigned) — it is both stricter and environment-independent. Never take the
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
ends up silently asserting the other cluster's expectations. L1 goldens are the one
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
### How a scaffold declares itself (and how `status.py` tells one from a verifier)
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
can never be reported as an executable test.
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
`render-check.sh`, `check.sh` — and either signal below **outranks**
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
an executable test. Declare a scaffold in **both** positions:
1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
declared state must be the first token after `STATUS:` (executable tests say
`# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
or an inline comment on a missing fixture:
`file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`.
What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.
conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml
+106
View File
@@ -0,0 +1,106 @@
# =============================================================================
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
# one-time checks", with the acceptance gate "The five auditability proofs above can
# be produced on demand."
#
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
# -----------------------------------------------------------------------------------
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
# failure modes are asserted instead, each by DRIVING the real machinery:
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
# the spec's own words require (fault injection / a DR drill => L3), taken from
# lib/status.py's own verdict so this check cannot disagree with the status report;
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
# attribute — a proof that passes but records nothing leaves the operator with a green
# terminal and no evidence;
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
# ci/ or run.sh ran before this test existed.
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
# so a sixth proof added to the spec fails here instead of being silently unverified.
#
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
# anyone needs it.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
# and two synthetic forgeries are presented to them,
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
# documented one-command invocation selects and asserts them, each is capturable as a
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
# it directly, and this Chainsaw Test wraps the SAME script.
#
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
# command, capturable, and that the chain refuses forgeries — not that isolation holds
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
# would need signed history.
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
# not re-asserted here.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: auditability-proofs-producible
labels:
conformance.home.k3s/R-SEC-03: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree and the harness's own tooling, when status.py, the requirement
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
of docs/07's five auditability proofs has an executable test at the layer the spec
requires, is selectable and asserted by the documented one-command invocation, is
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
evidence.
spec:
steps:
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
# ---------------------------------------------------------------------------------------------
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
# the wrong trade (R-TEST-06).
conformance/tests/l0-static/auditability-proofs-producible/check.sh
Executable
+431
View File
@@ -0,0 +1,431 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — auditability-proofs-producible
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
# prove isolation, egress path, cert validity, and backup integrity)."
# — spec docs/00-vision-and-scope.md).
#
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
# denied-probe test.
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
# status + external check.
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
# "These proofs are part of the platform acceptance suite, not one-time checks."
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
#
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
# ------------------------------------------------------------
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
# by DRIVING the real machinery rather than by reading files:
#
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
# repository's own status report.
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
# their coverage (that flag is the merge gate's own path: it passes --require to
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
# gate wiring break, R-SEC-03 fails.
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
# instead of silently making the requirement unprovable.
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
# real tools, in a throwaway directory:
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
# test. record_run.py must refuse to call it `proven`.
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
# implementing code — status.py must degrade it to `stale-proof`, never green.
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
# were themselves unexercised in the merge gate until this check.
#
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
# than quietly ignoring it.
#
# WHY L0: every clause above is a property of the repository and its own tooling, provable
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
# with one.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
# documented command, capturable as evidence, and that the evidence chain refuses
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
# proof, which is what is asserted.
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
# denied-probe half, which is the half that is automated. The Hubble half remains an
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
# report's prose, the ledger's provenance fields) is a usability property, not a testable
# one.
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
# nothing here can prove a row was never removed from it (that would need signed history).
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(sys.argv[1])
CONF = ROOT / "conformance"
LIB = CONF / "lib"
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
def die():
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def run(cmd, **kw) -> subprocess.CompletedProcess:
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
# reqs = the requirement(s) whose verifying test IS that proof
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
PROOFS = {
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
"denied-probe test"},
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
"— via cert status + external check"},
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
"why": '"irreplaceable data is restorable" — via a DR drill'},
}
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
spec = Path(os.environ.get("SPEC_REPO", ""))
sec_doc = spec / "docs" / "07-security-and-secrets.md"
if sec_doc.is_file():
text = sec_doc.read_text()
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
re.M | re.S)
check(m is not None,
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
"the proof list this check is coupled to has moved or been removed, so the mapping "
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
if m:
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
check(set(labels) == set(PROOFS),
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
"test, and a proof this check does not know about would be silently unverified")
check(len(labels) == 5,
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
"gate says 'The five auditability proofs above can be produced on demand' — the "
"prose and the list have drifted apart")
else:
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
"re-derived from docs/07, so the mapping below was used as written. Every other "
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
# Driven through lib/status.py so this check and the repository's own status report cannot
# disagree about what is executable and what is a scaffold.
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
try:
out = tmp / "status"
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out)])
check(st.returncode == 0,
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
status = {}
sj = out / "status.json"
if sj.is_file():
status = json.loads(sj.read_text())
check(bool(status.get("by_requirement")),
"lib/status.py produced no per-requirement report — the audit posture of each "
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
by_req = status.get("by_requirement", {})
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid)
check(rec is not None,
f"proof '{label}' maps to {rid}, which the status report does not know — "
"either the registry lost the requirement or this mapping is stale")
if not rec:
continue
tests = (rec.get("verification") or {}).get("tests") or []
executable = [t for t in tests if t.get("state", "").startswith("executable")]
check(bool(executable),
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
"operator cannot prove this property")
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
check(bool(at_layer),
f"proof '{label}' ({rid}) has no executable test at layer(s) "
f"{sorted(spec_proof['layers'])} — it has "
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
"weaker layer does not discharge it (a rendering check is not a drill)")
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
# The resolver: --requirements must translate the five R-IDs into real test dirs.
resolver = run(["bash", "-c",
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
"acceptance gate depends on ('produced on demand') cannot select the proofs")
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
# zero exit here is the repository's own statement that all five proofs are covered.
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
check(gate.returncode == 0,
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
# token can pass forever and never be recorded as proof of anything.
sys.path.insert(0, str(LIB))
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
sys.dont_write_bytecode = True
import record_run as rr # noqa: E402
smap = rr.sentinel_map(CONF / "tests")
check(len(smap) >= 10,
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
"grip on the test tree, so the assertions below would assert nothing")
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid) or {}
tests = (rec.get("verification") or {}).get("tests") or []
names = [t.get("name") for t in tests
if t.get("state", "").startswith("executable")
and t.get("layer") in spec_proof["layers"]]
for name in names:
ent = smap.get(name)
check(ent is not None,
f"proof '{label}': record_run.py does not know the test {name!r} — its "
"result could never be attributed to a ledger row")
if not ent:
continue
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
continue # offline layers are truthful by JUnit alone
check(bool(ent.get("pass")),
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
"so lib/record_run.py can never record it as proven. It would go green "
"in the terminal and leave the operator with no evidence — exactly the "
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
"platform acceptance suite, not one-time checks')")
check(bool(ent.get("terminal_pass")),
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
"record_run.py cannot decide its verdict")
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
for tool in ("record_run.py", "status.py"):
r = run([sys.executable, str(LIB / tool), "--selftest"])
check(r.returncode == 0,
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
"self-skipped or stale run from being recorded as proof are broken, so nothing "
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
forge_target, forge_skip = None, None
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
for name, ent in smap.items():
if rid in (ent.get("requirements") or []) and ent.get("skip") \
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
forge_target, forge_skip = name, ent["skip"][0]
break
if forge_target:
break
if forge_target:
break
check(forge_target is not None,
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
"could not be mounted. That is not a pass: it means this check could not verify "
"that a self-skip is refused as proof, which is the whole point of the ledger")
if forge_target:
run_dir = tmp / "20260101T000000Z-forged"
run_dir.mkdir(parents=True)
(run_dir / "report.junit.xml").write_text(
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
f'<testcase name="{forge_target}" classname="forged"/>'
"</testsuite></testsuites>\n")
(run_dir / "run.log").write_text(
f"=== RUN chainsaw/{forge_target}\n"
f" === STDOUT\n {forge_skip}\n")
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
forged.stdout, re.M)
check(verdicts and all(v != "proven" for v in verdicts),
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
f"({forge_skip}). A self-skip must never become proof — with this hole every "
"audit proof in docs/07 could be produced by a cluster that skipped it")
check(not (tmp / "never-written.jsonl").exists(),
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
"claiming not to cannot be trusted to record only real runs")
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
# status.py must degrade it to stale-proof (not green), or an old green run would keep
# vouching for code it never executed.
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
check(len(old_sha) == 40,
"the repository's root commit could not be resolved, so the stale-proof guard could "
"not be exercised (that is a gap in this check, not a pass)")
if len(old_sha) == 40 and forge_target:
stale_ledger = tmp / "stale.jsonl"
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
stale_ledger.write_text(json.dumps({
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
"requirements": [rid], "junit_result": "passed",
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
"proof": "proven", "run_id": "20260101T000000Z-forged",
"note": "synthetic row written by the R-SEC-03 conformance check",
}) + "\n")
out2 = tmp / "status-stale"
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out2),
"--ledger", str(stale_ledger)])
check(st2.returncode == 0,
f"lib/status.py failed against a synthetic ledger: "
f"{(st2.stderr or st2.stdout).strip()[:300]}")
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
check(per.get(forge_target) == "stale-proof",
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
"proof from a run that predates the code it vouches for is not evidence, and "
"honouring it would let one old green run certify every later change")
check((rec2.get("proof") or {}).get("state") != "proven",
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
"freshness guard is not holding")
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
ledger = CONF / "runs" / "ledger.jsonl"
if ledger.is_file():
rows, bad = [], []
for i, line in enumerate(ledger.read_text().splitlines(), 1):
if not line.strip():
continue
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
bad.append(i)
check(not bad,
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
"is not machine-readable, so no proof can be derived from it")
for r in rows:
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
"run_id"):
check(key in r,
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
"provenance cannot be audited")
if r.get("proof") == "proven":
check(bool(r.get("sentinel")),
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
"the sentinel is the only thing separating a real pass from a self-skip")
check(r.get("test") in smap,
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
"test in the suite any more — a proof of something that no longer exists")
check("-dirty" not in str(r.get("git_sha", "")),
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
f"({r.get('git_sha')}) — what ran is not what is committed")
else:
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
"were skipped; every other assertion ran, and the forgery probes above use "
"synthetic ledgers, not this file")
finally:
shutil.rmtree(tmp, ignore_errors=True)
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 30,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
for n in NOTES:
print(f" note: {n}")
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
"(see the header's gap list).")
PY
conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml
+75
View File
@@ -0,0 +1,75 @@
# =============================================================================
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
# created, baselined, cluster-joined, and managed as code"); the four stages
# are enumerated in docs/02 §"Node lifecycle stages".
#
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
# to any node), not by reading files: a chain that does not resolve fails the check.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
# The dev group names are read from the playbooks' own default host patterns, the node
# names are generated from the IaC module's prefix/count variables, and the two sets are
# compared against each other — so the assertion survives a rename or a resize.
#
# GIVEN the repository as git tracks it,
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
# provider — a cloud provider fails), from a for_each map rather than a count index,
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
# resolves and targets every dev node at every stage.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
# where present — ansible-playbook in no-connection modes; no cluster, no network);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
# wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
# hypervisor access, neither available offline.
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
# parsed, not evaluated; a network-allowed CI stage would close this.
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
# tests/l3-fault-injection/node-loss (R-STATE-02).
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
# divergence rather than hiding it.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: dev-nodes-provisioned-as-code
labels:
conformance.home.k3s/R-ENV-01: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-01"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the dev provisioning chain is parsed and driven
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
and handed to Flux by one code path that covers all of them — with no node outside
it and no stage missing.
spec:
steps:
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh
Executable
+442
View File
@@ -0,0 +1,442 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
# baselined, cluster-joined, and managed as code." — spec
# docs/00-vision-and-scope.md; the four stages are enumerated in
# docs/02-environments-and-provisioning.md §"Node lifecycle stages
# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
# 1. Create — OpenTofu instantiates the VM
# 2. Baseline — OS baseline and prerequisites
# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
# k3s bundled flannel/traefik/servicelb/kube-proxy
# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC
# and re-applied, never hand-edited on the box.)
#
# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
# ----------------------------------------------------------------------
# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
# cannot answer it: a Kubernetes node object looks identical whether it was built by
# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
# provisioning path exists for all four stages and covers every dev node — and, for the
# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
# does not actually resolve fails here. The IaC half is asserted by parsing the module:
# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
#
# GIVEN the repository as git tracks it,
# WHEN the provisioning chain is parsed and (where offline execution is possible)
# actually driven,
# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider
# is a KVM-family hypervisor provider — no cloud provider may appear),
# instantiates its nodes from a for_each map (never a count index, so one
# node can be replaced without disturbing its siblings), and gives each
# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
# cloud-init seed carrying the login user + SSH key;
# AND the node set it creates is EXACTLY the dev node set Ansible manages —
# names derived from the module's own prefix/count, so no node can be
# hand-added to the inventory or hand-created in the hypervisor;
# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
# modules persisted AND loaded, sysctls, the Longhorn data disk formatted
# and mounted, swap off, iscsid enabled), driven by variables with
# defaults rather than per-host special cases;
# JOIN: one role installs k3s for both roles of every node — `--cluster-init`
# on the primary, `--server` join on the secondaries — with flannel,
# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
# installed as the CNI afterwards;
# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
# `--list-hosts` shows every one of those stages targeting EVERY dev
# node, so no node is outside the code path.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
# The dev group names come from the playbooks' own default host patterns, the node names
# from the IaC module's prefix/count variables, and the two sets are compared to each
# other. This check therefore keeps working when the estate is renamed or resized.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
# access. `tofu plan` against the live estate is the operator's own check.
# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
# the provider from the registry; L0 is offline by contract, so the module is parsed,
# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
# with a network-allowed stage, not here.
# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
# - That the baseline actually ran on a node (cluster/host state), and that the
# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
# network topology fact.
# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
# cloud-init applying the OS baseline. The implementation moved the rich baseline into
# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
# because Proxmox provisioning is API-only and a custom cloud-init snippet needs
# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
# "baselined ... as code", not that cloud-init be the tool, so this check asserts the
# STAGE, wherever it lives, and records the divergence rather than hiding it.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import re
import shutil
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
ENV_ONLY_FAILURE = re.compile(
r"vault password|vault-id|Attempting to decrypt"
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
TOFU = ROOT / "infra" / "tofu"
ANS = ROOT / "infra" / "ansible"
check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
if not TOFU.is_dir() or not ANS.is_dir():
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
# ============================================================ STAGE 1 — CREATE (IaC)
# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
# here rather than being noticed later. Each entry is justified, not just listed:
# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate).
KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
[(m.group(1), m.group(2)) for m in
re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
check(len(providers) > 0,
"infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
"cannot be checked at all")
for alias, source in sorted(providers.items()):
check(source.lower() in KVM_PROVIDERS,
f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
TF, re.S)
check(len(vm_blocks) >= 1,
"no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
for rtype, rname, body in vm_blocks:
check("for_each" in body,
f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
"replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
"'created ... as code' would not survive a rebuild")
check(not re.search(r"^\s*count\s*=", body, re.M),
f'resource "{rtype}" "{rname}" uses a positional count index')
for want, why in (
(r"\bcpu\b", "CPU"),
(r"\bmemory\b", "RAM"),
(r"network_device|network_interface", "a NIC"),
(r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
):
check(re.search(want, body) is not None,
f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
"disks, NICs and a cloud-init seed to come from IaC")
disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
check(disks >= 2,
f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
"disk PLUS a dedicated data disk for Longhorn")
check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
"reached by the BASELINE stage without a hand step")
# Node identity comes from variables, and those variables define the whole node set.
def tf_default(var: str):
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
if not m:
return None
d = re.search(r'default\s*=\s*(.+)', m.group(1))
return d.group(1).strip() if d else None
prefix_raw = tf_default("node_name_prefix")
count_raw = tf_default("node_count")
ips_raw = tf_default("node_ips")
check(prefix_raw is not None and count_raw is not None,
"infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
"from code, so the created nodes cannot be compared with the managed ones")
prefix = (prefix_raw or "").strip('"')
try:
node_count = int((count_raw or "0").strip())
except ValueError:
node_count = 0
check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
if ips_raw is not None:
n_ips = len(re.findall(r'"[^"]+"', ips_raw))
check(n_ips >= node_count,
f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
"cannot address every node it creates")
check((TOFU / "outputs.tf").is_file() and
re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
"infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
"told the node facts by hand instead of consuming them from IaC")
# ------------------------------------------------- the dev node set, derived twice over
# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
# restates the estate's naming.
PLAYBOOKS = ANS / "playbooks"
def default_patterns(playbook: str) -> list[str]:
p = PLAYBOOKS / playbook
if not p.is_file():
return []
return re.findall(r"default\('([^']+)'\)", p.read_text())
dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
check(len(dev_groups) > 0,
"no playbook declares a default host pattern — the dev target is not expressed in code")
INV = ANS / "inventory" / "hosts.ini"
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
ini_groups: dict[str, list[str]] = {}
children: dict[str, list[str]] = {}
cur = None
for line in (INV.read_text().splitlines() if INV.is_file() else []):
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
(children if cur.endswith(":children") else ini_groups).setdefault(
cur.split(":")[0] if cur.endswith(":children") else cur, [])
continue
if cur is None or cur.endswith(":vars"):
continue
host = s.split()[0]
if cur.endswith(":children"):
children.setdefault(cur.split(":")[0], []).append(host)
else:
ini_groups.setdefault(cur, []).append(host)
def hosts_in(group: str, seen=None) -> set[str]:
seen = seen or set()
if group in seen:
return set()
seen.add(group)
out = set(ini_groups.get(group, []))
for child in children.get(group, []):
out |= hosts_in(child, seen)
return out
dev_hosts: set[str] = set()
for g in sorted(dev_groups):
dev_hosts |= hosts_in(g)
check(len(dev_hosts) > 0,
f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
expected = {f"{prefix}{i + 1}" for i in range(node_count)}
check(dev_hosts == expected,
f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
# ============================================================ STAGE 2 — BASELINE (code)
BASE_ROLE = ANS / "roles" / "node_baseline"
check(BASE_ROLE.is_dir(),
"no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
base_tasks = "\n".join(p.read_text(errors="replace")
for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
base_defaults = "\n".join(p.read_text(errors="replace")
for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
# a declarative, idempotent task, not a shell one-liner.
for pattern, what in (
(r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
(r"modprobe:", "kernel modules loaded (modprobe module)"),
(r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
(r"sysctl", "sysctl tuning"),
(r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
(r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
(r"swap", "swap disabled"),
(r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
):
check(re.search(pattern, base_tasks) is not None,
f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
"k3s/Longhorn/KubeVirt prerequisites to be applied as code")
check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
"node_baseline's package/module sets are not variables with defaults — a per-host special "
"case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
K3S_ROLE = ANS / "roles" / "k3s_server"
k3s_text = "\n".join(p.read_text(errors="replace")
for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
"no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
check("--cluster-init" in k3s_text,
"the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
check(re.search(r"--server\s+https", k3s_text) is not None,
"the k3s role never joins secondaries with --server — nodes would not form one cluster")
check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
"the k3s role never supplies a join token — the join could only be completed by hand")
# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
# flag that actually disables it.
for pattern, what in (
(r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
(r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
(r"traefik", "traefik (Gateway API via Cilium)"),
(r"servicelb", "servicelb"),
):
check(re.search(pattern, k3s_text) is not None,
f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
"install, or the bundled component fights the platform's own")
check((ANS / "roles" / "cilium").is_dir(),
"no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
check((ANS / "roles" / "flux_bootstrap").is_dir(),
"no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
# ============================================== STAGE 4 — MANAGE + the chain is REAL
SITE = PLAYBOOKS / "site.yml"
check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
site_txt = SITE.read_text() if SITE.is_file() else ""
order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
check(order[:1] == ["node-baseline.yml"],
f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
if "k3s.yml" in order and "cilium.yml" in order:
check(order.index("k3s.yml") < order.index("cilium.yml"),
"site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
# --syntax-check and --list-hosts open NO connection to any node: they resolve the
# playbook, roles, collections and inventory only. Running them is the difference between
# "the chain is written down" and "the chain resolves".
if shutil.which("ansible-playbook") is None:
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
"chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
else:
# cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
# is the one in force — the same resolution a real run gets.
def ansible(*args) -> subprocess.CompletedProcess:
return subprocess.run(["ansible-playbook", *args], cwd=ANS,
capture_output=True, text=True)
syn = ansible("--syntax-check", "playbooks/site.yml")
syn_out = (syn.stderr or "") + (syn.stdout or "")
lh = None
if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
# NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
"passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
"were not driven; every parse assertion above still ran")
else:
check(syn.returncode == 0,
"`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
f"does not even load: {syn_out.strip()[:400]}")
lh = ansible("--list-hosts", "playbooks/site.yml")
check(lh.returncode == 0,
"`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
f"resolve against the tracked inventory: "
f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
if lh is not None and lh.returncode == 0:
# Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
plays: list[tuple[str, set[str]]] = []
name, hosts, in_hosts = None, set(), False
for line in lh.stdout.splitlines():
m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
if m:
if name is not None:
plays.append((name, hosts))
name, hosts, in_hosts = m.group(1), set(), False
continue
if re.match(r"\s*hosts \(\d+\):", line):
in_hosts = True
continue
if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
hosts.add(line.strip())
if name is not None:
plays.append((name, hosts))
check(len(plays) >= 4,
f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
# Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
# nodes means the third was finished by hand.
STAGES = {
"baseline": r"baseline",
"cluster-join": r"^k3s (primary|secondar)",
"cni": r"cilium",
"manage (flux hand-off)": r"flux",
}
for stage, rx in STAGES.items():
covered: set[str] = set()
matched = 0
for pname, phosts in plays:
if re.search(rx, pname, re.I):
matched += 1
covered |= phosts
check(matched >= 1, f"no play in site.yml implements the {stage} stage")
check(covered == dev_hosts,
f"the {stage} stage targets {sorted(covered)}, not every dev node "
f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
"node (R-ENV-01)")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 30,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
for n in NOTES:
print(f" note: {n}")
print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
"clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
"proven here (see the header's gap list; rebuild is L3 node-loss).")
PY
conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml
+94
View File
@@ -0,0 +1,94 @@
# =============================================================================
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
# routing rules ... realized as two paths — the local `internet` path and the
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
# routing: two paths, one physical WAN": "the `internet` path is simply the default
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
# the nodes themselves — egress the local/private WAN by default and **never** the
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
# kill-switch blackholes **only** that class."
#
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
# ------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
# that select among them, which nothing else asserts:
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
# absence of a matching rule IS the default path;
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
# vpn-class pod's cluster/LAN traffic still takes the local path;
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
#
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
# route in the main table). A cluster can show which path two pods took; it cannot show that
# no third rule exists or that the vocabulary is closed.
#
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
# against its own values.
#
# GIVEN the repository as git tracks it,
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
# parsed and cross-checked,
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
# selects exactly the vpn class by label; the gateway is selected by node label with a
# named egress interface; all RFC1918 space and every address any environment assigns
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
# cannot take the main table; and no nftables DROP is unscoped.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
# that generate them; a hand-added rule on the box is invisible here.
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
# * IPv6 egress.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: egress-paths-selected-by-rules
labels:
conformance.home.k3s/R-NET-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
the vpn class, local destinations stay on the local path in every environment, the
tunnel never becomes the default route, and nothing unscoped can blackhole the default
path.
spec:
steps:
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh
Executable
+513
View File
@@ -0,0 +1,513 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
# uplinks) is **out of scope** unless a future ADR reopens it."
# — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against — docs/03
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
# router -> internet |
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
# egress gateway (ADR-0013) |
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
# to policy-route across; the `internet` path is simply the default route out."
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
# themselves — egress the local/private WAN by default and **never** the VPN. Only
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
# blackholes **only** that class."
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
# only what Cilium points at `tun0` uses the VPN."
#
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
# ---------------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
# and the class can be flipped via GitOps. Those are proven by
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
#
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
# distinctive content — none of which any existing test asserts — is:
#
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
# one closed set of two values agreed by the schema, the admission policy and the
# renderer. A third value appearing in one of them but not the others would be a path
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
# class would contradict OQ-1's resolution.
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
# path is entered only via rules that select the vpn class — a Cilium
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
# to the tunnel source address. Nothing selects the `internet` class, because
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
# pod's cluster/LAN traffic still takes the local path at the same time as its
# internet traffic takes the tunnel.
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
# lives in a DEDICATED routing table reached only through the scoped rule, the
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
# from touching the main table. Otherwise there would be one path (the VPN), not two.
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
# asserts the OTHER direction, which that test does not.)
#
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
# table") are statements about ALL rules, including the ones that would have to be absent.
# A running cluster can show you which path two pods took; it cannot show you that no third
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
# tests/l2-behavioral/egress-default-path (same requirement label).
#
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
# LAN is checked against its own addresses, and the check keeps working when they change.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
# invisible here (and to every other test); drift on the box is an Ansible check-mode
# exercise for the operator.
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
# because nobody tried" — and inventing one would be a hollow assertion.
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
# this check would NOT catch its absence.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
LBL = "homelab.keithjackson.dev"
CLASS_LABEL = f"{LBL}/egress-class"
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def die():
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def docs(path: Path):
"""Every YAML document in a file, skipping empties."""
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and "_artifacts" not in p.parts)
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
RENDER = ROOT / "catalog" / "render.py"
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
CLUSTERS = ROOT / "clusters"
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
"read at all (R-NET-04)")
if FAILURES:
die()
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
import json # noqa: E402 — only needed for the schema
schema = json.loads(SCHEMA.read_text())
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
enum = egress_prop.get("enum")
check(isinstance(enum, list) and len(enum) == 2,
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
"nothing routes; one value is not 'multiple egress paths'")
PATHS = set(enum or [])
check(PATHS == {"internet", "vpn"},
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
check(egress_prop.get("default") == "internet",
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
"the default route out'. A service that says nothing must get the WAN")
# The admission policy must constrain the pod label to the SAME closed set, or a workload
# could carry a class the schema never allowed (and that no rule routes).
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
check(len(pol_files) >= 1,
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
"carrying an egress class outside the declared path set")
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
for cls in sorted(PATHS):
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
f"the egress admission policy never names the {cls!r} path — a class the policy "
"does not know cannot be validated against the path set")
# No tracked object may carry a class outside the closed set — with ONE self-policing
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
# admission test can assert the policy REJECTS it; that pod is exempt only while a
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
tests_root = fixture.parent
while tests_root != ROOT and tests_root.name != "tests":
tests_root = tests_root.parent
if tests_root == ROOT:
return False
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
for d in docs(tf):
for case in (d.get("results") or []):
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
return True
return False
stray: list[str] = []
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
for f in TRACKED_YAML:
for d in docs(f):
labels = ((d.get("metadata") or {}).get("labels") or {})
cls = labels.get(CLASS_LABEL)
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
if value is None or value in PATHS or value == "CLASS":
continue
name = ((owner.get("metadata") or {}).get("name") or "?")
if rejected_by_a_policy_test(name, f):
continue
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
# Values written as raw text (templates, docs-as-code) are caught too.
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
if value in PATHS or value == "CLASS":
continue
if any(value in s for s in stray):
continue
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
# carries this value is a policy-test fixture proven to be rejected.
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
continue
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
check(not stray,
f"tracked YAML assigns egress-class value(s) outside the declared path set "
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
# between the declaration and the path.
rtext = RENDER.read_text()
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
"two independent facts that could disagree")
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
policies: list[tuple[Path, dict]] = []
for p in TRACKED_YAML:
for d in docs(p):
if d.get("kind") == "CiliumEgressGatewayPolicy":
policies.append((p, d))
check(len(policies) == 1,
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
"something the repository states")
for path, pol in policies:
rel = str(path.relative_to(ROOT))
spec = pol.get("spec") or {}
selectors = spec.get("selectors") or []
check(len(selectors) == 1,
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
"selector widens the set of pods steered off the default path, and R-NET-04's "
"selection must be exactly 'the vpn class'")
for sel in selectors:
pod_sel = sel.get("podSelector")
check(isinstance(pod_sel, dict) and pod_sel,
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
"default path at all")
ml = (pod_sel or {}).get("matchLabels") or {}
me = (pod_sel or {}).get("matchExpressions") or []
check(not me,
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
"single label equality; expressions (In/NotIn/Exists) can select pods that "
"never declared a class, so the rule set would no longer be disjoint from the "
"default path")
check(set(ml) == {CLASS_LABEL},
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
"than the declared class (and a namespace/app key would silently exempt pods)")
check(ml.get(CLASS_LABEL) == "vpn",
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
"route the default path through the tunnel, i.e. leave one path, not two")
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
gw = spec.get("egressGateway") or {}
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
check(bool(node_sel),
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
"nodes + Egress Gateway HA)")
check("nodeName" not in gw,
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
"label so the path survives losing that node")
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
f"{rel}: the egressGateway names no egress interface — without it the steered class "
"is not actually sent down a different path from the default")
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
dests = spec.get("destinationCIDRs") or []
excl = spec.get("excludedCIDRs") or []
check([str(c) for c in dests] == ["0.0.0.0/0"],
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
"minus the exclusions below — narrowing it here would silently leave public "
"destinations on the WAN for a vpn-class service")
try:
excluded = [ipaddress.ip_network(str(c)) for c in excl]
except ValueError as e:
excluded = []
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
check(len(excluded) >= 3,
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
net = ipaddress.ip_network(rfc)
check(covered(net),
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
"coexist for the same pod and east-west traffic would break")
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
# is what makes 'local stays local' true per environment rather than by coincidence.
def overlay_addresses() -> list[tuple[str, str]]:
found: list[tuple[str, str]] = []
base = CLUSTERS / "base" / "cluster-vars.yaml"
data = {}
if base.is_file():
for d in docs(base):
if d.get("kind") == "ConfigMap":
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
env = {**data}
for m in re.finditer(
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
label = f.parent.name
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
v = env.get(key, "")
if v and not v.startswith("TBD"):
found.append((f"{label}/{key}", v))
return found
addrs = overlay_addresses()
check(len(addrs) >= 5,
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
"the 'local traffic stays local' assertion would be checking almost nothing")
for label, value in addrs:
try:
net = ipaddress.ip_network(value, strict=False)
except ValueError:
continue
check(covered(net),
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
"would try to reach its own cluster/LAN through the VPN")
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
TPL = VPN_ROLE / "templates"
up = (TPL / "vpn-egress-up.sh.j2")
down = (TPL / "vpn-egress-down.sh.j2")
conf = (TPL / "vpn-egress.conf.j2")
nft = (TPL / "vpn-egress-killswitch.nft.j2")
for f in (up, down, conf, nft):
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
"rules are not authored")
if FAILURES:
die()
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
def render(text: str) -> str:
"""Substitute `{{ var }}` with the role's own default value, for every default.
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
exactly what an earlier draft of this check did. Rendering first means the assertions
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
substituted; anything else is left as-is (and the assertions are written so an
unsubstituted expression fails rather than passes).
"""
out = text
for k, v in defaults.items():
if isinstance(v, (str, int)):
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
return out
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
"putting the tunnel's default route in any of them makes the VPN the node's default "
"route instead of a selectable second path")
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
"needs a rule, at a defined precedence")
# Every `ip route ... default` the up-script installs must name the dedicated table.
route_lines = [l.strip() for l in up_t.splitlines()
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
check(len(route_lines) >= 2,
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
"default route via the tunnel AND the fail-closed blackhole behind it")
for l in route_lines:
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
"landing in the main table changes the path every other pod and the node itself "
"uses — the VPN would stop being opt-in")
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
"be blackholed")
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
# everything into the tunnel table, which would leave one path instead of two.
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
check(len(rule_lines) >= 1,
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
for l in rule_lines:
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
"VPN is no longer opt-in")
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
f"an ip rule looks up a table other than the tunnel's: {l!r}")
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
"distinguishes the steered class from every other flow on the gateway node")
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
"the up-script does not refuse to run when the tunnel address is unknown — installing "
"a partial rule set is how a path ends up half-selected")
# The provider must not be able to make the tunnel the default route either.
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
"routes itself, in the MAIN table, and the VPN would become the node's default route "
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
"routing)')")
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
"provider could redirect the node's whole default route into the tunnel, collapsing "
"the two paths into one")
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
"default route")
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
check(bool(src_set),
"the role declares no nftables source set — the kill-switch would have no way to name "
"'the steered class' and could only match broadly")
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
check(len(policies_lines) >= 1,
"the kill-switch nft template declares no base chain — nothing would be enforced")
for l in policies_lines:
check(re.search(r"policy\s+accept\s*;", l) is not None,
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
drop_lines = [l.strip() for l in nft_t.splitlines()
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
check(len(drop_lines) >= 1,
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
for l in drop_lines:
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
check(scoped,
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
"the default `internet` path, which must never be blackholed")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 35,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
PY
conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml
+88
View File
@@ -0,0 +1,88 @@
# =============================================================================
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
#
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
# cluster cannot tell you what the NEXT environment brought up from this repo would look
# like. The observed half is asserted separately at L2 by
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
#
# GIVEN the repository as git tracks it,
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
# environment's Flux overlay are parsed and cross-checked against each other,
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
# ONE --cluster-init founder and every other server joining it with --server;
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
# in the apiserver serving cert's SANs together with every server's own address, and
# it is neither a node address nor the ingress Gateway VIP nor inside that
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
# roles and plays retargeted by variables only.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
# infra/PROD-PROVISIONING.md).
#
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
# contract and per environment.
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
# API server.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: ha-control-plane-as-code
labels:
conformance.home.k3s/R-PLAT-01: "true"
annotations:
conformance.home.k3s/requirements: "R-PLAT-01"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
environment's Flux overlay are parsed and cross-checked, then every environment is
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
VIP that is in the serving cert's SANs and collides with no other allocator — built by
one code path for both environments.
spec:
steps:
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/ha-control-plane-as-code/check.sh
Executable
+729
View File
@@ -0,0 +1,729 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — ha-control-plane-as-code
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
# same multi-node topology." — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against:
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
# docs/02-environments-and-provisioning.md, table:
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
# docs/02 §"k3s topology decisions":
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
# nodes we do not isolate control-plane nodes."
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs"
#
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
# ---------------------------------------------------------------------
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
#
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
# plane in BOTH environments, from ONE code path": provable with no cluster, and
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
# brought up from this repo will also be HA, whether a second `--cluster-init`
# founder could exist, or whether an external datastore is configured anywhere. That
# is this check.
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
# etcd members, with the API answering through the floating VIP": that is cluster
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
#
# Neither half is sufficient alone, and this file claims only the first.
#
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
# its group name. Add a fourth prod node, rename a group, or add a third environment and
# the assertions follow; nothing here restates an address, a hostname or a node count.
#
# GIVEN the repository as git tracks it,
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
# Flux overlay are parsed and cross-checked against each other,
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
# installed as an `agent` (an agent is not a control-plane member);
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
# every other server joins it with `--server` (so a second founder — the
# split-brain shape — cannot be authored);
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
# `--disable-etcd` anywhere in the provisioning code, so every server is
# an etcd member (docs/02's "no external datastore", "on all 3");
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
# baked into the apiserver serving cert's SAN list ALONG WITH every
# server's own address (so TLS validates whichever node answers);
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
# environment's Cilium LoadBalancer pool — the two allocators can never be
# handed the same address;
# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
# the cold `--server` join, k3s_apiserver_endpoint derives from
# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
# K3S_API_HOST equals that environment's VIP and is none of its node
# addresses ([parity], both halves), the HelmRelease keeps a per-environment
# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
# Cilium with roles/kube_vip blocking until it answers;
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
# runs"), so "exercise the same multi-node topology" is a property of the
# code rather than a coincidence of two hand-written inventories.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
# answers through it. All of that is asserted at L2 by
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
# data-availability claim and does NOT assert control-plane survival. A dedicated
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
# is NOT claimed here or at L2.
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
# - IPv6 / dual-stack, and any claim about the edge network.
#
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
# overlay's own comment. Making the check fail over it would have turned the merge gate
# red on a design question this workstream may not decide (R-TEST-06), so it was reported
# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
# ordering is no longer a preference about tidiness, it is a dependency of the control
# plane coming up at all.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
def die():
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
K3S_ROLE = ANS / "roles" / "k3s_server"
VIP_ROLE = ANS / "roles" / "kube_vip"
PLAYBOOKS = ANS / "playbooks"
CLUSTERS = ROOT / "clusters"
for p, why in ((INV, "the inventory that defines the node set"),
(K3S_ROLE, "the role that installs the k3s control plane"),
(VIP_ROLE, "the role that floats the control-plane VIP"),
(CLUSTERS, "the Flux overlays that identify each environment")):
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
"control plane is authored at all (R-PLAT-01)")
if FAILURES:
die()
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
VIP_TEXT = "\n".join(p.read_text(errors="replace")
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
# ===================================================================== inventory parse
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
ini_hosts: dict[str, list[str]] = {}
ini_children: dict[str, list[str]] = {}
host_vars: dict[str, dict[str, str]] = {}
cur = None
for line in INV.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
continue
if cur is None or cur.endswith(":vars"):
continue
host = s.split()[0]
if cur.endswith(":children"):
ini_children.setdefault(cur.split(":")[0], []).append(host)
else:
ini_hosts.setdefault(cur, []).append(host)
hv = host_vars.setdefault(host, {})
for kv in s.split()[1:]:
if "=" in kv:
k, v = kv.split("=", 1)
hv[k] = v
def hosts_in(group: str, seen=None) -> set[str]:
seen = seen or set()
if group in seen:
return set()
seen.add(group)
out = set(ini_hosts.get(group, []))
for child in ini_children.get(group, []):
out |= hosts_in(child, seen)
return out
def load_yaml(p: Path) -> dict:
try:
d = yaml.safe_load(p.read_text()) or {}
except yaml.YAMLError:
return {}
return d if isinstance(d, dict) else {}
ALL_VARS = load_yaml(GV / "all.yml")
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
# another environment. Nothing below knows the words "dev" or "prod" as group names.
class Env:
def __init__(self, label, vars_, path):
self.label = label
self.vars = vars_
self.path = path
def var(self, name, default=None):
# group-scoped value wins, else the cluster-wide default from all.yml.
if name in self.vars:
return self.vars[name]
return ALL_VARS.get(name, default)
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
for vf in sorted(GV.glob("*/vars.yml")):
v = load_yaml(vf)
if "k3s_primary_group" in v or "k3s_servers_group" in v:
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
check(len(envs) >= 2,
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
"dev/test to exercise the same multi-node topology, so two must be authored")
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
# roles/flux_bootstrap's own default supplies the value for an environment that does not
# override it, so this is read from code, never guessed from a group name.
fb_default = None
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
if fb_defaults_file.is_file():
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
"an inventory environment to its Flux overlay is not expressed in code, so this check "
"cannot tell which environment is prod")
prod_envs = []
for e in envs:
overlay = e.var("flux_bootstrap_env_path", fb_default)
e.overlay = overlay if isinstance(overlay, str) else ""
if e.overlay.rstrip("/").endswith("/prod"):
prod_envs.append(e)
check(len(prod_envs) == 1,
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
"R-PLAT-01 has no unambiguous referent")
# ============================================== overlay knob values (per environment)
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
# - op: replace
# path: /data/<KEY>
# value: "<v>"
# (verified shape across both overlays). A key not patched keeps the base value.
BASE_VARS: dict[str, str] = {}
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
if base_cm.is_file():
d = load_yaml(base_cm)
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
check(len(BASE_VARS) > 5,
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
"cannot be read, so the VIP/pool separation cannot be checked")
_PATCH_RE = re.compile(
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
re.M)
def overlay_vars(overlay: str) -> dict[str, str]:
out = dict(BASE_VARS)
f = ROOT / overlay / "kustomization.yaml"
if not f.is_file():
return out
for m in _PATCH_RE.finditer(f.read_text()):
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
return out
# ============================================================ PER-ENVIRONMENT TOPOLOGY
def as_ip(s):
try:
return ipaddress.ip_address(str(s).strip())
except ValueError:
return None
for e in envs:
L = e.label
pg = e.var("k3s_primary_group")
sg = e.var("k3s_servers_group")
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
"resolve 'my environment's primary/servers' and would only ever build dev")
if not (isinstance(pg, str) and isinstance(sg, str)):
continue
primaries = hosts_in(pg)
servers = hosts_in(sg)
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
check(len(primaries) == 1,
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
"cluster at all")
check(primaries and primaries <= servers,
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
"it is not part of the HA control plane it founds")
n = len(servers)
check(n >= 3,
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
check(n % 2 == 1,
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
f"member count adds a member without adding fault tolerance and introduces a tie: "
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
if n:
tolerance = n - (n // 2 + 1)
check(tolerance >= 1,
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
if e in prod_envs:
check(n == 3,
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
else:
check(n >= 2,
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
"nodes than prod but requires dev/test to 'exercise the same multi-node "
"topology' — one node exercises no topology")
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
vip_raw = e.var("kube_vip_address")
vip = as_ip(vip_raw)
check(vip is not None,
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
"no floating endpoint, so losing the founder takes the API endpoint with it")
node_ips = {}
for h in sorted(servers):
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
check(ip is not None,
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
"target and the serving-cert SAN list all derive from it")
if ip is not None:
node_ips[h] = ip
if vip is not None:
check(vip not in set(node_ips.values()),
f"{L}: kube_vip_address {vip} is ALSO a node address "
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
"node's own address cannot float off that node")
ov = overlay_vars(getattr(e, "overlay", ""))
gw = as_ip(ov.get("GATEWAY_VIP", ""))
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
check(gw is not None and lo is not None and hi is not None,
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
"the Service-LB allocator cannot be checked")
if gw is not None:
check(vip != gw,
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
if lo is not None and hi is not None:
check(not (lo <= vip <= hi),
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
"Service, and the two allocators would fight over ARP for it")
check(lo <= hi,
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
"used to prove the VIP is outside it")
# ==================================================================== DISTRO: k3s only
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
is not None,
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
check("get.k3s.io" in K3S_TEXT,
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
"distro (ADR-0001) and no other distro may stand in for it")
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
for bad, why in (
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
"nodes (no external datastore)')"),
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
):
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
and re.search(bad, p.read_text(errors="replace"))]
check(not hits,
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
f"mode, and {bad!r} contradicts it")
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
"founded, so there is no HA datastore to join")
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
K3S_TEXT)
check(init_when is not None,
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
"ungated founder flag would re-found the cluster on every server (split brain)")
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
check(join is not None,
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
"— without a join target the servers are separate single-node clusters")
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
K3S_TEXT, re.S)
check(join_when is not None,
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
"would also try to join itself")
k3s_play = PLAYBOOKS / "k3s.yml"
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
if k3s_play.is_file():
play_txt = k3s_play.read_text()
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
check(sorted(modes) == ["primary", "secondary"],
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
"one secondary play must exist, or the founder/joiner split is not authored")
check(re.search(r"serial:\s*1", play_txt) is not None,
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
"converging two of three servers at once drops embedded etcd below quorum, which "
"is exactly the outage an HA control plane exists to avoid")
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
"group name means this play can only ever build ONE environment, so prod cannot "
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
# =========================================== SERVING-CERT SANs: VIP + every server
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
check("kube_vip_address" in sans,
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
"API through it fails TLS validation and the HA endpoint is unusable")
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
check(re.search(r"tls-san:", K3S_TEXT) is not None,
"the SAN list is passed only as an install flag and never persisted to "
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
# plane the CNI's control-plane path depended on one machine. Measured on live prod
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
# policy update lands at all.
#
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
# cluster can only tell you about itself.
#
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
# "Cilium uses the HA address" requires an HA address to name; before the split there was
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
# the Flux half agrees with the Ansible half per environment.
CIL_ROLE = ANS / "roles" / "cilium"
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
check(cil_values_tpl.is_file(),
"roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
"API endpoint its kube-proxy replacement dials, are not authored")
if cil_values_tpl.is_file():
# Parse the rendered-shape template, never its prose: strip comment lines first, exactly
# as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
check(m is not None,
"roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
"(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
if m is not None:
host_expr = m.group(1).strip()
check("k3s_apiserver_endpoint" in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
"variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
"expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
"control-plane path depend on ONE server of an HA control plane, and the failure "
"is silent: the datapath keeps running from Cilium's cache while no service or "
"policy update ever lands (measured on prod 2026-09-06)")
check("k3s_api_host" not in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
"k3s_api_host — the FOUNDER's own address. That variable exists only for the "
"cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
"failure by construction and must never be a steady-state endpoint")
# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
# than restated: an environment overrides one line (kube_vip_address) and both the floating
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
# hitting.
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
check("kube_vip_address" in endpoint_def,
f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
"from kube_vip_address, so that overriding the VIP for an environment also moves what "
"that environment's CNI dials. A restated literal can silently disagree with the VIP "
"kube-vip is actually floating")
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
"group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
"addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
"build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
for e in envs:
L = e.label
ov = overlay_vars(getattr(e, "overlay", ""))
k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
vip = as_ip(e.var("kube_vip_address"))
check(k3s_api_host_knob is not None,
f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
"token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
"kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
"fails the whole platform Kustomization")
if k3s_api_host_knob is not None and vip is not None:
check(k3s_api_host_knob == vip,
f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
f"but this environment's control-plane VIP (kube_vip_address, which is what "
f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
"endpoints into the same CNI and fight over it. If they differ because the "
"overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
"an HA control plane must not dial one server")
check(k3s_api_host_knob not in set(
as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
for h in hosts_in(str(e.var("k3s_servers_group")))),
f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
"environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
"single machine on a control plane built specifically to survive losing one")
# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
# being replaced could translate: deadlocked on 2 of 3 nodes).
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
"config after the bootstrap, so the parity contract has only one side")
if hr.is_file():
hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
"\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
"address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
"incident verbatim")
# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
site_play = PLAYBOOKS / "site.yml"
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
"whose ORDER can be checked")
if site_play.is_file():
imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
check(needed in imports,
f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
"k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
"CNI, not an optional extra step — leaving it out of the bring-up means a cold "
"build installs a CNI pointed at an address nothing announces")
if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
"kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
"unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
"on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
"ships as a k3s auto-deploy manifest and needs a cluster to apply it")
check(re.search(r"wait_for", VIP_TEXT) is not None
and re.search(r"kube_vip_address", VIP_TEXT) is not None,
"roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
"same as having a floating endpoint (k3s still has to apply it, the image still has to "
"pull, leader election still has to settle). Since the CNI is installed immediately "
"afterward against this exact address, an unverified VIP turns a missing endpoint into "
"unreachable cilium-agents diagnosed from the far side")
# ============================================ FLOATING ENDPOINT: kube-vip on every server
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
# thing that ships.) Parser contract: the template writes each env var as an adjacent
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
check(len(VIP_TEMPLATES) >= 1,
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
vip_manifest = "\n".join(
"\n".join(l for l in p.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
for p in VIP_TEMPLATES)
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
vip_manifest))
check(len(vip_env) >= 6,
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
"parser lost its grip on the file, so the assertions below would assert nothing")
check(vip_env.get("cp_enable") == "true",
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
"mode must be ON or it floats no API VIP at all")
check(vip_env.get("svc_enable") == "false",
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
check(vip_env.get("vip_leaderelection") == "true",
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
"— without leader election the VIP has no defined owner to fail over FROM, which is "
"the whole HA mechanism")
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
"prod (R-ENV-02) and would not match the serving-cert SAN list")
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
"node, which is the opposite of a floating endpoint")
vip_play = PLAYBOOKS / "kube-vip.yml"
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
if vip_play.is_file():
vp = vip_play.read_text()
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
"group through the same environment-scoped variable the other plays use, so the "
"manifest exists on EVERY server and the VIP survives losing any one of them")
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
"manifest somewhere k3s never reads and come up with no VIP at all")
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
env_words = re.compile(r"prod|dev", re.I)
forked = sorted(n for n in (role_names | play_names)
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
check(not forked,
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
for e in envs:
if e.path.name != "vars.yml":
continue
# Every key a non-default environment sets must be a variable something already reads,
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
for k in sorted(e.vars):
if k.startswith("ansible_"):
continue
check(consumers.count(k) > 1,
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
"environment override that nothing reads is not a scoped override, it is dead "
"configuration (and would make prod silently inherit a dev default)")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 40,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
for n in NOTES:
print(f" note: {n}")
summary = "; ".join(
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
for e in envs)
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
"embedded etcd on every server with no external datastore, a leader-elected "
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
"collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
"dials that VIP, not the founder, in both environments and on both sides of the "
"[parity] contract), from one code path for both environments. The LIVE "
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
PY
conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml
+63
View File
@@ -0,0 +1,63 @@
# =============================================================================
# L0 STATIC — Keycloak client secrets are declared consistently, ordered by a REAL
# Flux edge, and healed by a reconciler whose behaviour is proven offline
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-IDP-03 (identity config is declarative) and R-IDP-04 (IdP secrets SOPS-encrypted)
# are both implemented through ONE mechanism that FAILS OPEN: the KeycloakRealmImport
# resolves each client's secret from a `(env:NAME)` placeholder mapped to a key of the
# SOPS-encrypted keycloak-realm-clients Secret. When the operator cannot resolve a
# placeholder it stores the LITERAL placeholder text as the client secret AND THE IMPORT
# JOB STILL REPORTS SUCCESS — then rests at Done=True / Started=False and never re-runs.
#
# Measured 2026-09-06: prod had all 12 clients holding the literal placeholder (every
# OIDC login failed with "Invalid client or Invalid client credentials"); dev had exactly
# one (oauth2-proxy), because that Secret key was added AFTER dev's import completed.
# Declarative config that silently does not apply is not declarative config.
#
# GIVEN the repository as authored,
# THEN every confidential client declares a placeholder, every placeholder is defined,
# resolves to the keycloak-realm-clients Secret, and names a key that EXISTS in the
# SOPS file under the `<clientId>-client-secret` contract the reconciler derives
# from; public clients declare no secret; the Flux Kustomization applying the Secret
# is a transitive `dependsOn` ancestor (wait: true) of the ones applying the import
# and the reconciler — a real edge, not a comment; the reconciler is reconciled from
# BOTH cluster entrypoints, mounts only what it needs, carries no ServiceAccount
# token and contains no dollar sign; and its own `--selftest` (extracted from the
# shipped ConfigMap and executed) proves compare-then-write, idempotence,
# no-secret-logging, the endpoint allow-list and fail-closed behaviour.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
# Test wraps the SAME script.
#
# NOT proven here: that the LIVE realm's stored secrets match git (that is
# tests/l2-behavioral/keycloak-client-secrets-match-git), that the SOPS ciphertext
# decrypts to a usable value (key PRESENCE is offline-checkable; plaintext is not), or
# that the CronJob's schedule fires.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: keycloak-client-secret-reconciler
labels:
conformance.home.k3s/R-IDP-03: "true"
conformance.home.k3s/R-IDP-04: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every OIDC client's secret is declared once, resolvable, ordered behind a real Flux
dependency, and continuously reconciled — so an unresolved import placeholder can
neither be merged nor silently persist in the realm.
spec:
steps:
- name: audit-declaration-ordering-and-reconciler
description: >-
Assert the realm import, the SOPS Secret and the reconciler agree; that the Flux
ordering is a real dependsOn edge; and run the reconciler's own selftest.
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh
Executable
+529
View File
@@ -0,0 +1,529 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — keycloak-client-secret-reconciler
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
# — spec docs/15-identity-and-access.md)
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
# — spec docs/15-identity-and-access.md)
#
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
# ---------------------------------------------------------------
# platform/keycloak/keycloak-realm-import.yaml declares 12 confidential OIDC clients
# whose secrets are `(env:NAME)` import placeholders, mapped by spec.placeholders to
# keys of the SOPS-encrypted keycloak-realm-clients Secret. Measured on both clusters
# on 2026-09-06:
#
# * when the Keycloak Operator cannot resolve a placeholder it stores the LITERAL
# placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS;
# * the KeycloakRealmImport then rests at Done=True / Started=False and NEVER
# re-runs, so it never self-heals.
#
# Consequence: on prod all 12 clients held the literal placeholder and every OIDC
# login on the cluster failed ("Invalid client or Invalid client credentials"); on dev
# exactly one (oauth2-proxy) was broken, because that Secret key was added AFTER dev's
# import had already completed and a completed import is never re-run.
#
# Nothing in the repo asserted the declaration was self-consistent, and nothing
# corrected the realm afterwards. This check is the static half of the fix (the
# reconciler CronJob is the runtime half, and this check proves that too).
#
# GIVEN the repository as authored,
# THEN (1) DECLARATION IS COMPLETE: every confidential client in the realm import
# declares a secret placeholder; every placeholder it uses is defined in
# spec.placeholders; every placeholder resolves to the keycloak-realm-clients
# Secret; and every referenced key EXISTS in the SOPS file. A client whose
# key is missing is exactly the dev failure, and it is now unmergeable.
# (2) THE NAMING CONTRACT HOLDS: each key is `<clientId>-client-secret`. That is
# what lets the reconciler derive the client set from the mounted Secret with
# no second declaration — so it must be enforced, not assumed. No orphan
# placeholder, and no orphan Secret key (an orphan key would make every
# reconciler run fail with NOT FOUND).
# (3) PUBLIC CLIENTS DECLARE NO SECRET (the `kubernetes` client is PKCE-public;
# giving it a secret would be a silent downgrade).
# (4) ORDERING IS A REAL FLUX EDGE, NOT A COMMENT: the Flux Kustomization that
# renders the Secret is a transitive `dependsOn` ancestor — with `wait: true`
# — of the one that renders the realm import and the reconciler. The repo's
# hard-won rule: a dependency that is only DOCUMENTED is not a dependency.
# (5) THE RECONCILER IS WIRED AND SAFE: it is reconciled from BOTH cluster
# entrypoints, mounts exactly the two Secrets it needs at the paths its code
# defaults to, carries no ServiceAccount token, and contains no dollar sign
# (it ships inside a substituted Flux path, where an unescaped dollar-brace
# expansion is rewritten before any container runs).
# (6) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
# script from the ConfigMap and executes its `--selftest`, which drives the
# real code paths against an in-process stub Keycloak and asserts
# compare-then-write, idempotence (zero writes on a converged realm),
# no-secret-logging, the endpoint allow-list (users/groups/roles are
# unreachable) and fail-closed behaviour. What CI proves is byte-for-byte
# what ships.
# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
# blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
# platform/keycloak/ to these requirement IDs, and an L2 test exists that
# audits the LIVE realm. An unreferenced test is precisely the "looks applied
# and isn't" failure this repo keeps hitting.
#
# NOT proven here (named honestly, in the style of the sibling L0 checks):
# - that the LIVE realm's stored secrets match git. That is cluster state; it is
# conformance/tests/l2-behavioral/keycloak-client-secrets-match-git.
# - that the SOPS ciphertext decrypts to a usable secret. Key PRESENCE is checkable
# offline; the plaintext is not, and must not be.
# - that the CronJob's image can be pulled or its schedule fires.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
SOPS_FILE="platform/keycloak/keycloak-realm-clients.sops.yaml"
RECON_FILE="platform/keycloak/client-secret-reconciler.yaml"
# Honest SKIP (never green) if the thing under test is absent entirely.
for f in "$IMPORT_FILE" "$SOPS_FILE" "$RECON_FILE"; do
if [ ! -f "${ROOT}/${f}" ]; then
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
exit 3
fi
done
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit of the Keycloak client-secret declaration, its Flux ordering, and the
reconciler that heals it. PyYAML (the same dependency tests/l0-static/no-plaintext-secrets
and least-privilege-rbac already take) parses the manifests; lib/status.py is IMPORTED for
the Flux graph walk rather than forked, so there is one definition of "what a Flux path
renders" in the harness."""
from __future__ import annotations
import re
import subprocess
import sys
import tempfile
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
import status as st # noqa: E402 — reuse the walk, don't fork it
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
SOPS_FILE = "platform/keycloak/keycloak-realm-clients.sops.yaml"
RECON_FILE = "platform/keycloak/client-secret-reconciler.yaml"
SECRET_NAME = "keycloak-realm-clients"
ADMIN_SECRET = "keycloak-admin-credentials"
KEY_SUFFIX = "-client-secret"
SECRETS_MOUNT = "/etc/keycloak-realm-clients"
DOLLAR = chr(36)
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def load(rel):
return list(yaml.safe_load_all((ROOT / rel).read_text()))
# ================================================================= (1)-(3) declaration
realm_import = next((d for d in load(IMPORT_FILE)
if d and d.get("kind") == "KeycloakRealmImport"), None)
check(realm_import is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document found")
if realm_import is None:
print("CHECK FAIL: cannot parse the realm import", file=sys.stderr)
raise SystemExit(1)
placeholders = realm_import["spec"].get("placeholders") or {}
clients = realm_import["spec"]["realm"].get("clients") or []
check(len(clients) > 0, f"{IMPORT_FILE}: parsed zero clients — the file shape changed and "
f"this check would silently audit nothing")
# The operator's placeholder syntax, spelled without a literal dollar so this file is
# itself safe to grep for one. Matches `<dollar>(env:NAME)`.
PLACEHOLDER_RE = re.compile(re.escape(DOLLAR) + r"\(env:([A-Za-z_][A-Za-z0-9_]*)\)")
used_placeholders: set[str] = set()
declared_pairs: dict[str, str] = {} # clientId -> secret key in the Secret
for c in clients:
cid = c.get("clientId", "<unnamed>")
secret = c.get("secret")
public = bool(c.get("publicClient"))
if public:
# (3) a PKCE public client must not carry a secret at all.
check(secret is None,
f"{IMPORT_FILE}: client '{cid}' is publicClient but declares a secret — a "
f"public client holds no secret; declaring one is a silent downgrade")
continue
# (1) every confidential client must declare one.
check(isinstance(secret, str) and secret,
f"{IMPORT_FILE}: confidential client '{cid}' declares NO secret. The operator "
f"would generate a random one that exists nowhere in git, so nothing could "
f"ever reconcile it and the consuming app's mirrored credential would be wrong")
if not isinstance(secret, str):
continue
m = PLACEHOLDER_RE.fullmatch(secret.strip())
check(m is not None,
f"{IMPORT_FILE}: client '{cid}' has a secret that is not an (env:NAME) "
f"placeholder. A literal secret in git violates R-IDP-04/R-SEC-01; the only "
f"permitted form is a placeholder resolved from the {SECRET_NAME} Secret")
if m is None:
continue
name = m.group(1)
used_placeholders.add(name)
# (1) the placeholder must be DEFINED. An undefined one is stored literally by the
# operator, with a successful-looking import — the whole incident.
ph = placeholders.get(name)
check(ph is not None,
f"{IMPORT_FILE}: client '{cid}' uses placeholder '{name}' but spec.placeholders "
f"does not define it. The operator stores the LITERAL placeholder as the client "
f"secret and STILL reports the import successful (measured 2026-09-06)")
if ph is None:
continue
ref = (ph or {}).get("secret") or {}
check(ref.get("name") == SECRET_NAME,
f"{IMPORT_FILE}: placeholder '{name}' reads Secret '{ref.get('name')}', not "
f"'{SECRET_NAME}'. The reconciler mounts only {SECRET_NAME}, so a second source "
f"would be declared but never healed")
key = ref.get("key")
check(isinstance(key, str) and key,
f"{IMPORT_FILE}: placeholder '{name}' names no Secret key")
if not isinstance(key, str):
continue
# (2) the naming contract the reconciler derives its client set from.
check(key == cid + KEY_SUFFIX,
f"{IMPORT_FILE}: client '{cid}' reads Secret key '{key}', but the reconciler "
f"derives the clientId from the key name, so it MUST be "
f"'{cid}{KEY_SUFFIX}'. Rename the key (and the SOPS entry) or the reconciler "
f"would look up a client that does not exist")
declared_pairs[cid] = key
# (2) no orphan placeholder: a defined-but-unused placeholder is dead config that will
# quietly rot out of sync with the Secret.
for name in sorted(set(placeholders) - used_placeholders):
check(False,
f"{IMPORT_FILE}: spec.placeholders defines '{name}' but no client uses it — "
f"dead declaration; remove it or wire it to the client that needs it")
# ---- the SOPS Secret: keys are cleartext in a SOPS-encrypted file; values are not.
sops_doc = next((d for d in load(SOPS_FILE) if d and d.get("kind") == "Secret"), None)
check(sops_doc is not None, f"{SOPS_FILE}: no Secret document found")
sops_keys = set((sops_doc or {}).get("stringData") or {})
check(len(sops_keys) > 0,
f"{SOPS_FILE}: parsed zero stringData keys — the file shape changed and this check "
f"would silently audit nothing")
check((sops_doc or {}).get("metadata", {}).get("name") == SECRET_NAME,
f"{SOPS_FILE}: the Secret is not named {SECRET_NAME}")
for cid, key in sorted(declared_pairs.items()):
# (1) THE DEV FAILURE, made unmergeable: a client declared in the import whose key
# is absent from the Secret imports "successfully" with a literal placeholder.
check(key in sops_keys,
f"{SOPS_FILE}: client '{cid}' needs key '{key}' and it is NOT in the Secret. "
f"The realm import would still report SUCCESS and store the literal placeholder "
f"as that client's secret — this is exactly how dev's oauth2-proxy broke")
# (2) no orphan Secret key: the reconciler treats every `<x>-client-secret` key as a
# client that MUST exist, and fails loudly if it does not. Catch it here instead.
for key in sorted(k for k in sops_keys if k.endswith(KEY_SUFFIX)):
cid = key[: -len(KEY_SUFFIX)]
check(cid in declared_pairs,
f"{SOPS_FILE}: key '{key}' has no matching client in {IMPORT_FILE}. The "
f"reconciler would look up client '{cid}', not find it, and fail every run")
# ===================================================== (4) ordering is a real Flux edge
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`; a doc
# is an entrypoint only if it carries the kustomize.toolkit apiVersion, `kind:
# Kustomization` and a spec-level `path:`. Overlay `patch: |-` bodies carry Kustomizations
# WITHOUT a path and are correctly skipped (they are patches, not entrypoints).
_DEPENDSON_RE = re.compile(r"^(\s*)dependsOn:\s*$", re.M)
_DEP_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
_WAIT_RE = re.compile(r"^\s{2,6}wait:\s*true\s*(?:#.*)?$", re.M)
def depends_on(doc: str) -> list[str]:
m = _DEPENDSON_RE.search(doc)
if not m:
return []
indent = len(m.group(1))
out: list[str] = []
for raw in doc[m.end():].splitlines():
if raw.strip() == "" or raw.lstrip().startswith("#"):
continue
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
break # dedented out of the list
nm = _DEP_NAME_RE.match(raw)
if nm:
out.append(nm.group(1))
return out
kustomizations: dict[str, dict] = {}
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
text = yfile.read_text()
if st.FLUX_API not in text:
continue
for doc in st._DOC_SPLIT_RE.split(text):
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
continue
pm = st._PATH_RE.search(doc)
if not pm:
continue
nm = st._NAME_RE.search(doc)
name = nm.group(1) if nm else "?"
kustomizations[name] = {
"file": str(yfile.relative_to(ROOT)),
"path": pm.group(1),
"dependsOn": depends_on(doc),
"wait": bool(_WAIT_RE.search(doc)),
}
check(len(kustomizations) > 0,
"no Flux Kustomization with a spec.path found under clusters/ — the parser or the "
"repo layout changed; this check would silently audit nothing")
def direct_render(path_spec: str) -> set[str]:
"""Files a Flux `spec.path` renders ITSELF — a kustomize walk that deliberately does
NOT expand nested Flux Kustomizations.
lib/status.py's reconciled_files() follows every `spec.path` it discovers, which is
right for "is this file applied to the cluster at all" but wrong for OWNERSHIP: the
root Kustomization's path is clusters/<env>, so a transitive walk makes the root the
owner of every file in the repo and the ordering assertion below becomes vacuous
(measured: it reported the root as an owner of all three keycloak files). `resources:`
parsing itself is st.kustomize_resources, not forked.
"""
rel = path_spec.lstrip("./") or "."
out: set[str] = set()
seen: set[str] = set()
def walk(d: str) -> None:
if d in seen:
return
seen.add(d)
kfile = ROOT / d / "kustomization.yaml"
if not kfile.is_file():
return
out.add(str(kfile.relative_to(ROOT)))
for item in st.kustomize_resources(kfile):
if "://" in item:
continue # remote base: not repo state
target = (ROOT / d / item).resolve()
try:
trel = target.relative_to(ROOT).as_posix()
except ValueError:
continue # outside the repo
if target.is_dir():
walk(trel)
elif target.is_file():
out.add(trel)
if (ROOT / rel).is_dir():
walk(rel)
return out
rendered_by = {n: direct_render(k["path"]) for n, k in kustomizations.items()}
def owners(rel_file: str) -> list[str]:
return sorted(n for n, files in rendered_by.items() if rel_file in files)
def ancestors(name: str, seen=None) -> set[str]:
"""Transitive dependsOn closure of `name` (its dependencies, direct and indirect)."""
seen = seen if seen is not None else set()
for dep in kustomizations.get(name, {}).get("dependsOn", []):
if dep in seen:
continue
seen.add(dep)
ancestors(dep, seen)
return seen
secret_owners = owners(SOPS_FILE)
import_owners = owners(IMPORT_FILE)
recon_owners = owners(RECON_FILE)
check(bool(secret_owners),
f"{SOPS_FILE} is rendered by NO Flux Kustomization — the client secrets are in git "
f"but never applied")
check(bool(import_owners),
f"{IMPORT_FILE} is rendered by NO Flux Kustomization")
# (5) the reconciler must actually be reconciled, or it is decoration.
check(bool(recon_owners),
f"{RECON_FILE} is rendered by NO Flux Kustomization — the healer would sit in git "
f"and never run, which is the 'looks applied and isn't' failure it exists to end")
for consumer_file, consumer_owners in ((IMPORT_FILE, import_owners),
(RECON_FILE, recon_owners)):
for co in consumer_owners:
for so in secret_owners:
if so == co:
continue # same Kustomization: one atomic apply, ordering moot
anc = ancestors(co)
check(so in anc,
f"ORDERING IS ONLY DOCUMENTED: '{co}' renders {consumer_file} but does "
f"not (transitively) dependsOn '{so}', which renders the "
f"{SECRET_NAME} Secret. Without that edge the consumer can reconcile "
f"before the secret exists, and the Keycloak Operator's response to a "
f"missing placeholder is to store it LITERALLY and report success")
if so in anc:
check(kustomizations[so]["wait"],
f"'{so}' (which renders the {SECRET_NAME} Secret) does not set "
f"wait: true, so '{co}' can proceed before it is applied")
# ------ both environments must reconcile the reconciler, not just dev.
for env in ("dev", "prod"):
entry = f"clusters/{env}"
if not (ROOT / entry).is_dir():
continue
applied, _ = st.reconciled_files(ROOT, entry)
check(RECON_FILE in applied,
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the healer would "
f"exist on one cluster only, and prod is where all 12 clients were broken")
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
# ============================================ (5) the reconciler manifest is safe + wired
recon_docs = [d for d in load(RECON_FILE) if d]
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
script = ""
if cm is not None:
data = cm.get("data") or {}
check(len(data) == 1,
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
f"(found {sorted(data)}) — a second copy would be untested by this gate")
script = next(iter(data.values()), "")
check(len(script.splitlines()) > 50,
f"{RECON_FILE}: the extracted script is implausibly short "
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
f"selftest below would prove nothing")
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
# postBuild.substituteFrom, so envsubst rewrites it before any container runs. The
# script is written with NO dollar sign at all, so the hazard is structural rather
# than escaped and no future edit has to reason about escaping.
check(DOLLAR not in script,
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside a "
f"SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
f"never needs one; chr(36) spells it where a literal is genuinely required")
if cj is not None:
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
ctr = pod["containers"][0]
vols = {v["name"]: v for v in pod.get("volumes", [])}
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
secret_vols = [v for v in vols.values()
if (v.get("secret") or {}).get("secretName") == SECRET_NAME]
check(len(secret_vols) == 1,
f"{RECON_FILE}: the CronJob must mount the {SECRET_NAME} Secret — that mount IS "
f"the desired state it reconciles towards")
if secret_vols:
mp = mounts.get(secret_vols[0]["name"], {}).get("mountPath")
check(mp == SECRETS_MOUNT,
f"{RECON_FILE}: {SECRET_NAME} is mounted at {mp}, but the script defaults "
f"to {SECRETS_MOUNT} and no env var overrides it")
cm_vols = [v for v in vols.values()
if (v.get("configMap") or {}).get("name") == (cm or {}).get(
"metadata", {}).get("name")]
check(len(cm_vols) == 1,
f"{RECON_FILE}: the CronJob does not mount the reconciler ConfigMap, so it "
f"would run nothing")
env_srcs = {e["name"]: e for e in ctr.get("env", [])}
for var in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
ref = (env_srcs.get(var, {}).get("valueFrom") or {}).get("secretKeyRef") or {}
check(ref.get("name") == ADMIN_SECRET,
f"{RECON_FILE}: {var} must come from the {ADMIN_SECRET} Secret "
f"(got {ref.get('name')!r}) — the admin credential is never inlined")
for var in ("KEYCLOAK_BASE_URL", "KEYCLOAK_REALM"):
check(env_srcs.get(var, {}).get("value"),
f"{RECON_FILE}: {var} is not set; the script fails closed without it, but "
f"the CronJob should not ship broken")
# R-SEC-02: it talks only to Keycloak's REST API and needs no Kubernetes API rights.
check(pod.get("automountServiceAccountToken") is False,
f"{RECON_FILE}: the reconciler needs NO Kubernetes API access (its inputs are "
f"mounted Secrets) — set automountServiceAccountToken: false")
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
f"{RECON_FILE}: set concurrencyPolicy: Forbid so a run stalled on an "
f"unavailable Keycloak is not lapped by the next tick")
img = ctr.get("image", "")
check(re.search(r":[0-9]", img),
f"{RECON_FILE}: container image {img!r} is not pinned to an explicit version")
# ============================== (6) the reconciler's BEHAVIOUR, proven by its own selftest
if script:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "reconcile.py"
p.write_text(script)
proc = subprocess.run([sys.executable, str(p), "--selftest"],
capture_output=True, text=True, timeout=180)
tail = (proc.stdout + proc.stderr).strip().splitlines()
check(proc.returncode == 0,
"the reconciler's --selftest FAILED (compare-then-write / idempotence / "
"no-secret-logging / endpoint allow-list / fail-closed):\n "
+ "\n ".join(tail[-12:]))
if proc.returncode == 0:
print(" reconciler --selftest: " + (tail[-1] if tail else "(no output)"))
# ======================================================= (7) this check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
check("keycloak-client-secret-reconciler/check.sh" in L0,
"ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
"requirement-scoped conformance selection can filter this check out of a gate run, "
"so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
"it. An unreferenced test is the failure this whole change exists to end")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
"ci/touched-requirements.sh has no platform/keycloak/* entry in its path->requirement "
"map, so editing the realm import or this reconciler attributes NO requirement and "
"the L2 keycloak tests are filtered out of the merge gate's touched subset")
l2 = ROOT / "conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml"
check(l2.is_file(),
"the LIVE-realm counterpart (conformance/tests/l2-behavioral/"
"keycloak-client-secrets-match-git) is missing — this static check cannot see a "
"realm that drifted after apply, which is the whole failure mode")
if l2.is_file():
check("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in l2.read_text(),
"the L2 test does not assert the reconciler's audit sentinel, so it could pass "
"on a Job that never ran the audit")
# ------------------------------------------------------------------------------ report
print(f"realm import: {len(clients)} client(s), {len(declared_pairs)} confidential with a "
f"declared secret, {len(placeholders)} placeholder(s)")
print(f"{SECRET_NAME} Secret: {len(sops_keys)} key(s) in git (names only; values stay "
f"SOPS-encrypted)")
print("flux ordering: " + ", ".join(
f"{f.split('/')[-1]} <- {owners(f)}" for f in (SOPS_FILE, IMPORT_FILE, RECON_FILE)))
print()
if not FAILURES:
print(f"check OK: keycloak-client-secret-reconciler — {CHECKS} assertions "
f"(R-IDP-03, R-IDP-04)")
raise SystemExit(0)
print(f"CHECK FAIL (keycloak-client-secret-reconciler): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml
+68
View File
@@ -0,0 +1,68 @@
# =============================================================================
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
# COUPLING was untested, which is what makes the class reintroducible.
#
# GIVEN the repository as authored,
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
# INDEPENDENT detector is actually refused by that guard, in base and in every
# environment overlay;
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
# refuses;
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
# (the "three guards died at once" regression);
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
# — the exact anchoring defect above — and still accepts a real endpoint;
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
# testing a copy if the guard stops being a self-contained predicate.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
# wraps the SAME script.
#
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
# authored backup target cannot be a placeholder that the guard fails to recognise.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: placeholder-guard-coupling
labels:
conformance.home.k3s/R-ENV-02: "true"
conformance.home.k3s/R-STATE-03: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every placeholder-shaped value authored in cluster-vars, for any environment
overlay, is actually recognised as a placeholder by the real bootstrap guard that
is supposed to refuse it — so a value change in one file cannot silently disarm a
guard in another and enable velero against an endpoint that does not exist.
spec:
steps:
- name: couple-authored-placeholders-to-the-real-guard
description: >-
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
every authored cluster-vars value the repo actually carries, and assert the guard
refuses each one an independent detector calls a placeholder.
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/placeholder-guard-coupling/check.sh
Executable
+516
View File
@@ -0,0 +1,516 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — placeholder-guard-coupling
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
# — spec docs/02-environments-and-provisioning.md)
# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)."
# — spec docs/04-storage-and-state.md)
#
# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE
# --------------------------------------------------------------------
# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a
# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match —
# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped
# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname
# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and
# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident.
#
# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was
# never tested: a guard in one file keying on a value in another, with nothing tying them
# together. This check is that tie, and it is derived from BOTH real files — no current
# string is hardcoded here, so it does not rot when the placeholder or the guard changes.
#
# HOW THE REAL GUARD IS EXERCISED (and why not a copy)
# ---------------------------------------------------
# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
# --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
# those, then calls the real function. That is the real bytes, not a reimplementation:
# * every function whose name ends `_is_placeholder` is discovered, not named here;
# * a duplicate definition of one is a failure (which of the two would run?);
# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
# else, no command substitution, no call to another function. If the guard ever grows
# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
# remedy (move the predicate into a small sourceable library that standup.sh sources)
# rather than quietly falling back to testing a copy of the logic;
# * a CANARY pins the extraction itself: the extracted function must refuse the empty
# string and accept a realistic endpoint. An extraction that grabbed the wrong lines,
# or a predicate degraded to always-true / always-false, fails there — this repo has
# already been bitten by a check that passed when perturbed because it asked a question
# that was always satisfiable.
# There is deliberately NO second implementation of the matching logic anywhere in this file.
#
# THE INDEPENDENT DETECTOR (the other half of the comparison)
# ----------------------------------------------------------
# Asking "does the guard agree with the guard?" proves nothing, so the set of
# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the
# UNION of two independent sources:
# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates and the empty
# string. These are NOT derived from the guard, so DELETING a guard arm cannot make a
# value stop being placeholder-shaped — it makes assertion (1) fail, which is the point.
# A value using an idiom the guard does not know is a finding either way: fix it by
# giving the knob a real value, or by teaching the guard the idiom.
# * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED.
# Stripping the anchors is what makes the comparison non-circular in the dimension that
# actually broke: the detector matches a word ANYWHERE, case-insensitively, and the
# guard must agree. A prefix-anchored guard cannot pass.
#
# GIVEN the repository as authored,
# WHEN the real predicate(s) are extracted from bootstrap/standup.sh and executed,
# THEN (1) VALUES: for every knob a guard is responsible for, every authored value — the
# base default AND each environment overlay's value — that the independent
# detector calls placeholder-shaped is REFUSED by the real guard, and every value
# it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero
# suspended forever, which is the same class of silent failure);
# (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a
# placeholder-shaped value for a knob that no guard refuses. (Base alone is
# exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` —
# and its loud sentinels exist precisely to be overridden. A base placeholder an
# overlay forgets to override IS caught, because it becomes that overlay's
# effective value.)
# (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`,
# following one hop of variable aliasing) applies a placeholder guard to it. This
# is the "all three guards died at once" regression stated as a property: the
# guard being correct is worthless if a consumer stops calling it.
# (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
# value carrying that word in the MIDDLE (not at the start) — the precise defect
# of 2026-09-05 — while still accepting a realistic endpoint.
# (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim
# from the file and self-contained, the canary holds, no divergent second copy of
# the matching logic exists elsewhere in standup.sh, and a floor on the number of
# assertions run keeps a pass from degenerating into a no-op.
# AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
# ci/gate.sh sets the merge block when that stage fails.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps)
# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
# - It does not execute standup.sh's dispatch path, so it cannot prove the guard is
# REACHED in a real run — only that, when reached, it refuses what it must. The
# consumer property (3) is a static approximation of reachability: it is a text-level
# taint trace over one file, not an interpreter, and a guard whose result is computed
# and then ignored would satisfy it.
# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
# asserted by the node_baseline role itself, at play time).
# - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit
# to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so
# the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh
# load_touched_requirements) — this check runs. An edit to clusters/* maps to
# R-ENV-02, which this check carries, so a cluster-vars value change also selects it.
# A change that touches standup.sh ALONGSIDE some unrelated mapped path (say
# platform/cilium/) would scope the run away from this check; closing that needs a
# `bootstrap/*` entry in the path map, which is not this check's file to edit.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}"
cd "${ROOT}"
if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then
echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2
exit 3
fi
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT
# ---------------------------------------------------------------------------------------
# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files,
# and run every assertion that does not need a shell. Writes:
# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else
# ${WORK}/corpus.tsv <guard>\t<expect: placeholder|usable>\t<origin>\t<value>
# ${WORK}/findings.txt one line per failed assertion
# ${WORK}/counts.txt <assertions-run>
# ---------------------------------------------------------------------------------------
"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY'
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def bail(msg):
"""A failure that makes the rest of the check meaningless: report and stop."""
print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr)
(WORK / "findings.txt").write_text(msg + "\n")
(WORK / "counts.txt").write_text("0\n")
(WORK / "corpus.tsv").write_text("")
(WORK / "guards.sh").write_text("")
raise SystemExit(1)
TEXT = (ROOT / STANDUP_REL).read_text()
LINES = TEXT.splitlines()
# ------------------------------------------------------------------ (5) extract the guard
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
defs: dict[str, list[int]] = {}
for i, line in enumerate(LINES):
m = DEF_RE.match(line)
if m:
defs.setdefault(m.group(1), []).append(i)
if not defs:
bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed "
"(update this check's discovery pattern in the SAME change) or it was DELETED, which "
"is the 2026-09-05 failure with the guard removed instead of disarmed.")
# An "argument-shaped" call: <name> "<something>"
CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
guard_src: list[str] = []
guard_words: dict[str, list[str]] = {}
ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
ALLOWED = (
re.compile(r"^$"),
re.compile(r"^#"),
re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
re.compile(r"^esac$"),
re.compile(r"^return\s+[01]$"),
ARM_RE,
)
for name, at in sorted(defs.items()):
check(len(at) == 1,
f"{STANDUP_REL} defines {name}() {len(at)} times (lines "
f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
"this check can no longer say which one it extracted. Keep exactly one definition.")
if len(at) != 1:
continue
start = at[0]
end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
if end is None:
bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the "
"extraction boundary this check relies on is gone. Restore the conventional "
"layout, or move the predicate into a sourceable library and point this check at it.")
body = LINES[start:end + 1]
block = "\n".join(body)
check(block in TEXT,
f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; "
"the harness would be testing something the script does not contain")
words: list[str] = []
for raw in body[1:-1]:
s = raw.strip()
if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED "
f"case/return predicate (offending line: {s!r}). This check exercises the REAL "
"function by sourcing its own lines; a body that calls out to anything else "
"cannot be extracted faithfully, and silently testing a COPY of the logic is "
"exactly the failure mode this check exists to prevent. REMEDY: move the "
"predicate into a tiny library that standup.sh sources (e.g. "
"bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do "
"NOT reimplement the matching here.")
m = ARM_RE.match(s)
if not m:
continue
for pat in m.group("pats").split("|"):
pat = pat.strip()
if pat in ('""', "''", ""):
continue # the empty-value arm: covered by the canary
words.append(pat.strip("'\"").strip("*"))
check(len(words) >= 1,
f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
"loud placeholder string has been removed, so the guard now only refuses the empty "
"value and any 'TBD'-style endpoint would sail straight through it")
guard_words[name] = words
guard_src.append(block)
(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n")
# ------------------------------------------------- no divergent second copy of the logic
# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means
# a second, independently-drifting implementation of the same test (which is how three
# guards came to disagree with one value in the first place).
guard_line_nums = {j for name, at in defs.items() if len(at) == 1
for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
if LINES[k] == "}") + 1)}
all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
divergent: list[str] = []
for j, line in enumerate(LINES):
if j in guard_line_nums or line.strip().startswith("#"):
continue
code = line.split("#", 1)[0]
hit = [w for w in all_words if w in code.upper()]
if hit:
divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
check(not divergent,
"the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
f"predicate — a SECOND implementation of the same test, which is precisely how a value "
f"change disarmed three guards at once. Call the predicate instead. Sites: "
+ " | ".join(divergent))
# ------------------------------------------------------------- cluster-vars, as authored
CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
if not CV.is_file():
bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is "
"missing and this check cannot mean anything")
base: dict[str, str] = {}
for d in yaml.safe_load_all(CV.read_text()):
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
and (d.get("metadata") or {}).get("name") == "cluster-vars":
base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()}
check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
"ConfigMap parse failed, so every value assertion below would be vacuous")
overlays: dict[str, dict[str, str]] = {}
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
env = overlay.parent.name
if env == "base":
continue
eff = dict(base)
try:
k = yaml.safe_load(overlay.read_text()) or {}
except Exception as exc: # noqa: BLE001
check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this "
"check cannot compute that overlay's effective values")
continue
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
continue
try:
ops = yaml.safe_load(p.get("patch") or "") or []
except Exception: # noqa: BLE001
continue
for op in ops if isinstance(ops, list) else []:
if not isinstance(op, dict):
continue
m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
if m and str(op.get("op", "")) in ("replace", "add"):
eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
overlays[env] = eff
check(len(overlays) >= 1,
"no clusters/<env>/kustomization.yaml overlay was parsed — assertion (2) would be vacuous")
# ------------------------------------------------ which knobs is a guard responsible for?
# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a
# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here.
FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
funcs: dict[str, list[str]] = {}
i = 0
while i < len(LINES):
m = FUNC_RE.match(LINES[i])
if m:
end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
funcs[m.group(1)] = LINES[i + 1:end]
i = end + 1
continue
i += 1
check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the "
"consumer assertion (3) would be vacuous")
CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
guarded_keys: set[str] = set()
consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func)
for fname, body in funcs.items():
text = "\n".join(body)
keys_of_var: dict[str, set[str]] = {}
for m in CVAR_RE.finditer(text):
keys_of_var.setdefault(m.group(1), set()).add(m.group(2))
# one hop of aliasing: ep="${live}" carries live's taint
for _ in range(3):
for m in ALIAS_RE.finditer(text):
src = keys_of_var.get(m.group(2))
if src:
keys_of_var.setdefault(m.group(1), set()).update(src)
called_on = {m.group(2) for m in CALL_RE.finditer(text)}
called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
# Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its
# value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts.
guarded_here: dict[str, bool] = {}
for var, keys in keys_of_var.items():
for key in keys:
guarded_here[key] = guarded_here.get(key, False) or (var in called_vars)
for key, g in guarded_here.items():
consumers.append((fname, key, g))
if g:
guarded_keys.add(key)
if not guarded_keys:
bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. "
"Either the guards were removed, or the way the script reads cluster-vars changed and "
"this check's derivation must be updated in the SAME change — it must never silently "
"verify an empty set.")
# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it.
for fname, key, guarded in sorted(consumers):
if key not in guarded_keys:
continue # nothing claims responsibility for this knob at all
check(guarded,
f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
"*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
"of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
"the guard is that failure in one function.")
# ---------------------------------------------------------------- independent detector
# Deliberately NOT derived from the guard alone (see the header): SEED is universal
# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose
# ANCHORING is wrong still has its vocabulary tested against real values.
SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
"CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
ANGLE = re.compile(r"<[^>\s]+>")
def placeholder_shaped(value: str) -> bool:
if value.strip() == "":
return True # an unset endpoint is not a target
up = value.upper()
return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value
GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None
# (1) VALUES: base default and every overlay's value, for every guarded knob.
for key in sorted(guarded_keys):
seen: list[tuple[str, str]] = []
if key in base:
seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key]))
for env, eff in sorted(overlays.items()):
seen.append((f"clusters/{env} effective:{key}", eff.get(key, "")))
check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
"protects a knob nothing authors")
for origin, value in seen:
expect = "placeholder" if placeholder_shaped(value) else "usable"
for name in sorted(guard_words):
rows.append((name, expect, origin, value))
# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob.
for env, eff in sorted(overlays.items()):
for key, value in sorted(eff.items()):
if key in guarded_keys or not placeholder_shaped(value):
continue
check(False,
f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it "
"reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay "
"that ships a placeholder no guard recognises is the 2026-09-05 failure with a "
"different knob's name on it.")
# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
# and a realistic endpoint must still be accepted.
for name, words in sorted(guard_words.items()):
for w in sorted(set(words)):
rows.append((name, "placeholder",
f"anchoring:{name}:mid-string({w})",
f"https://account-{w}-id.r2.cloudflarestorage.com"))
rows.append((name, "placeholder", f"anchoring:{name}:tail({w})",
f"https://r2-endpoint.example.invalid/{w}"))
rows.append((name, "placeholder", f"canary:{name}:empty", ""))
rows.append((name, "usable", f"canary:{name}:real-r2",
"https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"))
rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000"))
# Every guarded knob must reach the oracle once per authored source (base + each overlay),
# so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
check(len(authored) == expected_rows,
f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from "
"an overlay's effective values, and the coupling is only partly exercised")
# ------------------------------------------------------- (6) the check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
check("--layer l0" in L0,
"ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate")
check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
"ci/gate.sh does not set the merge block when the l0 stage fails")
# --------------------------------------------------------------------------- handoff
with (WORK / "corpus.tsv").open("w") as fh:
for name, expect, origin, value in rows:
fh.write(f"{name}\t{expect}\t{origin}\t{value}\n")
(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
(WORK / "counts.txt").write_text(f"{CHECKS}\n")
print(f"guards: {', '.join(sorted(guard_words))} "
f"vocabulary: {', '.join(VOCAB)}")
print(f"guarded knobs: {', '.join(sorted(guarded_keys))} "
f"consumers traced: {len(consumers)} overlays: {', '.join(sorted(overlays))}")
for name, expect, origin, value in rows:
if not origin.startswith(("anchoring:", "canary:")):
print(f" authored {origin} = {value!r} -> expect {expect}")
PY
# ---------------------------------------------------------------------------------------
# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
# ---------------------------------------------------------------------------------------
# shellcheck disable=SC1090,SC1091
source "${WORK}/guards.sh"
oracle_failures=0
oracle_rows=0
while IFS=$'\t' read -r guard expect origin value; do
[ -n "${guard}" ] || continue
oracle_rows=$((oracle_rows + 1))
if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi
if [ "${verdict}" != "${expect}" ]; then
oracle_failures=$((oracle_failures + 1))
if [ "${expect}" = placeholder ]; then
printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
else
printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
fi
fi
done < "${WORK}/corpus.tsv"
# ------------------------------------------------------------------------------ verdict
py_checks="$(cat "${WORK}/counts.txt")"
total=$((py_checks + oracle_rows))
findings="$(cat "${WORK}/findings.txt")"
# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it
# asked a question that was always satisfiable. The floors below are the minimum evidence
# for the verdict to mean anything, not a style rule.
if [ "${oracle_rows}" -lt 6 ]; then
findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised
"
fi
if [ "${total}" -lt 20 ]; then
findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless
"
fi
if [ -n "${findings}" ]; then
echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2
printf '%s' "${findings}" | sed 's/^/ - /' >&2
exit 1
fi
echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml
+77
View File
@@ -0,0 +1,77 @@
# =============================================================================
# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
# storage; it MUST NOT be an active development surface until the roadmap
# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
# differs", "Never develop against prod"); roadmap/phases.md Phase 7.
#
# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
# governance rule is verified through the property that ENFORCES it, never by observing
# that nobody misbehaved today. Here that property is structural: prod receives the base
# configuration plus environment-scoped overrides and nothing else, from the same branch
# and the same Ansible code as dev, so there is no place to put prod-only work at all.
#
# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
# group_vars retargets the k3s roles away from the global default), the prod overlay is
# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
# node count that inventory resolves to. No hostname, address or environment name is
# written into the test.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and
# the shared provisioning chain is actually retargeted at prod with --list-hosts,
# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC
# resource creates any of them (bare metal is imaged, not instantiated); prod's
# Longhorn path is node-local with one replica per node and the default
# StorageClass is the local driver; prod runs the same playbooks/roles as dev,
# retargeted only by variables the shared code already defines; the prod overlay
# carries only the base plus environment-scoped patches on the same Git ref dev
# tracks, with no prod-only manifest anywhere in the shared configuration; and prod
# artifacts exist only alongside the operator's recorded opening of the prod phase.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
# where present — ansible-playbook in --list-hosts mode, which opens no connection);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
# wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
# node-health/BIOS surveys that establish it write to gitignored artifacts, and the
# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
# HA test; asserting it here would also make this test environment-specific, since
# the same suite must pass against dev.
# * Git HISTORY: this is the working tree at HEAD.
# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
# the recorded declaration.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: prod-baremetal-not-dev-surface
labels:
conformance.home.k3s/R-ENV-03: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
inspected, then prod is authored as three bare-metal nodes with node-local storage,
bootstrapped by dev's own code retargeted by variables, receiving only
environment-scoped overrides on the same branch — so there is nowhere to develop
against prod — and it exists only because the roadmap phase was opened.
spec:
steps:
- name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh
Executable
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
# MUST NOT be an active development surface until the roadmap declares it ready."
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
# "operator declares prod hardware ready"; note "Do not begin building on prod
# hardware before this phase is explicitly opened".)
#
# HOW A "MUST NOT" IS MADE TESTABLE
# --------------------------------
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
# be. In this repo that property is structural — prod receives the base configuration plus
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
# even if someone wanted to. This check asserts that, together with the first clause's
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
# bootstrap code as dev).
#
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
# see the gap list. What IS checkable is that the repository describes prod that way and
# that no IaC creates prod nodes as virtual machines.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
# inspected — and, where offline execution is possible, the provisioning chain is
# actually retargeted at prod with `--list-hosts`,
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
# (no NAT indirection, i.e. real machines on the LAN);
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
# disjoint from the prod host set and no prod hostname appears anywhere under
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
# local path, and the replica count equals the node count (one replica per node's
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
# is not counted against this;
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
# by variables only — every prod group_vars key is an existing role/global
# variable (an invented prod-only knob would be a fork), and no prod-only role
# or playbook exists;
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
# and no prod-only manifest exists anywhere in the rendered configuration;
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
# carries the phase that opens prod build (checked when the spec repo is mounted),
# and the repository records that the operator opened it.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
# (currently uncovered) HA test; asserting it here would also make this test
# environment-specific, since the same suite must pass against dev.
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
# prod work is accompanied by the operator's recorded declaration.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
ENV_ONLY_FAILURE = re.compile(
r"vault password|vault-id|Attempting to decrypt"
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
TOFU = ROOT / "infra" / "tofu"
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
if not INV.is_file():
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
raise SystemExit(1)
# --------------------------------------------------------------- inventory parsing
ini_groups: dict[str, list[dict]] = {}
children: dict[str, list[str]] = {}
cur = None
for line in INV.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
continue
if cur is None or cur.endswith(":vars"):
continue
if cur.endswith(":children"):
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
continue
parts = s.split()
host = {"name": parts[0]}
for kv in parts[1:]:
if "=" in kv:
k, v = kv.split("=", 1)
host[k] = v
ini_groups.setdefault(cur, []).append(host)
def hosts_in(group: str, seen=None) -> list[dict]:
seen = seen or set()
if group in seen:
return []
seen.add(group)
out = list(ini_groups.get(group, []))
for child in children.get(group, []):
out += hosts_in(child, seen)
return out
def load_gv(group: str) -> dict:
d: dict = {}
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
if p.is_file():
try:
d.update(yaml.safe_load(p.read_text()) or {})
except Exception:
pass
return d
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
ALL_VARS = load_gv("all")
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
candidates = []
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
if g == "all":
continue
v = load_gv(g)
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
candidates.append((g, v))
check(len(candidates) == 1,
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
"the same roles as dev")
if not candidates:
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
PROD_GROUP, PROD_VARS = candidates[0]
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
prod_hosts = hosts_in(PROD_GROUP)
prod_names = sorted(h["name"] for h in prod_hosts)
# ------------------------------------------------- (1) exactly three, directly reachable
check(len(prod_names) == 3,
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
"says prod is a 3× mini-PC cluster")
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
for h in prod_hosts:
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
# indirection a hypervisor-hosted estate needs and bare metal does not.
check(str(h.get("ansible_port", "22")) == "22",
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
check(h.get("k3s_node_ip") == h.get("ansible_host"),
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
"bare-metal node's SSH address IS its cluster address")
# ------------------------------------------------------ (2) nothing creates prod as a VM
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
if TOFU.is_dir() else ""
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
def tf_default(var: str):
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
if not m:
return None
d = re.search(r"default\s*=\s*(.+)", m.group(1))
return d.group(1).strip() if d else None
prefix = (tf_default("node_name_prefix") or "").strip('"')
try:
node_count = int((tf_default("node_count") or "0").strip())
except ValueError:
node_count = 0
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
check(iac_names.isdisjoint(set(prod_names)),
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
"hypervisor module")
for name in prod_names:
check(name not in TF,
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
"the VM module at all")
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
# still a VM).
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
check(tf_ips.isdisjoint(prod_ips),
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
# --------------------------------------------------------------- (3) node-local storage
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
def env_values(key: str) -> dict[str, str]:
vals: dict[str, str] = {}
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
if base.is_file():
for d in yaml.safe_load_all(base.read_text()):
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
and (d.get("metadata") or {}).get("name") == "cluster-vars":
v = (d.get("data") or {}).get(key)
if v is not None:
vals["base"] = str(v)
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
env = overlay.parent.name
if env == "base":
continue
try:
k = yaml.safe_load(overlay.read_text()) or {}
except Exception:
continue
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
continue
try:
ops = yaml.safe_load(p.get("patch") or "") or []
except Exception:
continue
for op in ops if isinstance(ops, list) else []:
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
vals[env] = str(op.get("value"))
return vals
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
cluster_envs = env_values("CLUSTER_ENV")
prod_overlays = [e for e, v in cluster_envs.items()
if e != "base" and v != cluster_envs.get("base")]
check(len(prod_overlays) >= 1,
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
prod_overlay = None
for e in prod_overlays:
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
prod_overlay = e
check(prod_overlay is not None,
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
"cluster config and the machine inventory disagree about which estate is prod")
data_paths = env_values("LONGHORN_DATA_PATH")
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
if prod_overlay:
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
check(p.startswith("/"),
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
"node-local filesystem path")
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
r = replicas.get(prod_overlay, replicas.get("base", ""))
check(r.isdigit() and int(r) == len(prod_names),
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
"'local storage' on a 3-node estate means one replica per node's own disk")
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
# counted here.
default_scs = []
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
capture_output=True, text=True, check=True).stdout.split("\0"):
if not f:
continue
try:
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
str(((d.get("metadata") or {}).get("annotations") or {})
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
default_scs.append((f, str(d.get("provisioner", ""))))
except Exception:
continue
check(len(default_scs) == 1,
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
"land on by default must be unambiguous")
for f, prov in default_scs:
check(prov == "driver.longhorn.io",
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
"tier would not be its own local disks")
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
PLAYBOOKS = ANS / "playbooks"
ROLES = ANS / "roles"
prod_token = PROD_GROUP.replace("k3s_", "")
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
for entry in sorted(d.iterdir()) if d.is_dir() else []:
check(prod_token not in entry.name.lower(),
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
# Every prod group_vars key must be a variable the shared code already knows: a role default,
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
known: set[str] = set(ALL_VARS)
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
try:
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
except Exception:
pass
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
for k in sorted(PROD_VARS):
check(k.startswith("ansible_") or k in known,
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
"configuration that the dev path never exercises is developing against prod")
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
# ---------------------------------------- (5) prod is not a development surface (overlay)
if prod_overlay:
odir = ROOT / "clusters" / prod_overlay
files = sorted(p.name for p in odir.iterdir() if p.is_file())
check(files == ["kustomization.yaml", "sync.yaml"],
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
body = p.get("patch") or ""
kind = tgt.get("kind")
if kind is None: # inline patch: kind comes from the body
try:
doc = yaml.safe_load(body) or {}
except Exception:
doc = {}
kind = doc.get("kind")
name = (doc.get("metadata") or {}).get("name")
if kind == "Kustomization":
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
# anything else (path, interval, images, values) changes what prod RUNS.
keys = set((doc.get("spec") or {}).keys())
check(keys <= {"decryption"},
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
"override; changing anything else is developing against prod")
continue
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
"environment-scoped object — prod must not carry configuration of its own")
else:
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
"an environment-scoped object — prod must not carry configuration of its own")
# Same Git ref as the other environment(s): a prod-only branch is a development line
# that only prod runs, which is exactly what R-ENV-03 forbids.
def git_ref(overlay: str):
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
for p in (kk.get("patches") or []):
body = p.get("patch") or ""
try:
doc = yaml.safe_load(body) or {}
except Exception:
continue
if doc.get("kind") == "GitRepository":
return (doc.get("spec") or {}).get("ref")
return None
others = [o.name for o in (ROOT / "clusters").iterdir()
if o.is_dir() and o.name not in ("base", prod_overlay)]
prod_ref = git_ref(prod_overlay)
check(prod_ref is not None,
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
"points at, with no record of what it runs")
for o in others:
check(git_ref(o) == prod_ref,
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
# No prod-only manifest anywhere in the rendered configuration.
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True
).stdout.split("\0") if f]
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
for f in tracked:
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
f, re.I):
check(False,
f"{f} is a prod-specific file inside the shared configuration — every environment "
"renders these paths, so prod-only content here is development against prod")
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
# --------------------------- (6) prod work exists only with a declaration of readiness
# In-repo: the operator's recorded decision to open the prod phase.
log = ROOT / "DECISIONS-LOG.md"
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
spec = Path(os.environ.get("SPEC_REPO", ""))
phases = spec / "roadmap" / "phases.md"
if phases.is_file():
txt = phases.read_text(errors="replace")
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
re.search(r"prod", txt, re.I) is not None,
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
"to does not exist")
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
is not None,
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
"names would be unenforceable")
else:
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
"(the in-repo declaration still was)")
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
if shutil.which("ansible-playbook") is None:
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
else:
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
args = ["--list-hosts", "playbooks/site.yml",
"-e", f"k3s_hosts={PROD_GROUP}",
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
if sec_groups:
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
out = (lh.stderr or "") + (lh.stdout or "")
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
"driven; every parse assertion above still ran")
else:
check(lh.returncode == 0,
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
"the same Ansible bootstrap to run for prod")
seen: set[str] = set()
in_hosts = False
for line in lh.stdout.splitlines():
if re.match(r"\s*hosts \(\d+\):", line):
in_hosts = True
continue
if re.search(r"play #\d+", line):
in_hosts = False
continue
if in_hosts and line.strip():
seen.add(line.strip())
check(seen == set(prod_names),
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
# --------------------------------------------------------------------------- verdict
check(CHECKS >= 25,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
for n in NOTES:
print(f" note: {n}")
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
f"plus environment-scoped patches only, local Longhorn path "
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
"live cluster's shape is NOT asserted here (see the header's gap list).")
PY
conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml
+69
View File
@@ -0,0 +1,69 @@
# =============================================================================
# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and
# revoked without manual steps"); mechanism in docs/08 + ADR-0006.
#
# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the
# clause — "without manual steps" — which in a GitOps platform is a property of the
# REPOSITORY: a human step in the certificate path leaves evidence in git (committed
# cert material, a Gateway serving a Secret nothing declares, an issuer with no
# automated solver, a credential never delivered, auto-approval switched off, or CRs
# nothing reconciles). The three verbs are observed by the companion L2 test.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`,
# so the check resolves it against the BASE cluster-vars default AND every overlay's value and
# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from
# one authored assertion, with no hostname, address or environment name written down.
#
# GIVEN the repository as git tracks it,
# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release
# is inspected offline,
# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate
# produces; no certificate material is committed anywhere; every Certificate's
# issuer resolves — in EVERY environment — to a tracked ACME issuer with an
# automated dns01/http01 solver whose credential is SOPS-delivered from git;
# cert-manager's automatic request approval is not disabled; and every one of
# those objects is rendered by a Flux Kustomization, so nothing in the chain
# depends on a human running kubectl.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and
# this Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them):
# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated).
# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation,
# and the spec's own reading (docs/08: "removing/rotating a certificate is a
# declarative change") needs `enableCertificateOwnerRef`, which the tracked
# cert-manager HelmRelease does not set — so a removed Certificate leaves its
# serving Secret behind. The L2 test asserts the withdrawal behaviour and is
# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06).
# * That the live cluster is serving the tracked Certificate (cluster state).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: tls-lifecycle-automated
labels:
conformance.home.k3s/R-TLS-02: "true"
annotations:
conformance.home.k3s/requirements: "R-TLS-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the certificate path is inspected offline, then no
manual step exists anywhere in it: serving Secrets are cert-manager-produced, no
cert material is committed, every environment's issuer is a tracked ACME issuer
with an automated solver and a git-delivered credential, auto-approval is on, and
the whole chain is Flux-reconciled.
spec:
steps:
- name: then-no-manual-step-exists-in-the-certificate-path
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh
and conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/tls-lifecycle-automated/check.sh
Executable
+478
View File
@@ -0,0 +1,478 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — tls-lifecycle-automated
# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked**
# without manual steps." — spec docs/00-vision-and-scope.md; mechanism in
# docs/08-tls-and-certificates.md, ADR-0006.)
#
# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT
# ------------------------------------------------------
# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs
# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps*
# clause IS a repository property, and it is the half that can be proven with no
# cluster: if a human had to mint, install, or approve a serving certificate, the
# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway
# pointing at a Secret nothing declares, an issuer that no automated solver can
# satisfy, a per-cert credential handed over out of band, or an admission path that
# waits for a manual CertificateRequest approval. This check asserts that none of
# those exist and that the automated path is completely wired instead.
#
# The three VERBS are runtime events. They are asserted by the companion L2 test
# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an
# issuance, an unattended renewal, and the withdrawal of a removed certificate. This
# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below.
#
# GIVEN the repository as git tracks it,
# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release
# are inspected offline,
# THEN (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo
# with no Certificate cannot pass this check by having nothing to check);
# (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets
# that a tracked Certificate declares as its `secretName` — so the serving
# credential is produced by the controller, never dropped in by hand;
# (3) NO certificate material is tracked anywhere: no Secret manifest of type
# kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret
# (plain OR SOPS-encrypted), and no PEM certificate body in any tracked
# YAML — i.e. issuance is not, and cannot silently become, a human step;
# (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY
# ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the
# base cluster-vars default AND every overlay's value, so an environment
# whose issuer does not exist is caught here rather than at prod bring-up;
# (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list
# whose every solver is an automated challenge type (dns01 / http01) — the
# same code path serves issuance AND renewal, so a solver a human must
# complete would defeat both;
# (6) each dns01 solver's provider credential is referenced BY NAME and that
# Secret is delivered SOPS-encrypted from git via a kustomization — the
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
# (7) cert-manager's automatic CertificateRequest approval is NOT disabled
# (`disableAutoApproval`), and any `approveSignerNames` narrowing still
# covers the ClusterIssuers actually used — a request that waits for a
# human approver is the definition of a manual step;
# (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the
# ClusterIssuers and the Certificate are each included in a kustomization
# that a Flux Kustomization renders — so a deleted Certificate is
# re-created and re-issued with no human action;
# (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration`
# would mean the renewal instant never arrives).
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of
# l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path,
# evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert.
# - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate
# is explicit that this must be "observed across a renewal" — L2 step 3 does that
# by fast-forwarding a certificate's renewal window; nothing static can.
# - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct
# readings, neither established: (a) ACME revocation — cert-manager does not
# revoke certificates at all, so no repo state could show it; (b) the spec's own
# reading in docs/08 ("removing/rotating a certificate is a declarative change")
# — withdrawal of a REMOVED certificate's serving material requires cert-manager's
# `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart
# default false). L2 step 4 asserts the withdrawal behaviour and is expected to
# FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06).
# - That the tracked Certificate is the one the live Gateway is actually serving
# (cluster state), or that the ACME account/DNS credential is still valid.
# - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see
# l3-fault-injection/alerting).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import re
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def tracked() -> list[str]:
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout
return [p for p in out.split("\0") if p]
FILES = tracked()
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)]
def docs_of(rel: str):
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)]
def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]:
out = []
for rel, d in DOCS:
if d.get("kind") != kind:
continue
if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix):
continue
out.append((rel, d))
return out
def ns_of(d: dict, default: str = "default") -> str:
return (d.get("metadata") or {}).get("namespace") or default
def name_of(d: dict) -> str:
return (d.get("metadata") or {}).get("name") or "<unnamed>"
# ------------------------------------------------------------ cluster-vars resolution
# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902
# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This
# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an
# issuer name can be resolved per environment instead of being asserted for dev only.
def env_values(key: str) -> dict[str, str]:
vals: dict[str, str] = {}
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
if base.is_file():
for d in yaml.safe_load_all(base.read_text()):
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
and name_of(d) == "cluster-vars":
v = (d.get("data") or {}).get(key)
if v is not None:
vals["base"] = str(v)
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
env = overlay.parent.name
if env == "base":
continue
try:
k = yaml.safe_load(overlay.read_text()) or {}
except Exception:
continue
for p in (k.get("patches") or []):
tgt = p.get("target") or {}
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
continue
try:
ops = yaml.safe_load(p.get("patch") or "") or []
except Exception:
continue
for op in ops if isinstance(ops, list) else []:
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
vals[env] = str(op.get("value"))
return vals
TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
def resolve_per_env(value: str) -> dict[str, str]:
"""A literal resolves to itself in every environment; a ${TOKEN} to each env's value."""
m = TOKEN_RE.match(value.strip())
if not m:
return {"(literal)": value.strip()}
return env_values(m.group(1))
# --------------------------------------------- what Flux actually renders (reachability)
# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through
# the kustomize resource graph: <path>/kustomization.yaml -> resources -> (file | directory
# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by
# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere":
# the loose form silently passes when an entry is deleted from the very kustomization.yaml
# that the directory reference renders (verified — it made two assertions here no-ops).
KUST_RESOURCES: dict[str, list[str]] = {}
for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]:
refs: list[str] = []
for d in docs_of(rel):
for r in (d.get("resources") or []) + (d.get("components") or []):
if not isinstance(r, str) or "://" in r: # skip remote bases
continue
try:
p = (ROOT / rel).parent.joinpath(r).resolve()
refs.append(str(p.relative_to(ROOT)))
except Exception:
continue
KUST_RESOURCES[rel] = refs
FLUX_PATHS: set[str] = set()
for _rel, _d in DOCS:
if _d.get("kind") == "Kustomization" and \
str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"):
p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/")
if p:
FLUX_PATHS.add(p)
def _walk(entry: str, seen: set[str], out: set[str]) -> None:
if entry in seen:
return
seen.add(entry)
kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml"
if kust in KUST_RESOURCES: # a directory (or a kustomization)
for r in KUST_RESOURCES[kust]:
_walk(r, seen, out)
return
out.add(entry) # a leaf manifest
RENDERED: set[str] = set()
for _p in FLUX_PATHS:
_walk(_p, set(), RENDERED)
def delivered(rel: str) -> bool:
"""True iff Flux renders this file from one of its Kustomization paths."""
return rel in RENDERED
# --------------------------------------------------- (1) there is something to check
CERTS = of_kind("Certificate", "cert-manager.io/")
check(len(CERTS) >= 1,
"no cert-manager.io Certificate is authored anywhere in the tracked tree — every "
"assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)")
ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/")
check(len(ISSUERS) >= 1,
"no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically")
# ---------------------------- (2) every terminating listener serves a controller cert
CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) -> file
(ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel
for rel, d in CERTS
}
listeners_checked = 0
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
gw_ns = ns_of(gw)
for lis in ((gw.get("spec") or {}).get("listeners") or []):
tls = lis.get("tls") or {}
if str(tls.get("mode", "")) != "Terminate":
continue
listeners_checked += 1
refs = tls.get("certificateRefs") or []
check(len(refs) > 0,
f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates "
"TLS with NO certificateRefs — it can only serve a certificate someone puts "
"there by hand")
for r in refs:
kind = str(r.get("kind", "Secret"))
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
check(kind == "Secret",
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
"Secret refs can be traced to a cert-manager Certificate")
check(key in CERT_SECRETS,
f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which "
"NO tracked cert-manager Certificate declares as its secretName — that "
"credential would have to be created by hand (R-TLS-02 'without manual steps')")
check(listeners_checked > 0,
"no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no "
"serving-TLS reference to trace, so (2) proved nothing")
# --------------------------------- (3) no certificate material is tracked, anywhere
TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"}
for rel, d in DOCS:
if d.get("kind") != "Secret":
continue
stype = str(d.get("type", ""))
check(stype != "kubernetes.io/tls",
f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name="
f"{name_of(d)}) — a serving certificate authored by a human, not issued")
for field in ("data", "stringData"):
for k in (d.get(field) or {}):
check(str(k) not in TLS_KEYS,
f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate "
"material in git means a manual issuance step exists (R-TLS-02)")
PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----")
for rel in YAML_FILES:
body = (ROOT / rel).read_text(errors="replace")
# Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header
# in an example, and a .md file is not applied to any cluster. A manifest cannot.
check(not PEM_RE.search(body),
f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest")
# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env
# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace`
# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the
# HelmRelease, never written down here, so this holds if the release namespace ever changes.
cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"]
CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager"
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
issuers_verified: set[str] = set()
for rel, cert in CERTS:
spec = cert.get("spec") or {}
ref = spec.get("issuerRef") or {}
kind = str(ref.get("kind", "Issuer"))
per_env = resolve_per_env(str(ref.get("name", "")))
check(len(per_env) > 0,
f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in "
"clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue")
for env, iname in sorted(per_env.items()):
hit = ISSUER_INDEX.get((kind, iname))
check(hit is not None,
f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in "
f"environment {env!r}, but no such issuer is tracked — that environment cannot "
"issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)")
if hit is None:
continue
irel, issuer = hit
issuers_verified.add(iname)
ispec = issuer.get("spec") or {}
acme = ispec.get("acme")
check(isinstance(acme, dict),
f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME "
f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate "
"ACME so both are unattended")
if not isinstance(acme, dict):
continue
check(str(acme.get("server", "")).startswith("https://"),
f"{irel}: {kind}/{iname} has no https ACME directory endpoint")
check(bool((acme.get("privateKeySecretRef") or {}).get("name")),
f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key "
"would have to be provisioned by hand")
solvers = acme.get("solvers") or []
check(len(solvers) > 0,
f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved "
"until a human completed the challenge")
for s in solvers:
types = [t for t in ("dns01", "http01") if t in s]
check(len(types) == 1,
f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only "
"the automated dns01/http01 solvers satisfy 'without manual steps'")
for provider, cfg in (s.get("dns01") or {}).items():
if not isinstance(cfg, dict):
continue
for field, val in cfg.items():
if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")):
n = str(val.get("name", ""))
check(bool(n),
f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name")
if n:
# A ClusterIssuer resolves secret refs in the controller's
# cluster-resource-namespace; a namespaced Issuer in its own.
want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS)
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
check(len(solver_creds) > 0,
"no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped "
"Cloudflare token to drive DNS-01, so this check would otherwise be vacuous")
# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential.
# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy
# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.)
SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d)
for rel, d in DOCS if d.get("kind") == "Secret"}
for iname, sns, sname, field in solver_creds:
hit = SECRET_DOCS.get((sns, sname))
check(hit is not None,
f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest "
f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the "
"credential would be created by hand in-cluster (R-TLS-02)")
if hit is None:
continue
srel, sdoc = hit
check("sops" in sdoc,
f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)")
check(delivered(srel),
f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would "
"never deliver it, so issuance would wait on a manual kubectl apply")
# ------------------------------- (7) automatic CertificateRequest approval is intact
check(len(cm_releases) == 1,
f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — "
"the controller that issues and renews must itself be GitOps-managed")
for rel, hr in cm_releases:
values = ((hr.get("spec") or {}).get("values") or {})
check(values.get("disableAutoApproval") in (None, False),
f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} "
"— every CertificateRequest would then wait for an approver, which is a manual step")
signers = values.get("approveSignerNames")
if signers is not None:
pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])]
for iname in sorted(issuers_verified):
ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and
(p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats)
check(ok,
f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — "
"requests for it would never be auto-approved")
# The chart's own CRD install must be on, else the CRs below cannot even be applied.
check((values.get("crds") or {}).get("enabled") is True,
f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD "
"would have to be installed by hand")
# ------------------------------- (8) the whole chain is continuously reconciled from git
check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20,
f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} "
"Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing")
for rel, _ in cm_releases + CERTS + ISSUERS:
check(delivered(rel),
f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a "
"deleted Certificate/issuer/controller would stay deleted until a human re-applied it")
# --------------------------------------------- (9) renewal is not defeated arithmetically
def to_seconds(v) -> int | None:
m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v))
if not m:
return None
return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)]
renew_pairs = 0
for rel, cert in CERTS:
spec = cert.get("spec") or {}
dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore"))
if dur is None or rb is None:
continue
renew_pairs += 1
check(rb < dur,
f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= "
f"duration={spec.get('duration')} — the renewal instant never arrives")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op.
# The floor below is a count of assertions that MUST have run for the verdict to mean
# anything; it is not a style rule.
check(CHECKS >= 20,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), "
f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate "
f"across every environment, {listeners_checked} terminating listener(s), "
f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). "
"R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs "
"(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).")
PY
conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml
+391
View File
@@ -0,0 +1,391 @@
# =============================================================================
# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02)
# [RUNNABLE / NOT YET EXECUTED — see STATUS]
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed,
# and revoked** without manual steps."
# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well
# before expiry automatically; no human action." / "Revocation: removing/
# rotating a certificate is a declarative change; cert-manager handles
# issuance of replacements."
# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry
# with no manual step (**observed across a renewal**)."
# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml.
# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps"
# clause statically (in the merge gate). This test observes the three VERBS.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The
# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow
# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name`
# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no
# address: dev proves dev's cert and prod proves prod's from the same authored assertions.
#
# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS,
# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW
# Certificate is then requested, left to renew itself, and removed,
# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its
# Secret carries cert-manager's certificate-name annotation, its
# CertificateRequest was submitted by cert-manager's own ServiceAccount and
# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer;
# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and
# strictly before status.notAfter, so no human is in the renewal path;
# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears
# with certificate material, with no step between declaring and having it;
# 4. RENEWED — that certificate, given a renewal window that opens one minute
# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and
# the serving bytes actually change, with no human action. This is docs/08's
# "observed across a renewal", fast-forwarded;
# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material.
#
# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS
# -------------------------------------------------------------------
# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every
# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order
# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate
# limits for evidence that is not about ACME: cert-manager's renewal scheduler and
# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created
# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2
# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls
# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03).
# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the
# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while
# steps 1-2 prove the real serving certificate is on exactly that automated path.
#
# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an
# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's
# per-test namespace — the same class of mutation every other L2 test here performs,
# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored
# them against live dev/prod read-only inspection under an explicit instruction not to
# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the
# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true.
#
# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06)
# ------------------------------------------------------
# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects
# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`;
# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the
# live wildcard Secret on both clusters has NO ownerReferences (verified read-only,
# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked
# without manual steps" does not hold. That is a finding about the platform, not a
# defect in this test: the assertion stays as the requirement reads and is escalated to
# the PM. (ACME revocation proper is not implemented by cert-manager at all.)
#
# KNOWN, DELIBERATE GAPS (this test claims none of these):
# * Not proven: that the CA would refuse a revoked certificate — there is no
# revocation request to make, so nothing observable exists to assert.
# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a
# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the
# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the
# mechanism that will act on it.
# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04
# (l3-fault-injection/alerting).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: cert-lifecycle-automated
labels:
conformance.home.k3s/R-TLS-02: "true"
annotations:
conformance.home.k3s/requirements: "R-TLS-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given cert-manager and a TLS-terminating Gateway, when the serving certificate's
provenance and renewal schedule are read and a new certificate is requested, left
to renew, and removed, then issuance, renewal and withdrawal all happen with no
human step — the serving cert was controller-issued and auto-approved, its
renewal is scheduled ahead of expiry, and a declared certificate issues and
reissues itself unattended.
spec:
timeouts:
apply: 60s
assert: 3m
exec: 10m
steps:
# ---------------------------------------------------------------- READ-ONLY HALF
- name: given-cert-manager-is-reconciled-and-healthy
description: >-
The controller that issues and renews must be running, and must have been
installed by Flux rather than by hand — otherwise "automatic" has no engine.
try:
- script:
content: |
set -u
# Discover cert-manager's Deployments by chart label, in whatever namespace
# the release lives (nothing about the environment is assumed).
sel='app.kubernetes.io/instance=cert-manager'
rows="$(kubectl get deploy -A -l "$sel" \
-o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \
--no-headers 2>/dev/null || true)"
echo "$rows"
[ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; }
n=0
echo "$rows" | while read -r ns name rdy; do
[ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; }
done || exit 1
n="$(echo "$rows" | wc -l)"
# controller + webhook + cainjector: without the webhook a Certificate is
# rejected, without cainjector the webhook has no CA — all three or nothing.
[ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; }
# GitOps-managed, not hand-installed: Flux labels what it applies.
ns="$(echo "$rows" | head -1 | awk '{print $1}')"
lbl="$(kubectl get deploy -n "$ns" -l "$sel" \
-o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)"
[ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; }
echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)"
check:
(contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true
- name: then-the-serving-certificate-was-issued-by-the-controller
description: >-
Follow the Gateway's terminating listener to the Secret it serves, and prove
that Secret's provenance is cert-manager — annotation, CertificateRequest
requester, automatic approval, and a Ready ACME issuer.
try:
- script:
content: |
set -u
# --- discover the serving Secret from the Gateway itself -------------------
gw="$(kubectl get gateway -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
2>/dev/null | head -1)"
[ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; }
gns="${gw% *}"; gname="${gw#* }"
sec="$(kubectl -n "$gns" get gateway "$gname" \
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \
2>/dev/null | awk '{print $1}')"
[ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; }
echo "gateway=$gns/$gname serving-secret=$sec"
# --- provenance (1): only the controller writes this annotation ------------
cert="$(kubectl -n "$gns" get secret "$sec" \
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)"
[ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; }
crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
[ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; }
# --- provenance (2): the Certificate is Ready and has a revision -----------
ready="$(kubectl -n "$gns" get certificate "$cert" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)"
echo "certificate=$gns/$cert ready=$ready revision=$rev"
[ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; }
[ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; }
# --- provenance (3): the request was made BY THE CONTROLLER and auto-approved
# spec.username is the identity that created the CertificateRequest. A human
# operator would appear as a user/OIDC identity, not a ServiceAccount.
cr="$(kubectl -n "$gns" get certificaterequest \
-o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \
2>/dev/null | tail -1)"
[ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; }
crname="${cr% *}"; requester="${cr#* }"
echo "certificaterequest=$crname requester=$requester"
case "$requester" in
system:serviceaccount:*) : ;;
*) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;;
esac
appr="$(kubectl -n "$gns" get certificaterequest "$crname" \
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
[ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; }
# --- provenance (4): an automated ACME issuer, and it is Ready -------------
ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')"
iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')"
echo "issuer=$ikind/$iname"
if [ "$ikind" = "ClusterIssuer" ]; then
iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
else
iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
fi
[ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; }
[ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; }
echo "SERVING_CERT_CONTROLLER_ISSUED_OK"
check:
(contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true
- name: then-an-unattended-renewal-is-already-scheduled
description: >-
cert-manager writes status.renewalTime only after computing when it will renew.
Asserting it is in the future and before expiry proves the renewal is scheduled
and that nothing is waiting on a human (docs/08 "renews well before expiry").
try:
- script:
content: |
set -u
gw="$(kubectl get gateway -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)"
gns="${gw% *}"; gname="${gw#* }"
sec="$(kubectl -n "$gns" get gateway "$gname" \
-o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')"
cert="$(kubectl -n "$gns" get secret "$sec" \
-o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')"
renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')"
after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')"
[ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; }
[ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; }
now_s="$(date -u +%s)"
r_s="$(date -u -d "$renew" +%s)"
a_s="$(date -u -d "$after" +%s)"
echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after"
echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d"
[ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; }
[ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; }
echo "RENEWAL_SCHEDULED_OK"
check:
(contains($stdout, 'RENEWAL_SCHEDULED_OK')): true
# ------------------------------------------------- MUTATING HALF (hermetic namespace)
- name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step
description: >-
Declare an Issuer and a Certificate in this test's own namespace, with a renewal
window that opens one minute after issuance. Nothing else is done to them for the
rest of the test — every later transition is the controller acting on its own.
try:
- apply:
resource:
# selfSigned: no external CA, no rate limit, no DNS write. See the header for
# what this proves (controller automation) and what it does not (the ACME path).
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: conformance-lifecycle
spec:
selfSigned: {}
- apply:
resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: conformance-lifecycle
spec:
secretName: conformance-lifecycle-tls
# 1h is cert-manager's minimum duration; renewing 59m before expiry puts the
# renewal instant ~1 minute after each issuance, so an unattended renewal is
# observable inside a test run instead of 30 days from now.
duration: 1h
renewBefore: 3540s
commonName: conformance-lifecycle.invalid
dnsNames:
- conformance-lifecycle.invalid
privateKey:
rotationPolicy: Always
issuerRef:
kind: Issuer
name: conformance-lifecycle
- assert:
resource:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: conformance-lifecycle
status:
conditions:
- type: Ready
status: "True"
revision: 1
- script:
content: |
set -u
crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
-o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
-o jsonpath='{.data.tls\.key}' 2>/dev/null || true)"
[ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; }
# Record the serving bytes so the renewal step can prove they CHANGED.
echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum
echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)"
echo "CERT_ISSUED_OK"
check:
(contains($stdout, 'CERT_ISSUED_OK')): true
- name: then-it-renews-itself-before-expiry-with-no-human-step
description: >-
Nothing touches the Certificate here. Poll until the controller has reissued it
at least twice on its own and the serving bytes differ from revision 1 — docs/08's
"observed across a renewal", and every request auto-approved.
try:
- script:
content: |
set -u
want=3 # revision 1 = issuance; >=3 means TWO unattended renewals
for i in $(seq 1 30); do
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
echo "poll $i: revision=${rev:-0}"
[ "${rev:-0}" -ge "$want" ] && break
sleep 20
done
rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
-o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
[ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; }
# The serving material must actually have rotated, not just the revision count.
now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
-o jsonpath='{.data.tls\.crt}' | cksum)"
was="$(cat /tmp/conformance-cert-r1.cksum)"
echo "r1=$was now=$now"
[ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; }
# Every renewal request was approved by the controller, never by a person.
bad=0
for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do
st="$(kubectl -n "$NAMESPACE" get "$cr" \
-o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)"
echo " $cr approved=$st requester=$who"
[ "$st" = "True" ] || bad=1
case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac
done
[ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; }
echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)"
check:
(contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true
- name: then-removing-the-certificate-withdraws-the-serving-material
description: >-
docs/08 reads revocation as "removing/rotating a certificate is a declarative
change". The declarative change is the deletion; withdrawal of the credential
must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs
`enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the
tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06).
try:
- delete:
ref:
apiVersion: cert-manager.io/v1
kind: Certificate
name: conformance-lifecycle
- script:
content: |
set -u
for i in $(seq 1 15); do
if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then
echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)"
exit 0
fi
sleep 4
done
owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
-o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)"
echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still"
echo " holds usable certificate material (ownerReferences=${owners:-none})."
echo " Removing a certificate therefore does NOT withdraw the credential, so"
echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:"
echo " set enableCertificateOwnerRef: true in platform/cert-manager values"
echo " (owner decision — WS-TEST does not weaken the assertion instead)."
exit 1
check:
(contains($stdout, 'CERT_WITHDRAWN_OK')): true
finally:
# Namespace teardown removes both objects anyway; explicit so a leftover Secret
# from the expected failure above is never left holding key material.
- script:
content: |
kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true
rm -f /tmp/conformance-cert-r1.cksum || true
conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml
+17
View File
@@ -153,3 +153,20 @@ spec:
finally:
- script:
content: 'echo "cleanup handled by chainsaw namespace teardown"'
# -----------------------------------------------------------------------------------
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
# chainsaw already fails the test before reaching this step if any of them fails — but a
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
# forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
# is reached only when everything above has passed. Asserted by
# tests/l0-static/auditability-proofs-producible.
- name: then-emit-the-proof-of-pass-sentinel
try:
- script:
content: echo DEFAULT_DENY_PROVEN_OK
check:
(contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true
conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml
+50 -8
View File
@@ -3,11 +3,18 @@
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
# Spec: docs/09 (ExternalDNS auto-manages records; split-horizon foundation)
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER
# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the
# expected answer is the LB IP that THIS Service was actually assigned (read back from its own
# status) — so neither the name nor the address is authored per environment, and no run-time flag
# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two
# environments from ever describing the same public name (the 2026-09-04 incident precondition).
#
# GIVEN a LoadBalancer Service annotated with an external-dns hostname,
# WHEN ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync),
# THEN a PUBLIC DNS record for that hostname is auto-created, pointing at the
# Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the
# cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint.
# Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that
# Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint.
#
# CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown;
# ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass.
@@ -38,6 +45,13 @@ spec:
apply: 90s
assert: 5m
exec: 6m
# -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST.
# flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this
# binding is the same source of truth the platform is rendered from — dev asks dev, prod
# asks prod, and there is no per-environment fork and no flag to forget.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: given-a-service-requesting-a-dns-name
try:
@@ -48,7 +62,8 @@ spec:
metadata:
name: dns-probe-svc
annotations:
external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
spec:
type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -59,7 +74,27 @@ spec:
- name: http
port: 80
targetPort: 80
# Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will
# publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal
# keeps the expression nil-safe while the address is still absent, so the assert POLLS
# (fails and retries) instead of erroring out on the first evaluation.
- assert:
resource:
apiVersion: v1
kind: Service
metadata:
name: dns-probe-svc
status:
(length(loadBalancer.ingress || `[]`)): 1
- name: then-externaldns-creates-the-public-record
# Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's
# assert guarantees it exists by now — step bindings are evaluated when the step runs).
# Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60
# range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is
# 192.168.2.20-.30, and neither is written down here.
bindings:
- name: lbIp
value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip)
try:
- apply:
resource:
@@ -76,20 +111,27 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
# Both values come from the cluster under test: the suffix from
# flux-system/cluster-vars, the address from this Service's own status.
env:
- name: BASE_DOMAIN
value: ($baseDomain)
- name: LB_IP
value: ($lbIp)
command:
- sh
- -c
- |
set -u
host="conformance-dns.home.keithjackson.dev"
host="conformance-dns.${BASE_DOMAIN}"
# Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and
# points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it.
# points at THIS Service's assigned LB IP. Proves ExternalDNS created it.
for i in $(seq 1 28); do
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)"
ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)"
if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi
echo "poll $i: record not yet published"; sleep 10
echo "poll $i: record not yet published as ${LB_IP}"; sleep 10
done
echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1
echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1
- assert:
resource:
apiVersion: batch/v1
conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml
+40 -14
View File
@@ -2,17 +2,24 @@
# L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE]
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
# Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
# <svc>.home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin.
# <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
# Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
# answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}).
# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
# per-environment and no flag is passed at run time, so the suite cannot be run against the
# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
#
# GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
# WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name
# (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10),
# THEN the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a
# public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the
# Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates
# (R-DNS-01 "services reachable at <svc>.home..."; R-DNS-03 split-horizon).
# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
# IP. This is what lets pod->public-hostname traffic hairpin through the
# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
# (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
#
# Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
# split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
@@ -34,13 +41,23 @@ metadata:
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal
Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution.
An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
spec:
timeouts:
apply: 60s
assert: 2m
exec: 2m
# -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
# here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
# (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
# same source of truth the platform is rendered from. Test-level => in scope for every step
# (unlike step `outputs`, which are step-scoped).
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
- name: gatewayVip
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
steps:
- name: then-cluster-dns-answers-with-the-internal-gateway-vip
try:
@@ -59,23 +76,32 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
# The cluster's OWN identity, resolved from flux-system/cluster-vars by the
# test-level bindings — never a literal (R-ENV-02).
env:
- name: BASE_DOMAIN
value: ($baseDomain)
- name: GATEWAY_VIP
value: ($gatewayVip)
command:
- sh
- -c
- |
set -u
host="keycloak.home.keithjackson.dev"
host="keycloak.${BASE_DOMAIN}"
# Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
# 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP
# (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway.
# 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
# (the address coredns-custom is templated with), proving pods hairpin
# through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
# old hardcoded 192.168.1.53-.60 range AND correct in both environments.
out="$(nslookup "$host" 2>/dev/null)"
echo "$out"
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)"
ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
if [ -n "$ip" ]; then
echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
exit 0
fi
echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);"
echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
exit 1
- assert:
conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml
+199
View File
@@ -0,0 +1,199 @@
# =============================================================================
# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE]
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
# routing rules ... realized as two paths — the local `internet` path and the
# in-cluster `vpn` path").
# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is
# simply the default route out".
# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.**
# Every pod — and the nodes themselves — egress the local/private WAN by default and
# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the
# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0`
# uses the VPN."
#
# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT
# --------------------------------------------------------------
# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their
# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing
# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds
# that pod, and asserts the two claims that make a rule-selected path set complete:
#
# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the
# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is
# the local one. If the steering rule ever widened (an empty selector, a
# matchExpression, a second rule), this pod's IP would move to the tunnel and
# this assertion is what would notice.
# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one
# pass: three pods, two distinct public IPs. "Multiple egress paths" is a
# simultaneity claim, not a claim that one switch has two positions.
# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918
# destination over the LOCAL path while its public traffic is tunnelled. The
# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod
# uses BOTH paths at once.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a
# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike
# and survives the WAN or the VPN provider rotating addresses.
#
# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three
# times, differing only in the class they declare (or do not),
# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod
# also resolves a name through the in-cluster resolver,
# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out
# is the WAN, and the VPN is strictly opt-in);
# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path);
# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not
# swallowed by the tunnel), so the same pod is using both paths at once.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one
# steering rule is authored, that it selects only the vpn class, that no `ip rule` is
# unscoped and no nftables DROP unqualified is the L0 half, in
# tests/l0-static/egress-paths-selected-by-rules (same requirement label).
# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo
# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need
# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in
# this suite does.
# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn`
# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically
# by the L0 half, not here.
# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That
# is a host-level fact this test cannot see from a pod.
# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster,
# so a WAN outage or the echo being down fails this test. That is the same dependency
# tests/l2-behavioral/egress-class already carries; the alternative — trusting the
# cluster's own account of which path it used — would not be a behavioural test at all.
#
# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside
# that namespace is created or changed — no label on a platform object, no policy edit.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: egress-default-path
labels:
conformance.home.k3s/R-NET-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-04"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public
IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the
default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two
paths live at once — and the vpn-class pod still reaches in-cluster destinations over the
local path, so one pod uses both paths simultaneously.
spec:
timeouts:
apply: 60s
assert: 2m
exec: 3m
steps:
# -----------------------------------------------------------------------------------
- name: given-three-pods-differing-only-in-their-declared-egress-class
try:
- apply:
file: fixtures/three-egress-pods.yaml
# Bare pods (restartPolicy: Never) — Ready == Running here.
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: egress-unlabelled }
status: { phase: Running }
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: egress-internet }
status: { phase: Running }
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: egress-vpn }
status: { phase: Running }
# -----------------------------------------------------------------------------------
- name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too
try:
- script:
content: |
set -eu
ns="$NAMESPACE"
get_ip() {
# Echo the pod's observed public IP, or empty on failure.
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
# Poll until all three answer AND the vpn class has diverged from the WAN:
# Cilium may still be programming the egress datapath just after the pods go
# Ready. Bounded by the loop, never a bare sleep-and-hope.
none="" inet="" vpn=""
for i in $(seq 1 12); do
[ -n "$none" ] || none="$(get_ip egress-unlabelled)"
[ -n "$inet" ] || inet="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then
[ "$vpn" != "$inet" ] && break
# Steering may not have converged; re-read the vpn pod only.
vpn=""
fi
sleep 5
done
echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn"
[ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; }
[ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; }
# TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN —
# the same WAN the explicitly-internet pod uses.
if [ "$none" != "$inet" ]; then
echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2
exit 1
fi
# COEXISTENCE: and the vpn path is a genuinely different path, live right now.
if [ "$vpn" = "$inet" ]; then
echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2
exit 1
fi
echo "EGRESS_DEFAULT_PATH_IS_WAN_OK"
check:
(contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true
# -----------------------------------------------------------------------------------
- name: then-the-vpn-class-pod-still-reaches-private-destinations-locally
try:
- script:
# PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all
# RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the
# local datapath. The cluster resolver is an in-cluster (RFC1918) address reached
# from the pod's own /etc/resolv.conf — no literal is written here — so a
# successful lookup from the vpn-class pod proves the tunnel did not swallow its
# local traffic, i.e. ONE pod is using BOTH paths at the same time.
content: |
set -eu
ns="$NAMESPACE"
resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)"
[ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; }
echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)"
# A cluster-internal name, resolvable only by the in-cluster resolver: reaching
# it proves the private-destination path works from a vpn-class pod.
out=""
for i in $(seq 1 6); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)"
case "$out" in *"Address"*) break ;; esac
sleep 5
done
case "$out" in
*Address*) : ;;
*) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;;
esac
echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK"
check:
(contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true
conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml
+67
View File
@@ -0,0 +1,67 @@
# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing
# ONLY in how (or whether) they declare an egress class:
#
# egress-unlabelled NO egress-class label at all -> selected by NO routing rule
# egress-internet egress-class: internet -> the declared default path
# egress-vpn egress-class: vpn -> the only class any rule steers
#
# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress
# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no
# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must
# come out the same WAN as the explicitly-`internet` pod. That is the totality half of
# R-NET-04's "selected by routing rules" — absence of a rule is itself a path.
#
# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an
# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in
# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee
# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe
# the datapath's own default at all.
apiVersion: v1
kind: Pod
metadata:
name: egress-unlabelled
labels:
app: egress-default-path-test
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
resources:
requests: { cpu: 10m, memory: 16Mi }
limits: { memory: 32Mi }
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: egress-internet
labels:
app: egress-default-path-test
homelab.keithjackson.dev/egress-class: internet
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
resources:
requests: { cpu: 10m, memory: 16Mi }
limits: { memory: 32Mi }
restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
name: egress-vpn
labels:
app: egress-default-path-test
homelab.keithjackson.dev/egress-class: vpn
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
resources:
requests: { cpu: 10m, memory: 16Mi }
limits: { memory: 32Mi }
restartPolicy: Never
conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml
+32 -2
View File
@@ -68,6 +68,15 @@ spec:
timeouts:
apply: 60s
assert: 180s
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
# authored here. Test-level => in scope for the fixture apply AND every probe step below
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
@@ -139,12 +148,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-declared-conformance.home.keithjackson.dev"
host="gw-declared-conformance.${BASE_DOMAIN}"
last=000
i=0
while [ "$i" -lt 30 ]; do
@@ -207,12 +218,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-undeclared-conformance.home.keithjackson.dev"
host="gw-undeclared-conformance.${BASE_DOMAIN}"
reached_gw=no
i=0
while [ "$i" -lt 8 ]; do
@@ -300,3 +313,20 @@ spec:
apiVersion: v1
kind: Namespace
name: conformance-gw-probe
# -----------------------------------------------------------------------------------
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
# chainsaw already fails the test before reaching this step if any of them fails — but a
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
# forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
# is reached only when everything above has passed. Asserted by
# tests/l0-static/auditability-proofs-producible.
- name: then-emit-the-proof-of-pass-sentinel
try:
- script:
content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
check:
(contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true
conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml
+11 -2
View File
@@ -10,6 +10,13 @@
# The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
# rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
# probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
#
# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
---
apiVersion: v1
kind: Namespace
@@ -134,7 +141,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-declared-conformance.home.keithjackson.dev
# gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-declared-conformance', $baseDomain]))
rules:
- matches:
- path:
@@ -158,7 +166,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-undeclared-conformance.home.keithjackson.dev
# gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-undeclared-conformance', $baseDomain]))
rules:
- matches:
- path:
conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml
+379
View File
@@ -0,0 +1,379 @@
# =============================================================================
# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane (R-PLAT-01) [RUNNABLE]
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
# exercise the same multi-node topology.")
# docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02"
# docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA
# embedded etcd (k3s server on all 3)"
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
# docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across
# all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload
# (3-node converged cluster)".
#
# WHY L2 — AND WHAT L0 ALREADY DID
# --------------------------------
# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED
# half offline: the repo builds a 3-server embedded-etcd control plane in both environments,
# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving
# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory
# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test:
# every assertion below is read from the cluster the suite is pointed at.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written
# here. The expected node count comes from the cluster's OWN declared identity
# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same
# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip
# DaemonSet the cluster is actually running; the server addresses come from the Node
# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars
# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY
# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the
# environment from the cluster, never from a flag or a literal, which is precisely the
# mechanism the tests/README.md convention prescribes.
#
# GIVEN the cluster this suite is pointed at,
# WHEN its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected,
# THEN DISTRO: every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s;
# SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node
# topology docs/02 fixes — no agent-only node, no control-plane-only node),
# the count is ODD and quorum-capable (tolerates >= 1 loss), and it is
# EXACTLY three when the cluster declares itself prod;
# ETCD: every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps
# `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so
# the datastore is embedded and replicated across all of them;
# VIP: kube-vip is Ready on EVERY control-plane node, is control-plane-only
# (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and
# the address it floats is neither a node address, nor the ingress Gateway
# VIP, nor inside the Cilium LoadBalancer pool;
# ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that
# validates against the cluster CA for the VIP itself — i.e. the VIP really
# fronts the API and the serving cert really carries it;
# EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own
# address, again with a CA-valid cert for that address. This is the assertion
# that separates a real HA control plane from one server plus two joiners:
# if only the founder ran an apiserver, the other addresses would refuse.
#
# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node
# addresses from the RUNNER. That works from a host on the node LAN (verified against prod)
# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a
# forwarded port — i.e. it would have made the test a property of where it was launched
# from. A Job on the cluster is on the node network in both environments, so the probe is
# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than
# as a fixture file because its address list is DISCOVERED from the cluster at run time —
# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
# - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP
# migrates, and that the API stays reachable when a server dies is an L3 fault-injection
# claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss
# drains a node for R-STATE-02's data-availability claim and asserts nothing about the
# control plane). This is the largest remaining R-PLAT-01 gap.
# - THE NEXT environment. That the repo would build an HA control plane again is the L0
# half (tests/l0-static/ha-control-plane-as-code).
# - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of
# infra/PROD-PROVISIONING.md).
# - THAT THE RUNNING cilium-agents ACTUALLY DIAL THIS VIP. As of 2026-09-06 the repository
# authors `K3S_API_HOST` = the control-plane VIP in both environments (asserted at L0 by
# tests/l0-static/ha-control-plane-as-code, on both sides of the [parity] contract), but
# this test does not read the agents' live `KUBERNETES_SERVICE_HOST`. Until the operator
# applies that change the LIVE agents still carry the PRIMARY NODE's address — measured
# 2026-09-06: all three prod agents on 192.168.2.10 while kube-vip held 192.168.2.9. An
# in-cluster assertion on the agents' endpoint belongs here and is NOT yet written; it is
# the obvious follow-up once the change is applied and would catch drift the L0 check
# cannot see (a hand-edited DaemonSet, a Helm value overridden out of band).
# - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes
# it only on the node-local etcd metrics port, which is not reachable through the
# Kubernetes API this test is allowed to use.
#
# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it
# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test
# creates nothing outside its own namespace and mutates no platform object.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: ha-control-plane
labels:
conformance.home.k3s/R-PLAT-01: "true"
annotations:
conformance.home.k3s/requirements: "R-PLAT-01"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver
endpoints are probed, then every node is a k3s server that is both control-plane and a
registered embedded-etcd member, the count is odd and quorum-capable (exactly three
when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that
collides with no allocator, and the apiserver answers /readyz through that VIP AND on
every server's own address with a CA-valid cert.
spec:
timeouts:
apply: 60s
assert: 2m
exec: 3m
# -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE
# CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes
# from, so these are the same values the platform was rendered with. Test-level, so
# they are in scope for every step (step outputs/bindings are step-scoped).
bindings:
- name: clusterEnv
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV)
- name: gatewayVip
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
- name: lbPoolStart
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START)
- name: lbPoolStop
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP)
steps:
# -----------------------------------------------------------------------------------
- name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane
description: >-
Read the Node objects: k3s on every node, every node both control-plane and a
registered embedded-etcd member, an odd quorum-capable count, exactly three in prod.
try:
- script:
env:
- name: CLUSTER_ENV
value: ($clusterEnv)
content: |
set -eu
fail() { echo "FAIL: $*" >&2; exit 1; }
total="$(kubectl get nodes -o name | grep -c . || true)"
[ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all"
# --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) -----
# A non-k3s kubelet here would mean the platform is not running on k3s.
notk3s="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \
| grep -v '+k3s' | grep -c . || true)"
[ "$notk3s" = "0" ] \
|| fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s"
# --- every node is BOTH control-plane AND etcd (docs/02 converged topology) -
# Counted by label EXISTENCE, so it does not matter whether k3s writes "true"
# or "". A control-plane node without the etcd role would be a server with an
# external/absent datastore; an etcd node without control-plane would be an
# isolated datastore member. Both are excluded by requiring all three counts
# to be equal.
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)"
echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-<unset>}"
[ "$cp" = "$total" ] \
|| fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')"
[ "$etcd" = "$total" ] \
|| fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'"
# --- every node is a REGISTERED embedded-etcd member -----------------------
# k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the
# embedded-etcd cluster. The role label alone is a label; this annotation is
# the datastore's own record of membership.
members="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \
| grep -c . || true)"
[ "$members" = "$total" ] \
|| fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server"
# --- Ready ---------------------------------------------------------------
notready="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \
| grep -v '=True$' | grep -c . || true)"
[ "$notready" = "0" ] \
|| fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test"
# --- quorum arithmetic ---------------------------------------------------
# Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is
# N - quorum; 'production-grade HA' must survive at least one loss, which
# rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds
# a tie, so it is also rejected.
quorum=$(( total / 2 + 1 ))
tolerance=$(( total - quorum ))
echo "quorum=$quorum tolerance=$tolerance"
[ "$tolerance" -ge 1 ] \
|| fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server"
[ $(( total % 2 )) -eq 1 ] \
|| fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance"
# --- the count R-PLAT-01 itself fixes, per environment --------------------
# The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a
# flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST
# exercise the same multi-node topology'.
case "${CLUSTER_ENV:-}" in
prod)
[ "$total" = "3" ] \
|| fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;;
"")
fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;;
*)
[ "$total" -ge 2 ] \
|| fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;;
esac
echo "HA_TOPOLOGY_OK"
check:
(contains($stdout, 'HA_TOPOLOGY_OK')): true
# -----------------------------------------------------------------------------------
- name: then-a-control-plane-only-vip-is-floated-on-every-server
description: >-
Read the running kube-vip DaemonSet: Ready on every control-plane node,
control-plane-only, and floating an address that collides with no other allocator.
try:
- script:
env:
- name: GATEWAY_VIP
value: ($gatewayVip)
- name: LB_POOL_START
value: ($lbPoolStart)
- name: LB_POOL_STOP
value: ($lbPoolStop)
content: |
set -eu
fail() { echo "FAIL: $*" >&2; exit 1; }
# dotted quad -> integer, for range comparison without ipcalc/python.
n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; }
# The DaemonSet is found by the label roles/kube_vip stamps on it, not by a
# hardcoded name in a namespace this test guessed.
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
[ -n "$ds" ] \
|| fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it"
desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')"
ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')"
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
vip="$(kubectl -n kube-system get "$ds" \
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
cp_enable="$(kubectl -n kube-system get "$ds" \
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')"
svc_enable="$(kubectl -n kube-system get "$ds" \
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')"
echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable"
[ -n "$vip" ] \
|| fail "the kube-vip DaemonSet announces no address — nothing is floated"
[ "$cp_enable" = "true" ] \
|| fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP"
[ "$svc_enable" = "false" ] \
|| fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)"
[ "$desired" = "$cp" ] \
|| fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane"
[ "$ready" = "$desired" ] \
|| fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over"
# --- the VIP collides with nothing --------------------------------------
node_ips="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \
| grep -c "^${vip}$" || true)"
[ "$node_ips" = "0" ] \
|| fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node"
[ "$vip" != "${GATEWAY_VIP:-}" ] \
|| fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it"
if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then
v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")"
[ "$lo" -le "$hi" ] \
|| fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it"
if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then
fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service"
fi
else
fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)"
fi
echo "HA_VIP_CONTROL_PLANE_ONLY_OK"
check:
(contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true
# -----------------------------------------------------------------------------------
- name: then-the-apiserver-answers-through-the-vip-and-on-every-server
description: >-
From inside the cluster, probe /readyz on the floated VIP and on every server's own
address, validating TLS against the cluster CA for the address dialled.
try:
- script:
# The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the
# Job cannot be a static fixture — writing the addresses down is the
# per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the
# same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped
# fixtures. The Job lives in chainsaw's per-test namespace and is torn down with
# it; nothing outside that namespace is created or changed.
content: |
set -eu
ns="$NAMESPACE"
fail() { echo "FAIL: $*" >&2; exit 1; }
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
vip="$(kubectl -n kube-system get "$ds" \
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')"
[ -n "$vip" ] || fail "no floated VIP to probe"
[ -n "$servers" ] || fail "no control-plane node addresses to probe"
echo "probing vip=$vip servers=$servers"
# The probe pod authenticates with its own ServiceAccount token and validates
# TLS against the cluster CA, both from the projected SA volume. /readyz is
# authorized for any authenticated identity via the built-in
# system:public-info-viewer role — anonymous access is NOT relied on (verified:
# an unauthenticated /readyz on this platform answers 401).
kubectl -n "$ns" apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: ha-api-endpoint-probe
spec:
backoffLimit: 1
activeDeadlineSeconds: 150
template:
spec:
restartPolicy: Never
containers:
- name: probe
image: curlimages/curl:8.11.1
env:
- name: VIP
value: "${vip}"
- name: SERVERS
value: "${servers}"
command:
- sh
- -c
- |
set -u
SA=/var/run/secrets/kubernetes.io/serviceaccount
probe() { # \$1 = address to dial
curl -sS --max-time 15 \\
--cacert "\$SA/ca.crt" \\
-H "Authorization: Bearer \$(cat \$SA/token)" \\
"https://\$1:6443/readyz"
}
rc=0
out="\$(probe "\$VIP" 2>&1)" || rc=1
echo "vip \$VIP -> [\$out]"
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)"
exit 1
fi
for s in \$SERVERS; do
rc=0
out="\$(probe "\$s" 2>&1)" || rc=1
echo "server \$s -> [\$out]"
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node"
exit 1
fi
done
echo "HA_API_ENDPOINT_OK"
EOF
# Bounded wait; on failure surface the probe's own output before failing.
if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then
kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true
fail "the in-cluster API-endpoint probe did not complete — see its log above"
fi
logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)"
echo "$logs"
case "$logs" in
*HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;;
*) fail "the probe completed without reporting success" ;;
esac
check:
(contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true
conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml
+212
View File
@@ -0,0 +1,212 @@
# =============================================================================
# L2 BEHAVIORAL — every OIDC client's STORED secret equals the one git declares
# (R-IDP-02, R-IDP-04) [RUNNABLE]
# Spec: docs/15 identity-and-access; ADR-0014.
# Impl: platform/keycloak/client-secret-reconciler.yaml (the CronJob + its script).
#
# WHY THIS TEST EXISTS
# --------------------
# The KeycloakRealmImport resolves each confidential client's secret from a `(env:NAME)`
# placeholder mapped to a key of the SOPS-encrypted keycloak-realm-clients Secret. When
# the Keycloak Operator cannot resolve a placeholder it stores the LITERAL placeholder
# text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; the CR then rests
# at Done=True / Started=False and never re-runs. Measured 2026-09-06: prod had all 12
# clients holding the literal placeholder — every OIDC login on the cluster failed with
# "Invalid client or Invalid client credentials" — and dev had one (oauth2-proxy), whose
# Secret key was added after dev's import had already completed.
#
# Every SIGNAL the cluster offered was green: the import Job succeeded, the CR read
# Done=True, the Keycloak pods were healthy, and the existing keycloak-config-as-code
# test (which asserts Done=True) PASSED throughout. The only observable was a human
# failing to log in. This test is that observable, automated: it compares STORED against
# DECLARED for every client and fails on any difference, so the class cannot sit
# undetected for months again.
#
# GIVEN the `homelab` realm and the SOPS-encrypted keycloak-realm-clients Secret,
# WHEN the reconciler's script is run in --audit mode (READ-ONLY: it cannot write, and
# its endpoint allow-list makes users/groups/roles unreachable),
# -> THEN every declared client's stored secret matches git, and none holds an
# unresolved import placeholder. Emits KEYCLOAK_CLIENT_SECRETS_AUDIT_OK.
#
# WHY IT RUNS THE SHIPPED SCRIPT rather than re-implementing the comparison: a detector
# that duplicates the healer's logic drifts from it. This audits with the same code, in
# the same mode the L0 gate selftests, so a passing audit is evidence about the healer
# too. It runs as a Job IN the keycloak namespace (not in the ephemeral chainsaw
# namespace) because the desired state is the mounted keycloak-realm-clients Secret and
# the admin credential is keycloak-admin-credentials — both namespace-local, which is
# also why neither this Job nor the CronJob needs any Kubernetes API access.
#
# chainsaw v0.2.12: Job + status assert; the leftover-Job delete keeps a re-run from
# colliding with an immutable Job left by a previous failure.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: keycloak-client-secrets-match-git
labels:
conformance.home.k3s/R-IDP-02: "true"
conformance.home.k3s/R-IDP-04: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-02, R-IDP-04"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every OIDC client's stored secret equals the SOPS-declared one and none holds an
unresolved import placeholder — the observable that was missing when all 12 prod
clients were broken behind a green realm import.
spec:
timeouts:
apply: 60s
assert: 5m
exec: 5m
steps:
- name: given-the-reconciler-is-reconciled-onto-the-cluster
description: >-
The audit Job runs the CronJob's own script from its ConfigMap; if Flux has not
applied the reconciler, there is nothing to audit WITH and that is the failure.
try:
- assert:
resource:
apiVersion: v1
kind: ConfigMap
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
- assert:
resource:
apiVersion: batch/v1
kind: CronJob
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
spec:
concurrencyPolicy: Forbid
- name: then-every-stored-client-secret-matches-git
try:
- script:
content: |
kubectl -n keycloak delete job keycloak-client-secret-audit \
--ignore-not-found --wait=true
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: keycloak-client-secret-audit
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-client-secret-audit
app.kubernetes.io/part-of: conformance
spec:
backoffLimit: 0
activeDeadlineSeconds: 240
template:
metadata:
labels:
app.kubernetes.io/name: keycloak-client-secret-audit
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: audit
image: python:3.13.15-alpine3.23
command:
- python3
- /opt/reconciler/reconcile-client-secrets.py
# READ-ONLY. --audit never issues a write on any path; the
# script's --selftest proves that (L0 gate).
- --audit
env:
- name: KEYCLOAK_BASE_URL
value: http://keycloak-service.keycloak.svc.cluster.local:8080
- name: KEYCLOAK_REALM
value: homelab
- name: PYTHONDONTWRITEBYTECODE
value: "1"
- name: PYTHONUNBUFFERED
value: "1"
- name: KEYCLOAK_ADMIN_USER
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-user
- name: KEYCLOAK_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-password
volumeMounts:
- name: script
mountPath: /opt/reconciler
readOnly: true
- name: client-secrets
mountPath: /etc/keycloak-realm-clients
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
volumes:
- name: script
configMap:
name: keycloak-client-secret-reconciler
defaultMode: 0444
- name: client-secrets
secret:
secretName: keycloak-realm-clients
defaultMode: 0400
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: keycloak-client-secret-audit
namespace: keycloak
status:
succeeded: 1
# The Job's exit status already encodes the verdict, but the sentinel is what the
# run LEDGER records (conformance/runs/*/run.log): a JUnit alone cannot tell a real
# audit from a Job that succeeded without auditing anything.
- script:
content: |
set -u
out="$(kubectl -n keycloak logs job/keycloak-client-secret-audit \
--tail=-1 2>&1 || true)"
echo "$out"
echo "$out" | grep -q 'KEYCLOAK_CLIENT_SECRETS_AUDIT_OK' \
|| { echo "FAIL: the audit did not report every client secret matching git"; exit 1; }
echo "$out" | grep -q 'UNRESOLVED IMPORT PLACEHOLDER' \
&& { echo "FAIL: a client still holds the literal import placeholder"; exit 1; }
echo KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK
check:
(contains($stdout, 'KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK')): true
catch:
- describe:
apiVersion: batch/v1
kind: Job
namespace: keycloak
name: keycloak-client-secret-audit
- podLogs:
namespace: keycloak
selector: app.kubernetes.io/name=keycloak-client-secret-audit
tail: 60
finally:
- script:
content: |
kubectl -n keycloak delete job keycloak-client-secret-audit \
--ignore-not-found --wait=false
conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml
+12
View File
@@ -11,6 +11,18 @@
#
# Goes GREEN once Keycloak is up + the realm client-secret hand-off (keycloak-realm-clients) is filled
# so the placeholder resolves and the import completes. chainsaw v0.2.12: script + block contains check.
#
# NOT PROVEN HERE — READ THIS BEFORE TRUSTING A GREEN RUN (added 2026-09-06).
# Done=True means the import JOB EXITED 0. It does NOT mean the placeholders resolved: when the
# operator cannot resolve one it stores the LITERAL `(env:...)` text as the client secret and the
# import still reports success, then rests at Done=True / Started=False and never re-runs. THIS
# TEST PASSED THROUGHOUT the 2026-09-06 outage in which all 12 prod clients held the literal
# placeholder and every OIDC login on the cluster failed. Done=True is a necessary, badly
# insufficient condition.
# The sufficient check is tests/l2-behavioral/keycloak-client-secrets-match-git, which compares
# each client's STORED secret against the SOPS-declared one; the static half is
# tests/l0-static/keycloak-client-secret-reconciler. Do not "simplify" by folding those into this
# test: the whole lesson is that the import's own success signal is not evidence.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml
+19 -5
View File
@@ -3,10 +3,15 @@
# Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client
# (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the
# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by
# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which
# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster.
#
# GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm,
# WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth)
# -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on
# keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP
# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP
# (R-IDP-01: central Identity Provider as the single user directory).
#
# Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up.
@@ -30,6 +35,11 @@ spec:
apply: 60s
assert: 3m
exec: 3m
# -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST
# (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from).
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: given-grafana-is-an-oidc-client
try:
@@ -59,6 +69,9 @@ spec:
containers:
- name: probe
image: busybox:1.36.1
env:
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
@@ -68,13 +81,14 @@ spec:
# busybox wget has no --max-redirect; -S dumps every response header
# (incl. the initial 302 Location to the IdP) before following, so the
# grep below still matches the Keycloak authorize URL. (Grafana verified
# live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.)
# live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.)
idp="keycloak.${BASE_DOMAIN}/realms/homelab"
out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)"
echo "$out"
if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then
echo GRAFANA_OIDC_REDIRECT_OK
if echo "$out" | grep -qiF "$idp"; then
echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}"
else
echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1
echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1
fi
- assert:
resource:
conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml
+6 -1
View File
@@ -3,8 +3,13 @@
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
# Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its
# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and
# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform
# substitutes; it is prose, not an input — nothing here needs parameterizing.
#
# GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available
# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is
# after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is
# the proxy->Keycloak flow working),
# WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth)
# -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal
conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml
+33 -3
View File
@@ -3,8 +3,13 @@
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
# Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
# prod's on prod, with nothing to pass at run time.
#
# GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
# WHEN an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and
# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
# verifies the chain against the SYSTEM trust store,
# THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
# chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
@@ -36,6 +41,11 @@ metadata:
that chains to a publicly-trusted CA root (LE production); the wildcard SAN
covers internal-only hostnames too.
spec:
# -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
# — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
- name: then-gateway-cert-chains-to-trusted-root
try:
@@ -54,13 +64,16 @@ spec:
containers:
- name: probe
image: curlimages/curl:8.11.1
env:
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -eu
# Probe TWO wildcard hostnames — one "public", one "internal-only". Both
# match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA
# match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
# bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
# R-TLS-03 (the same trusted wildcard covers internal-only names). Names
# resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
@@ -68,7 +81,7 @@ spec:
# the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
# hostname fails. TLS terminates at the gateway before HTTP routing, so no
# backend is needed.
for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do
for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
-o /dev/null "https://${host}/"
echo "TLS_TRUSTED ${host}"
@@ -80,3 +93,20 @@ spec:
kind: Job
metadata: { name: tls-probe }
status: { succeeded: 1 }
# -----------------------------------------------------------------------------------
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
# chainsaw already fails the test before reaching this step if any of them fails — but a
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
# forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
# is reached only when everything above has passed. Asserted by
# tests/l0-static/auditability-proofs-producible.
- name: then-emit-the-proof-of-pass-sentinel
try:
- script:
content: echo TLS_TRUSTED_PROVEN_OK
check:
(contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true
conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml
+11 -4
View File
@@ -40,10 +40,17 @@
# alternative source for the data.
#
# KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them):
# * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are
# VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9),
# so this drill does not survive loss of that hypervisor. Off-site/off-host
# replication is an operator concern outside R-STATE-03's wording.
# * OFF-HOST is NOT proven, only OFF-CLUSTER — TRUE AS RUN, and being fixed. On the
# dev estate the k3s nodes are VMs on the same Proxmox host that runs the
# versitygw/MinIO S3 target (192.168.1.9), so this drill does not survive loss of
# that hypervisor: one machine takes the cluster AND its backups.
# That is exactly why the operator moved off-cluster DR to Cloudflare R2 on
# 2026-09-05 (commit 32d6766 — S3_BACKUP_ENDPOINT is now an R2 endpoint). Once that
# merges and this drill is RE-RUN against R2, off-host stops being a gap and becomes
# proven, because the target no longer shares a failure domain with the cluster.
# UNTIL THEN THIS TEST'S PASS IS AGAINST MinIO. R-STATE-03 is currently proven only
# for the same-host target; re-proving it against R2 is a prerequisite for claiming
# off-host DR, and the ledger row that proves R-STATE-03 predates the move.
# * The test does NOT read the backup objects out of the bucket with an S3 client.
# Doing so would require handing the conformance runner the backup credential,
# widening a privileged test identity for weaker evidence than (1)-(4) above.
infra/ansible/inventory/hosts.ini
+11 -4
View File
@@ -31,10 +31,17 @@ k3s_secondary
[hypervisor]
localhost ansible_connection=local
# VPN egress-gateway node(s) — run the node-level split-tunnel OpenVPN + kill-switch
# (ADR-0015, playbooks/vpn.egress.gateway.yml). Initially just k3s-dev-3 (single gateway;
# add a second host here for Cilium Egress Gateway HA). Members are already defined above
# with their ansible_host under k3s_secondary; this group only tags the egress role target.
# VPN egress-gateway node — runs the node-level split-tunnel OpenVPN + kill-switch
# (ADR-0015, playbooks/vpn.egress.gateway.yml). EXACTLY ONE host belongs here.
#
# THIS GROUP IS SINGLETON BY DESIGN — a second host does NOT buy failover. Cilium OSS has no
# egress-gateway failover: the CRD's own `egressGateway` description says "In case multiple nodes
# are a match for the given set of labels, the first node in lexical ordering based on their name
# will be selected". Two labelled nodes therefore means one silently wins by NAME ORDER (a labelled
# `k3s-prod-1` beats a labelled `k3s-prod-3`) and the other carries no traffic. See
# platform/vpn-egress/README.md "Resilience — there is no gateway failover" for the full statement
# and the ADR-0016 escalation. Members are already defined above with their ansible_host under
# k3s_secondary; this group only tags the egress role target.
[vpn_egress_gateway]
k3s-dev-3
platform-config/kustomization.yaml
+17
View File
@@ -43,6 +43,11 @@ resources:
- ../platform/longhorn/volumesnapshotclass.yaml
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
- ../platform/monitoring/alerts.yaml
# Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
# (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
- ../platform/monitoring/pvc-capacity-alerts.yaml
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
@@ -73,9 +78,21 @@ resources:
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
# cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
# CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
# identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
# clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
- ../platform/cnpg/alerts.yaml
- ../platform/keycloak/postgres-cluster.yaml
- ../platform/keycloak/keycloak-instance.yaml
- ../platform/keycloak/keycloak-realm-import.yaml
# The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
# stored LITERALLY and the import still reports success, then rests at Done=True and
# never re-runs. This CronJob is the level-triggered loop that corrects that drift
# (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
# so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
# alongside the import whose failure mode it covers.
- ../platform/keycloak/client-secret-reconciler.yaml
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
- ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml
platform/cnpg/alerts.yaml
+182
View File
@@ -0,0 +1,182 @@
---
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
#
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
#
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
# warning for a storage outage. The generic PVC-fill backstop is the separate
# platform/monitoring/pvc-capacity-alerts.yaml.
#
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
# failure this repo keeps hitting.
#
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
# 2026-09-06). CNPG's exporter publishes TWO independent families:
#
# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
# and the backup catalogue). Present for EVERY instance.
# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
#
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
# exporter cannot log in to its own database:
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter]
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
# itself so the blind spot cannot recur unnoticed.
#
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3
# dev: 0, 0, 0 (min over 24h also 0)
# cnpg_collector_first_recoverability_point prod: 0, 0, 0
# dev: 1788018752 / 1788018762 / 1788018770
# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old
# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet)
# cnpg_last_error prod+dev: nextcloud-db = 1
#
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
# which fires only when an outage is genuinely imminent.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-cnpg
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.cnpg
rules:
# ---------------------------------------------------------------------------------------
# THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
# the hour instead of 38 hours later.
#
# pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
# archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
# seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
# segment is 16 MiB that Postgres may not recycle.
#
# THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
# condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
# queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
# returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
# 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
# A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
# backlog sits at 3 because it writes almost nothing.
- alert: CNPGWALArchiveBacklog
expr: |
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
for: 1h
labels:
severity: warning
annotations:
summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
# ---------------------------------------------------------------------------------------
# The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
# time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
# is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
# be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
# and an old base backup can exist while archiving has just broken (only the other fires).
#
# `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
# (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
# start quiet while still catching a bootstrap that never produced a recoverability point.
- alert: CNPGNoRecoverabilityPoint
expr: |
max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0
for: 6h
labels:
severity: warning
annotations:
summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}"
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
# ---------------------------------------------------------------------------------------
# The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
# platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
# databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
# backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
# CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
# ~57 years and always true). 36h = one missed daily run plus slack.
- alert: CNPGNoRecentBaseBackup
expr: |
(time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0
for: 2h
labels:
severity: warning
annotations:
summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}"
description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
# ---------------------------------------------------------------------------------------
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
# which means checkpoints are no longer reclaiming space — the mechanism that actually fills
# the PVC. Both series come from the same metric under different `value` labels, so each side
# is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
# label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
# both clusters), `count` = segments currently in pg_wal.
#
# This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
# CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
# to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
# at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
# runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
# any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
# `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
# therefore NOT usable as a signal.
- alert: CNPGWALVolumeNotRecycling
expr: |
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
for: 2h
labels:
severity: warning
annotations:
summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
# ---------------------------------------------------------------------------------------
# THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
# queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
# `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
#
# THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
# 2026-09-06: `cnpg_errors_total` carries
# pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
# database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
# i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
# Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
# restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
# permanently-firing `critical` this repo has been burned by.
- alert: CNPGMetricsCollectorErroring
expr: |
max by (namespace, pod) (cnpg_last_error) == 1
for: 6h
labels:
severity: warning
annotations:
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
platform/keycloak/README.md
+48 -4
View File
@@ -29,6 +29,7 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
@@ -47,10 +48,14 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
## Operator hand-offs (fill before / during deploy)
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. Two client
secrets (`grafana-client-secret`, `oauth2-proxy-client-secret`), `openssl rand -hex 32` each. The
realm import injects them into the Keycloak clients via `spec.placeholders`. Add to
`kustomization.yaml` once filled. The realm import stays PENDING until this exists.
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
`<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
import injects them into the Keycloak clients via `spec.placeholders`. Add to
`kustomization.yaml` once filled.
**The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
from it, and CI rejects any other name.
*(This step used to read "the realm import stays PENDING until this exists". It does not — see
the next section. That false belief is what allowed the 2026-09-06 outage.)*
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
`client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
@@ -60,6 +65,45 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
exact keys + steps. No plaintext secret is ever committed.
## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
**The realm import is a one-shot that fails open.** Measured on both clusters:
* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
error condition, and `HasErrors` stays `False`.
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
added later is never picked up.
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
signal was green throughout, including the conformance test that asserts `Done=True`.
Three layers now cover it:
| Layer | Where | What it does |
|---|---|---|
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
**Manual repair / one-off run** (the CronJob normally does this):
```sh
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
```
**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
landed.
## Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
platform/keycloak/client-secret-reconciler.yaml
+720
View File
@@ -0,0 +1,720 @@
---
# =============================================================================
# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04)
#
# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only
# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients
# Secret and the Keycloak Operator, and alongside the realm import itself.
#
# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06)
# -------------------------------------------------------------------
# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are
# import placeholders resolved from the keycloak-realm-clients Secret via
# spec.placeholders. When the operator cannot resolve a placeholder it stores the
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and
# NEVER RE-RUNS, so nothing ever corrects it:
#
# * prod: all 12 clients held the literal placeholder. Every OIDC login on the
# cluster failed with "Invalid client or Invalid client credentials".
# * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added
# AFTER dev's realm import had already completed, and a completed import is
# never re-run, so the new key was never picked up.
#
# Both were repaired live via the admin API on 2026-09-06 and verified. This
# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is
# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same
# posture Flux gives every other resource in this repo (R-GITOPS-03 drift
# correction, applied to the one piece of state Flux itself cannot reach --
# Keycloak's own database).
#
# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph
# already guarantees the Secret is applied before the realm import (the Secret is
# rendered by ./platform; the import by ./platform-config, which dependsOn
# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of
# the Secret lands before revision N of the import). Ordering still cannot help,
# because the operator will not re-run a completed import when a LATER revision
# adds a key -- which is precisely how dev broke. Only a reconciler closes that.
#
# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state
# lives in Keycloak's Postgres database, not in a Kubernetes object, so no
# admission or mutation policy can see it. The admin REST API is the only
# interface to it.
#
# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token,
# client lookup, client-secret read, minimal client PUT). Users, groups, roles,
# authentication flows and realm settings are structurally unreachable from it --
# prod's `homelab` realm carries a real user and the groups media-users /
# platform-admins / viewers, and they must survive every run. It never logs a
# secret, and it writes only where the stored value differs from git.
#
# PROOF: the script's own `--selftest` drives these real code paths against an
# in-process stub Keycloak and asserts compare-then-write, idempotence (a second
# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint
# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script
# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships:
# conformance/tests/l0-static/keycloak-client-secret-reconciler/.
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
data:
# SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy:
# the L0 gate extracts THIS block and executes its --selftest, so a second copy
# would be untested. Contains no dollar sign at all, on purpose -- this file is
# rendered by a Flux Kustomization with postBuild.substituteFrom, so an
# unescaped dollar-brace expansion here would be rewritten before any container
# ran (the trap documented at length in apps/wave-1/nextcloud.yaml).
reconcile-client-secrets.py: |
#!/usr/bin/env python3
# =============================================================================
# reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret
# equal the value git declares for it, idempotently, without touching anything
# else in the realm.
#
# WHY THIS EXISTS (the incident it prevents)
# ------------------------------------------
# keycloak-realm-import.yaml declares each confidential client's secret as an
# `(env:<NAME>_CLIENT_SECRET)` import placeholder and maps <NAME> to a key of the
# SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the
# Keycloak Operator cannot resolve a placeholder at import time it stores the
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
# SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and
# never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12
# clients with secrets held the literal placeholder and every OIDC login failed
# with "Invalid client or Invalid client credentials"; on dev exactly one
# (oauth2-proxy) was broken, because that Secret key was added after dev's import
# had already completed.
#
# So the import is a ONE-SHOT that fails OPEN. This is the level-triggered
# reconciler that closes it.
#
# CONTRACT
# --------
# Desired state files in SECRETS_DIR named `<clientId>-client-secret`, each
# holding exactly the secret for that clientId (this is the
# `keycloak-realm-clients` Secret, mounted).
# Actual state GET /admin/realms/<realm>/clients/<uuid>/client-secret
# Action only where they differ: PUT /admin/realms/<realm>/clients/<uuid>
# with the MINIMAL representation {"clientId":..,"secret":..}.
#
# WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a
# two-field PUT rewrites the secret and nothing else. PUTting the full
# round-tripped representation back has a far larger blast radius and is in fact
# rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the
# JSON"} (measured during the live repair). The stub in --selftest models that
# rejection, so a regression to the fat PUT fails in CI, not on a live realm.
#
# SAFETY PROPERTIES, each enforced in code and proved by --selftest:
# 1. compare-then-write: a client already correct is SKIPPED, no PUT issued.
# 2. idempotent: a second run over a converged realm issues zero writes.
# 3. no secret is ever printed, in any mode, on any path.
# 4. the ONLY endpoints this program may touch are the master-realm token
# endpoint, the client LOOKUP, the client-secret READ and the minimal client
# PUT. Every other URL is refused by _http() before a socket is opened, so
# users, groups, roles, authentication flows and realm settings are
# structurally unreachable from here -- prod's `homelab` realm carries a
# real user and the groups media-users / platform-admins / viewers, and they
# MUST survive.
# 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped.
#
# MODES
# (default) reconcile: compare, and write only what differs.
# --audit READ-ONLY. Report drift and exit 1 if any client's stored secret
# differs from the declared one. Never writes. This is what the L2
# conformance test runs.
# --selftest the offline behavioural proof, against an in-process stub
# Keycloak. No cluster; loopback only. This is what the L0 static
# gate runs, so the code the gate proves is the code that ships.
#
# NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor
# (hence re.fullmatch throughout), and not even inside these comments. It ships
# inside a ConfigMap that the `platform-config` Flux Kustomization renders with
# postBuild.substituteFrom, so Flux envsubst rewrites this text before any
# container runs: an unescaped dollar-brace expansion would be silently emptied at
# reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's
# OIDC guard dead code for months -- read the comment block there. Python needs no
# dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and
# the character is banned outright so no future edit has to reason about escaping.
# Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh.
# =============================================================================
from __future__ import annotations
import hmac
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients")
KEY_SUFFIX = "-client-secret"
TIMEOUT = 20
READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24"))
READY_SLEEP = float(os.environ.get("READY_SLEEP", "5"))
# Endpoint allow-list (safety property 4). fullmatch-ed against the path of every
# URL _http() is asked to open; anything else raises before a socket exists.
_ALLOWED = (
r"/realms/master/protocol/openid-connect/token",
r"/admin/realms/[^/]+/clients",
r"/admin/realms/[^/]+/clients/[^/]+",
r"/admin/realms/[^/]+/clients/[^/]+/client-secret",
)
class Refused(RuntimeError):
"""A URL outside the allow-list was requested. Never expected in a normal run."""
def _http(method, base, path, query=None, headers=None, form=None, body=None):
"""Return (status, text). Refuses any path outside the allow-list."""
if not any(re.fullmatch(p, path) for p in _ALLOWED):
raise Refused("endpoint not on the allow-list: %s %s" % (method, path))
url = base.rstrip("/") + path
if query:
url = url + "?" + urllib.parse.urlencode(query)
hdrs = dict(headers or {})
data = None
if form is not None:
data = urllib.parse.urlencode(form).encode()
hdrs["Content-Type"] = "application/x-www-form-urlencoded"
elif body is not None:
data = json.dumps(body).encode()
hdrs["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=hdrs, method=method)
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
return resp.status, resp.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, e.read().decode("utf-8", "replace")
def declared(secrets_dir):
"""{clientId: secret} from the mounted Secret.
Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal
entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A
key must be named `<clientId>-client-secret`: that naming contract is what
lets this reconciler derive the client set from git without a second
declaration, and the L0 static check enforces it against
keycloak-realm-import.yaml so the two can never drift apart.
"""
out = {}
for name in sorted(os.listdir(secrets_dir)):
if name.startswith(".") or not name.endswith(KEY_SUFFIX):
continue
path = os.path.join(secrets_dir, name)
if not os.path.isfile(path):
continue
with open(path, "r", encoding="utf-8") as fh:
value = fh.read()
# A trailing newline in the SOPS source would otherwise become part of the
# secret and mismatch forever. Keycloak secrets never contain whitespace.
out[name[: -len(KEY_SUFFIX)]] = value.strip()
return out
def admin_token(base, user, password, log):
"""Bearer token for admin-cli in the master realm, with a bounded readiness wait."""
last = ""
for attempt in range(1, READY_ATTEMPTS + 1):
try:
st, text = _http(
"POST", base, "/realms/master/protocol/openid-connect/token",
form={"client_id": "admin-cli", "grant_type": "password",
"username": user, "password": password})
except OSError as e:
st, text = 0, "connection error (%s)" % e.__class__.__name__
if st == 200:
token = json.loads(text).get("access_token")
if token:
return token
last = "http 200 but no access_token in the response"
else:
# Deliberately do NOT echo the response body: it is harmless today, but
# the request that produced it carried the admin password, and keeping
# that class of mistake impossible by habit is cheaper than reviewing it.
last = "http %s" % st
log("waiting for the Keycloak admin API (attempt %d/%d): %s"
% (attempt, READY_ATTEMPTS, last))
if attempt < READY_ATTEMPTS:
time.sleep(READY_SLEEP)
raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s"
% (READY_ATTEMPTS, last))
def reconcile(base, realm, token, want, audit, log):
"""Return (changed, problems). Writes only when audit is False."""
hdrs = {"Authorization": "Bearer " + token}
changed = 0
problems = []
for cid in sorted(want):
st, text = _http("GET", base, "/admin/realms/%s/clients" % realm,
query={"clientId": cid}, headers=hdrs)
if st != 200:
problems.append("%s: client lookup failed (http %s)" % (cid, st))
continue
found = json.loads(text)
if not isinstance(found, list) or not found:
problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it "
"but the realm has no such client (the operator runs the "
"realm import exactly once)" % (cid, realm))
continue
client = found[0]
uuid = client.get("id")
if client.get("publicClient"):
log("%-14s skipped (public client, holds no secret)" % cid)
continue
st, text = _http("GET", base,
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
headers=hdrs)
if st != 200:
problems.append("%s: could not read the stored secret (http %s)" % (cid, st))
continue
stored = json.loads(text).get("value") or ""
if hmac.compare_digest(stored, want[cid]):
log("%-14s ok (already correct, no write)" % cid)
continue
# Describe the drift WITHOUT quoting either value. The literal-placeholder
# case is named because it is the failure this program exists for, and
# saying so reveals nothing secret.
kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER"
if "env:" in stored else "stored value differs from the declared one")
if audit:
problems.append("%s: DRIFT -- %s" % (cid, kind))
continue
log("%-14s DRIFT (%s) -- repairing" % (cid, kind))
# MINIMAL representation only -- see the header. Keycloak merges non-null
# fields, so protocol mappers, redirect URIs, flows, groups, roles and users
# are all untouched by this write.
st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid),
headers=hdrs, body={"clientId": client.get("clientId"),
"secret": want[cid]})
if not 200 <= st < 300:
problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160]))
continue
st, text = _http("GET", base,
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
headers=hdrs)
readback = (json.loads(text).get("value") or "") if st == 200 else ""
if not hmac.compare_digest(readback, want[cid]):
problems.append("%s: PUT was accepted but the readback still does not match"
% cid)
continue
changed += 1
log("%-14s REPAIRED and verified by readback" % cid)
return changed, problems
def run(argv, log):
audit = "--audit" in argv
base = os.environ.get("KEYCLOAK_BASE_URL", "").strip()
realm = os.environ.get("KEYCLOAK_REALM", "").strip()
user = os.environ.get("KEYCLOAK_ADMIN_USER", "")
password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")
for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm),
("KEYCLOAK_ADMIN_USER", user),
("KEYCLOAK_ADMIN_PASSWORD", password)):
if not value:
log("FATAL: %s is empty" % name)
return 1
log("mode=%s realm=%s base=%s"
% ("audit (read-only)" if audit else "reconcile", realm, base))
try:
want = declared(SECRETS_DIR)
except OSError as e:
log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__))
return 1
if not want:
log("FATAL: no `<clientId>%s` keys found in %s -- the keycloak-realm-clients "
"Secret is missing or empty, and reporting success here would repeat the "
"very fail-open the realm import already has"
% (KEY_SUFFIX, SECRETS_DIR))
return 1
log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want))))
try:
token = admin_token(base, user, password, log)
changed, problems = reconcile(base, realm, token, want, audit, log)
except Refused as e:
log("FATAL: %s" % e)
return 1
except (RuntimeError, OSError, ValueError) as e:
log("FATAL: %s: %s" % (e.__class__.__name__, e))
return 1
if problems:
log("")
for p in problems:
log("PROBLEM %s" % p)
log("")
if audit:
log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems))
else:
log("RECONCILE FAIL: %d client secret(s) could not be reconciled"
% len(problems))
return 1
if audit:
log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match "
"git" % len(want))
else:
log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired"
% (len(want), changed))
return 0
# =============================================================================
# --selftest -- the offline behavioural proof. It drives the REAL code paths above
# against an in-process stub Keycloak, so what the gate proves is what ships.
# =============================================================================
def _refused(path):
"""True iff _http() refuses `path` outright (allow-list, safety property 4)."""
try:
_http("PUT", "http://127.0.0.1:1", path, body={})
except Refused:
return True
except OSError:
return False
return False
def _selftest():
import contextlib
import http.server
import io
import tempfile
import threading
global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP
SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
SECRET_C = "cccccccccccccccccccccccccccccccc"
STALE = "0ld-r0tated-value"
ADMIN_PW = "stub-admin-password"
# The exact value the broken clusters stored. chr(36) is the dollar sign,
# spelled this way so the file stays free of the character envsubst reacts to.
LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)"
state = {"clients": {}, "writes": [], "touched": []}
class Stub(http.server.BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def _send(self, code, obj):
payload = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def _route(self, method):
parsed = urllib.parse.urlparse(self.path)
path, qs = parsed.path, urllib.parse.parse_qs(parsed.query)
state["touched"].append("%s %s" % (method, path))
if path == "/realms/master/protocol/openid-connect/token":
return self._send(200, {"access_token": "stub-token"})
if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path):
cid = (qs.get("clientId") or [""])[0]
c = state["clients"].get(cid)
return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)])
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret",
path)
if m and method == "GET":
c = state["clients"].get(m.group(1))
if c is None:
return self._send(404, {"error": "not found"})
return self._send(200, {"type": "secret", "value": c.get("secret", "")})
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path)
if m and method == "PUT":
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
rep = json.loads(raw or b"{}")
# The live server rejects a full round-tripped representation with
# 400 "Cannot parse the JSON"; modelled so a regression to the fat
# PUT fails here rather than against a real realm.
if set(rep) != {"clientId", "secret"}:
return self._send(400, {
"error": "unknown_error",
"error_description": "Cannot parse the JSON"})
state["writes"].append(m.group(1))
state["clients"][m.group(1)]["secret"] = rep["secret"]
return self._send(204, {})
return self._send(500, {"error": "an endpoint the reconciler must not touch"})
def do_GET(self):
self._route("GET")
def do_PUT(self):
self._route("PUT")
def do_POST(self):
self._route("POST")
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub)
threading.Thread(target=srv.serve_forever, daemon=True).start()
base = "http://127.0.0.1:%d" % srv.server_address[1]
tmp = tempfile.mkdtemp()
for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B),
("kubernetes", SECRET_C), ("gitea", SECRET_A)):
with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh:
fh.write(secret + "\n") # trailing newline: must be tolerated
# Kubernetes' own projection entries must be ignored, not read as clients.
os.mkdir(os.path.join(tmp, "..data"))
SECRETS_DIR = tmp
READY_ATTEMPTS, READY_SLEEP = 2, 0.01
os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab",
"KEYCLOAK_ADMIN_USER": "admin",
"KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW})
failures = []
checks = [0]
def expect(cond, what):
checks[0] += 1
if not cond:
failures.append(what)
def invoke(*argv):
state["writes"] = []
state["touched"] = []
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = run(list(argv), print)
return rc, buf.getvalue()
def reset():
state["clients"] = {
# already correct
"grafana": {"clientId": "grafana", "publicClient": False,
"secret": SECRET_A},
# THE INCIDENT: the literal, unresolved import placeholder
"pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL},
# public client: no secret, must be skipped
"kubernetes": {"clientId": "kubernetes", "publicClient": True},
# ordinary drift (rotated in git, stale in the realm)
"gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE},
}
def leaked(text):
return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW)
if s in text]
# -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING
reset()
rc, out = invoke("--audit")
expect(rc == 1, "audit must exit 1 when a stored secret differs from git")
expect(state["writes"] == [],
"audit mode must never write (writes=%r)" % state["writes"])
expect("UNRESOLVED IMPORT PLACEHOLDER" in out,
"audit must name the unresolved-placeholder case explicitly")
expect("pihole: DRIFT" in out and "gitea: DRIFT" in out,
"audit must report BOTH drifted clients, not stop at the first")
expect(re.search(r"grafana\s+ok \(already correct", out),
"an already-correct client must be reported as such")
expect(re.search(r"kubernetes\s+skipped \(public client", out),
"a public client must be skipped, not treated as drift")
expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out))
# -- 2. reconcile: repairs exactly the drifted clients, verified by readback
reset()
rc, out = invoke()
expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:])
expect(sorted(state["writes"]) == ["gitea", "pihole"],
"reconcile must write exactly the drifted clients, got %r" % state["writes"])
expect("grafana" not in state["writes"],
"compare-then-write: an already-correct client must not be written")
expect("kubernetes" not in state["writes"], "a public client must not be written")
expect("REPAIRED and verified by readback" in out,
"every repair must be readback-verified")
expect("2 repaired" in out, "the summary must count the repairs")
expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out))
# -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing
rc, out = invoke()
expect(rc == 0, "the second reconcile must exit 0")
expect(state["writes"] == [],
"IDEMPOTENCE: a converged realm must take ZERO writes, got %r"
% state["writes"])
expect("0 repaired" in out, "the second run must report 0 repaired")
rc, out = invoke("--audit")
expect(rc == 0, "audit must pass once the realm has converged")
expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out,
"a clean audit must emit the sentinel the L2 conformance test greps for")
# -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched
reset()
invoke()
for touched in state["touched"]:
expect(re.fullmatch(
r"POST /realms/master/protocol/openid-connect/token"
r"|GET /admin/realms/homelab/clients"
r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?",
touched), "the reconciler touched an off-list endpoint: %s" % touched)
expect(not [t for t in state["touched"]
if "/users" in t or "/groups" in t or "/roles" in t
or "/authentication" in t],
"the reconciler must never touch users/groups/roles/authentication")
expect(_refused("/admin/realms/homelab/users"),
"the allow-list must refuse a users endpoint before opening a socket")
expect(_refused("/admin/realms/homelab/groups"),
"the allow-list must refuse a groups endpoint before opening a socket")
expect(_refused("/admin/realms/homelab"),
"the allow-list must refuse a whole-realm update endpoint")
# -- 5. a declared client the realm never got is LOUD, not a silent pass
reset()
del state["clients"]["gitea"]
rc, out = invoke("--audit")
expect(rc == 1, "a declared client missing from the realm must fail the audit")
expect("NOT FOUND" in out, "the missing client must be named in the output")
# -- 6. an empty / unmounted Secret must fail closed, never report success
SECRETS_DIR = tempfile.mkdtemp()
rc, out = invoke()
expect(rc == 1, "an empty secrets dir must fail closed")
expect("keys found in" in out, "the empty-secret failure must say what is missing")
SECRETS_DIR = tmp
srv.shutdown()
print("")
if failures:
print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0]))
for f in failures:
print(" FAIL %s" % f)
return 1
print("SELFTEST OK: %d assertions (compare-then-write, idempotence, "
"no-secret-logging, endpoint allow-list, fail-closed)" % checks[0])
return 0
def main():
argv = sys.argv[1:]
if "--selftest" in argv:
return _selftest()
return run(argv, lambda m: print(m, flush=True))
if __name__ == "__main__":
sys.exit(main())
---
# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key
# added after the realm import already completed) converges within one interval
# instead of waiting for a human to notice a failed login. Runs are cheap: a
# converged realm performs 12 GETs and ZERO writes.
apiVersion: batch/v1
kind: CronJob
metadata:
name: keycloak-client-secret-reconciler
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
spec:
schedule: "*/15 * * * *"
# Forbid: a run that is still waiting for Keycloak must not be lapped by the next
# tick. Two concurrent reconcilers would be harmless (both converge to the same
# value) but would double the write attempts during an outage recovery.
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
ttlSecondsAfterFinished: 3600
template:
metadata:
labels:
app.kubernetes.io/name: keycloak-client-secret-reconciler
app.kubernetes.io/part-of: keycloak
spec:
restartPolicy: Never
# The reconciler talks ONLY to the Keycloak admin API. It needs no
# Kubernetes API access at all — its two inputs are mounted Secrets — so
# it gets no ServiceAccount token (R-SEC-02, least privilege).
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: reconciler
# Stock upstream python; the reconciler is pure standard library, so
# nothing is installed at runtime. Pinned exactly, like busybox /
# curlimages elsewhere in the repo. Renovate does not manage plain
# `image:` tags (renovate.json disables image-tag updates so it never
# fights Flux image-automation), so this is bumped by hand.
image: python:3.13.15-alpine3.23
imagePullPolicy: IfNotPresent
command:
- python3
- /opt/reconciler/reconcile-client-secrets.py
env:
- name: KEYCLOAK_BASE_URL
value: http://keycloak-service.keycloak.svc.cluster.local:8080
- name: KEYCLOAK_REALM
value: homelab
# readOnlyRootFilesystem is on; keep CPython from trying to write
# bytecode caches beside the read-only ConfigMap mount.
- name: PYTHONDONTWRITEBYTECODE
value: "1"
- name: PYTHONUNBUFFERED
value: "1"
# Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to
# bootstrap the admin; keys admin-user / admin-password.
- name: KEYCLOAK_ADMIN_USER
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-user
- name: KEYCLOAK_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-password
volumeMounts:
- name: script
mountPath: /opt/reconciler
readOnly: true
# The DESIRED state: one file per declared client, named
# `<clientId>-client-secret`. Mounting the Secret (rather than
# reading it through the API) is what lets this pod run with no
# ServiceAccount token at all.
- name: client-secrets
mountPath: /etc/keycloak-realm-clients
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
volumes:
- name: script
configMap:
name: keycloak-client-secret-reconciler
defaultMode: 0444
- name: client-secrets
secret:
secretName: keycloak-realm-clients
defaultMode: 0400
platform/monitoring/alerts.yaml
+80 -1
View File
@@ -247,8 +247,37 @@ spec:
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
#
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
# to have a tunnel, via two independent guards OR'd together:
#
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
# while the gateway node is DOWN: gateway-node-down is still alerted.
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
# of the KSM rollout.
#
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
# dev (tunnel up), and clause 2 already true on dev.
- alert: VPNTunnelDown
expr: absent(node_network_receive_bytes_total{device="tun0"})
expr: |
absent(node_network_receive_bytes_total{device="tun0"})
and on()
(
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
or
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
)
for: 5m
labels:
severity: critical
@@ -359,6 +388,56 @@ spec:
summary: "Keycloak database unavailable"
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
# --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
# Done=True passed). The only observable was a human failing to log in. The CronJob is
# the healer; these two rules make its SILENCE observable, so the class cannot sit
# undetected again. They are the runtime counterpart of the L2 audit test.
#
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
# these series had no samples to check against. `kube_job_status_failed` /
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
# import Job exports them); the reconciler's own series appear with its first run.
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
# job_name=~"keycloak-client-secret-reconciler-.*"}
- alert: KeycloakClientSecretReconcilerFailing
# A reconciler run failed. Either it found drift it could not repair, or it could not
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak client-secret reconciler failing"
description: >-
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
secret may differ from git (unresolved import placeholder, or an unrepairable
drift) and SSO logins for that client will fail. Read the Job log: it names the
client and never prints a secret.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
- alert: KeycloakClientSecretReconcilerStale
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
# the healer being quietly absent, which is exactly the "looks applied and isn't"
# failure this whole change exists to end.
expr: |
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
for: 2h
labels:
severity: warning
annotations:
summary: "Keycloak client-secret reconciler has not succeeded"
description: >-
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
The CronJob may be suspended, unapplied, or unable to schedule — client-secret
drift would go uncorrected and unnoticed (docs/15).
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
platform/monitoring/kube-prometheus-stack.yaml
+11
View File
@@ -221,6 +221,17 @@ spec:
limits:
memory: 256Mi
kube-state-metrics:
# Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
# have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
# `kube_node_labels` does not exist and any join against it silently matches nothing.
# Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
# cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
# — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
metricLabelsAllowlist:
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
resources:
requests:
cpu: 20m
platform/monitoring/pvc-capacity-alerts.yaml
+148
View File
@@ -0,0 +1,148 @@
---
# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
#
# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
# the SSO database with nothing watching it.
#
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
# working on both clusters.
#
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
# from platform-config/kustomization.yaml.
#
# ------------------------------------------------------------------------------------------------
# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
#
# Two facts, both established live rather than assumed:
#
# 1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
# DIFFERENT labels for the same PVC:
# job="kubelet" -> namespace=<the PVC's real namespace>, node=<node>
# job="apiserver" -> namespace="default", exported_namespace=<the PVC's real namespace>
# Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
# only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
# rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
# `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
# pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
# say namespace="default") and this file WOULD double-fire on them — and would report the wrong
# namespace while doing it.
#
# 2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
# covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
# only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
# which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
#
# ------------------------------------------------------------------------------------------------
# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
#
# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
# it, unifi-rootdisk is the only match at every threshold.
#
# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
# at 88% free.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-pvc-capacity
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.pvc-capacity
rules:
# Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
# (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
# scope differs.
- alert: PersistentVolumeFillingUp
expr: |
(
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
) < 0.15
and kubelet_volume_stats_inodes{job="kubelet"} > 0
for: 15m
labels:
severity: warning
annotations:
summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
# Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
# volume is genuinely about to run out, which for a Postgres data volume means an imminent
# disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
# permanently-firing critical. There is no counterpart for this tier anywhere else in the
# repo, so it cannot duplicate one.
- alert: PersistentVolumeAlmostFull
expr: |
(
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
) < 0.05
and kubelet_volume_stats_inodes{job="kubelet"} > 0
for: 15m
labels:
severity: critical
annotations:
summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
# TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
# for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
# ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
# surprise. This rule instead asks "is this volume on course to hit zero?".
#
# 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
# rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
# horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
# hour, so a single burst of writes cannot notify on its own.
#
# The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
# not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
# dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
# platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
# every OTHER PVC in the cluster, which is what was missing entirely.
#
# NOTE the monitoring namespace is excluded here too (partition rule above). That also
# side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
# retention high-water mark looks exactly like an unbounded leak until it plateaus.
- alert: PersistentVolumeFillingUpPredicted
expr: |
(
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
) < 0.50
and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
and kubelet_volume_stats_inodes{job="kubelet"} > 0
for: 1h
labels:
severity: warning
annotations:
summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
platform/vpn-egress/README.md
+39
View File
@@ -28,6 +28,45 @@ Three layers, two owners:
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
`internet`-class and node traffic are unaffected.
## Resilience — there is NO gateway failover (spec correction, escalated)
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
Verified against the CRD actually deployed on this cluster
(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
Cilium **v1.20.1**) — the field's own description:
> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
> **In case multiple nodes are a match for the given set of labels, the first node in lexical
> ordering based on their name will be selected.**"
Consequences, stated plainly:
- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
`[vpn_egress_gateway]` group is singleton by design.
- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
(fail-closed, per R-NET-06 — no leak, but no service).
- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
`k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
blackholes that entry's share and re-shards the remainder (breaking live connections), and each
shard leaves from a *different* public IP. Upstream confirmation:
[cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
support failover between nodes") and
[cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
tolerate single node failure").
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
## Secrets — host-side now
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
platform/vpn-egress/egress-steering.yaml
+20
View File
@@ -69,6 +69,26 @@ spec:
# ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
# vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
# internet fails OPEN).
#
# SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
# `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
# deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
# `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
#
# "EgressGateway is the gateway node responsible for SNATing traffic.
# In case multiple nodes are a match for the given set of labels, the first node
# in lexical ordering based on their name will be selected."
#
# So labelling a second node does not create an active/standby pair — it just hands the role to
# whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
# labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
# LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
# it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
egressGateway:
nodeSelector:
matchLabels: