From 6a8953e28839a482ac502da93135fdda2f2417bc Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 00:59:22 -0500 Subject: [PATCH] feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Minimal prod delivery, authored directly on main (= prod's live revision d1a6628) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 --- ci/l0.sh | 38 + ci/touched-requirements.sh | 32 + conformance/incluster/rbac.yaml | 7 + conformance/lib/status.py | 232 +++++- conformance/tests/README.md | 51 +- .../chainsaw-test.yaml | 106 +++ .../auditability-proofs-producible/check.sh | 431 +++++++++++ .../chainsaw-test.yaml | 75 ++ .../dev-nodes-provisioned-as-code/check.sh | 442 +++++++++++ .../chainsaw-test.yaml | 94 +++ .../egress-paths-selected-by-rules/check.sh | 513 ++++++++++++ .../chainsaw-test.yaml | 88 +++ .../ha-control-plane-as-code/check.sh | 729 ++++++++++++++++++ .../chainsaw-test.yaml | 63 ++ .../check.sh | 529 +++++++++++++ .../chainsaw-test.yaml | 68 ++ .../placeholder-guard-coupling/check.sh | 516 +++++++++++++ .../chainsaw-test.yaml | 77 ++ .../prod-baremetal-not-dev-surface/check.sh | 528 +++++++++++++ .../chainsaw-test.yaml | 69 ++ .../tls-lifecycle-automated/check.sh | 478 ++++++++++++ .../chainsaw-test.yaml | 391 ++++++++++ .../default-deny/chainsaw-test.yaml | 17 + .../dns-exposure/chainsaw-test.yaml | 58 +- .../chainsaw-test.yaml | 54 +- .../egress-default-path/chainsaw-test.yaml | 199 +++++ .../fixtures/three-egress-pods.yaml | 67 ++ .../chainsaw-test.yaml | 34 +- .../fixtures/app-and-probe.yaml | 13 +- .../ha-control-plane/chainsaw-test.yaml | 379 +++++++++ .../chainsaw-test.yaml | 212 +++++ .../chainsaw-test.yaml | 12 + .../oidc-sso-grafana/chainsaw-test.yaml | 24 +- .../sso-forward-auth/chainsaw-test.yaml | 7 +- .../trusted-tls/chainsaw-test.yaml | 36 +- .../dr-restore/chainsaw-test.yaml | 15 +- infra/ansible/inventory/hosts.ini | 15 +- platform-config/kustomization.yaml | 17 + platform/cnpg/alerts.yaml | 182 +++++ platform/keycloak/README.md | 52 +- .../keycloak/client-secret-reconciler.yaml | 720 +++++++++++++++++ platform/monitoring/alerts.yaml | 81 +- .../monitoring/kube-prometheus-stack.yaml | 11 + platform/monitoring/pvc-capacity-alerts.yaml | 148 ++++ platform/vpn-egress/README.md | 39 + platform/vpn-egress/egress-steering.yaml | 20 + 46 files changed, 7898 insertions(+), 71 deletions(-) create mode 100644 conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/auditability-proofs-producible/check.sh create mode 100644 conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh create mode 100644 conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh create mode 100644 conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/ha-control-plane-as-code/check.sh create mode 100644 conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh create mode 100644 conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/placeholder-guard-coupling/check.sh create mode 100644 conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh create mode 100644 conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/tls-lifecycle-automated/check.sh create mode 100644 conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml create mode 100644 conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml create mode 100644 conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml create mode 100644 conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml create mode 100644 conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml create mode 100644 platform/cnpg/alerts.yaml create mode 100644 platform/keycloak/client-secret-reconciler.yaml create mode 100644 platform/monitoring/pvc-capacity-alerts.yaml diff --git a/ci/l0.sh b/ci/l0.sh index efcedda..7101823 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -137,6 +137,44 @@ else esac fi +# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ---- +# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)` +# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as +# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and +# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or +# Invalid client credentials") behind an entirely green cluster; dev had one. The check +# asserts the declaration is self-consistent (every placeholder defined, every Secret key +# present, names matching the reconciler's contract), that the Flux ordering is a REAL +# dependsOn edge rather than a comment, and — by extracting the shipped script from its +# ConfigMap and running its --selftest — that the reconciler is compare-then-write, +# idempotent, non-logging and blast-radius-bounded. +# +# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance +# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/, +# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 +# and the requirement-scoped selection would filter it out of exactly the change that +# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no +# side effects, so running it twice when 3b also selects it is not worth avoiding. +KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}" +log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})" +if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then + skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green" + summary "- keycloak-client-secrets: **SKIPPED** (check missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + kc_rc=0 + bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$? + case "$kc_rc" in + 0) ok "Keycloak client-secret declaration + reconciler passed" + summary "- keycloak-client-secrets: **passed**" ;; + 3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass" + summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi ;; + *) err "Keycloak client-secret declaration + reconciler FAILED" + summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;; + esac +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/ci/touched-requirements.sh b/ci/touched-requirements.sh index 181c36a..89bf931 100755 --- a/ci/touched-requirements.sh +++ b/ci/touched-requirements.sh @@ -180,9 +180,41 @@ path_requirements() { platform/externaldns/*) echo "R-DNS-02" ;; platform/velero/*) echo "R-STATE-03" ;; platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;; + # platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db / + # nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change + # there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules + # for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO + # requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate. + platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;; + # vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the + # R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07 + # steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements` + # annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not + # happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed + # ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins + # when a diff does carry the tags. + platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;; + # Keycloak is the whole SSO surface: the realm import (declarative identity config), + # the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after + # the client-secret incident, in which editing platform/keycloak/ attributed NO + # requirement, so the L2 keycloak tests were filtered out of the merge gate's touched + # subset — the change that could break every OIDC login was gated by nothing. + # R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and + # its off-cluster backup credential, not only the realm. + platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;; + platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;; platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;; conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;; clusters/*) echo "R-GITOPS-01 R-ENV-02" ;; + # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing: + # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be + # enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any + # already-mapped path scoped the L0 run to that other path's requirements and silently + # excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists + # precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that + # stopped matching when the value it guards changed, in a different file) is exactly the + # class that omission would hide. + bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;; .gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;; *) : ;; esac diff --git a/conformance/incluster/rbac.yaml b/conformance/incluster/rbac.yaml index 36b7b79..72150cf 100644 --- a/conformance/incluster/rbac.yaml +++ b/conformance/incluster/rbac.yaml @@ -57,6 +57,13 @@ rules: verbs: [create, delete, get, list, watch, patch] - apiGroups: [""] resources: [pods, services, configmaps, secrets, persistentvolumeclaims] + # LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment- + # agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable, + # oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run + # time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via + # chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings. + # If this rule is ever narrowed to the test's own namespace, those bindings fail and the + # tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap. verbs: [create, delete, get, list, watch, patch] - apiGroups: [""] resources: [pods/exec] diff --git a/conformance/lib/status.py b/conformance/lib/status.py index a91f748..d3e7ffa 100755 --- a/conformance/lib/status.py +++ b/conformance/lib/status.py @@ -297,6 +297,109 @@ _EXECUTABLE_RE = re.compile( _BLOCKED_RE = re.compile( r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M) +# ---------------------------------------------------------- scaffold-marker detection +# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text` +# --------------------------------------------------- +# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT +# is placeholder detection has to name the idioms it hunts for: +# `l0-static/placeholder-guard-coupling/check.sh` carries +# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data. +# A whole-file substring test read that as the test declaring itself unfinished, classified +# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation +# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as +# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose. +# +# The rule below is derived from the corpus (43 tests), not invented. Every genuine +# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/ +# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of +# two structural positions, and no executable test does either: +# +# 1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the +# repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say +# `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare +# `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing. +# 2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC +# bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a +# message is a note about work that is missing. A marker that is the ENTIRE content of +# a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the +# marker as a *datum*, which is what a placeholder DETECTOR necessarily contains. +# +# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any +# `render-check.sh` / `check.sh`, and either one still OUTRANKS an +# `executable: "true"` annotation (see classify_test) — the invariant that an annotation +# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an +# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker +# carrying any trailing text all classify as scaffold. +# +# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with +# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding +# several idioms — is still called a scaffold. That is the conservative direction, and the +# author's remedy is the same structural discipline the corpus already uses: keep each +# idiom a standalone literal in a list. + +# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`, +# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in +# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares +# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches +# `placeholder.*`), and none of them is used here to mark a placeholder body. +# Case-insensitive: no English word contains either token, and the corpus carries no +# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`). +_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I) + +# The dedicated declaration slot, on a comment line, case-insensitive. The declared state +# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state, +# and all three real scaffolds read `# STATUS: scaffold — `. Anchoring it +# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a +# loose `.*scaffold` would turn any prose that merely uses the word into a declaration. +_STATUS_SCAFFOLD_RE = re.compile( + r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I) + +# A quoted literal whose entire content is one marker word: the marker as data. +_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I) + + +def _split_doc_header(text: str) -> tuple[str, str]: + """Split a test file into (documentation header, body). + + Both file kinds in this tree open with a banner comment block that is PROSE about the + test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT + prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses + placeholder idioms when the test hunts them, so an incidental mention there is + documentation, not a placeholder body. + + The body starts at the first line that is neither blank nor a `#` comment: + *.sh -> `set -euo pipefail` (the shebang is a comment) + *.yaml -> `apiVersion: chainsaw.kyverno.io/v1alpha1` + Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture + with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml + # TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting. + """ + lines = text.splitlines(keepends=True) + for i, ln in enumerate(lines): + s = ln.strip() + if s and not s.startswith("#"): + return "".join(lines[:i]), "".join(lines[i:]) + return text, "" # no executable line at all: it is all header, body is empty + + +def scaffold_marker(text: str) -> str: + """Why `text` declares itself a placeholder body, or "" if it does not. + + Returns a short human-readable reason (used verbatim in the hygiene report) so the + defect is actionable rather than a bare path. + """ + head, body = _split_doc_header(text) + m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body) + if m: + return m.group(0).strip().lstrip("#").strip()[:80] + # Blank out markers held as pure data before looking for a marker that has a message. + m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body)) + if m2: + line = body[:m2.start()].count("\n") + snippet = body.splitlines()[line].strip() + return f"placeholder marker in body: {snippet[:80]}" + return "" + def classify_test(t: dict) -> dict: """Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively. @@ -304,14 +407,18 @@ def classify_test(t: dict) -> dict: A tagged test is only worth counting if it can actually assert something. Rules, in order, chosen so ambiguity NEVER upgrades a test: - scaffold the body contains TODO markers (a placeholder), *even if* it - is annotated executable — an annotation must never outrank the - body. That mismatch is also reported as a hygiene defect. - executable annotated `conformance.home.k3s/executable: "true"` and no TODOs. - executable-untagged no TODOs and a real runnable body, but the annotation is - missing (true of the L1 render tests, which execute a - render-check.sh + golden diff, of the L0 static checks, and - of the META self-check). + scaffold the test DECLARES itself a placeholder — a `# STATUS: scaffold` + header line, or a TODO/FIXME marker carrying a message in the + body (see `scaffold_marker` for the rule and the corpus it was + derived from) — *even if* it is annotated executable. An + annotation must never outrank the body. That mismatch is also + reported as a hygiene defect. + executable annotated `conformance.home.k3s/executable: "true"` and no + scaffold declaration. + executable-untagged no scaffold declaration and a real runnable body, but the + annotation is missing (true of the L1 render tests, which + execute a render-check.sh + golden diff, of the L0 static + checks, and of the META self-check). Counted as executable, and flagged as an annotation gap. scaffold anything else. """ @@ -322,16 +429,26 @@ def classify_test(t: dict) -> dict: # downgrades the test to a scaffold — an annotation must never outrank the body, # whichever file the body lives in. extra = "" + scripts: list[tuple[str, str]] = [] for script in ("render-check.sh", "check.sh"): sp = p.parent / script if sp.is_file(): - extra += sp.read_text() + body = sp.read_text() + extra += body + scripts.append((script, body)) declared = bool(_EXECUTABLE_RE.search(text)) - has_todo = "TODO" in text or "TODO" in extra + # Each file is classified on ITS OWN structure — the yaml and the script have different + # header blocks, so they must not be concatenated before the header/body split. + reason = scaffold_marker(text) + for script, body in scripts: + if reason: + break + r = scaffold_marker(body) + reason = f"{script}: {r}" if r else "" bm = _BLOCKED_RE.search(text) has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M)) - if has_todo: + if reason: state = "scaffold" elif declared: state = "executable" @@ -348,7 +465,8 @@ def classify_test(t: dict) -> dict: "name": t["name"], "path": rel, "layer": t["layer"], "state": state, "declared_executable": declared, - "annotation_conflict": bool(declared and has_todo), + "annotation_conflict": bool(declared and reason), + "scaffold_reason": reason, "blocked_by": bm.group(1).strip().strip('"') if bm else "", } @@ -640,6 +758,10 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l # --- hygiene defects this tool owns ------------------------------------------- hygiene = { "annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]], + # Same set, with WHY each one was called a placeholder body, so the fix is obvious + # from the report instead of requiring the reader to re-derive the rule. + "annotation_conflict_reasons": {t["path"]: t["scaffold_reason"] + for t in classified if t["annotation_conflict"]}, "executable_untagged": [t["path"] for t in classified if t["state"] == "executable-untagged"], "registry_not_in_spec": [], @@ -910,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str: else: o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; " "this is a skip, not a pass)") - o.append(f"- tests annotated executable but containing TODO bodies: " - f"{h['annotation_conflicts'] or 'none'}") + o.append("- tests annotated executable but declaring a placeholder body: " + + (", ".join(f"`{k}` ({v})" + for k, v in sorted(h["annotation_conflict_reasons"].items())) + if h["annotation_conflicts"] else "none")) o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: " f"{h['executable_untagged'] or 'none'}") @@ -930,9 +1054,12 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str: "reconciled |") o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test " "(the authoritative tag, same source as `coverage.md`) | exact |") - o.append("| executable vs scaffold | TODO markers in the test body outrank the " - "`executable` annotation; a real body without the annotation still counts " - "and is flagged | conservative — ambiguity never upgrades |") + o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a " + "TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare " + "quoted datum — a placeholder detector's own vocabulary — is not one); " + "either outranks the `executable` annotation, and a real body without the " + "annotation still counts and is flagged | conservative — ambiguity never " + "upgrades |") o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for " "every verifying test: JUnit result **plus** that test's own pass sentinel " "token grepped from the run log | exact where a run was recorded, refused " @@ -1127,6 +1254,70 @@ def _selftest() -> int: classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"] == "scaffold") + # === THE BUG THIS DETECTOR WAS REWRITTEN FOR ============================== + # A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard- + # coupling` — a 32-assertion executable check whose SUBJECT is placeholder + # detection, so it must name the idioms it hunts — as a scaffold, and through it + # reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These + # assertions pin BOTH directions: a marker held as DATA must not downgrade, and a + # real placeholder body must still be caught. Both halves are load-bearing; the + # second is what stops this fix from degrading into "always executable". + yml = ("metadata:\n annotations:\n" + " conformance.home.k3s/executable: \"true\"\n" + "spec:\n steps:\n - script:\n" + " content: bash ./check.sh\n") + f.write_text(yml) + detector = td2 / "check.sh" + # Shaped exactly like the real check: a prose header that discusses the idioms, + # then executable code holding them as a vocabulary list. + detector.write_text( + "#!/usr/bin/env bash\n" + "# L0 static check — placeholder vocabulary\n" + "# * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n" + "# STATUS: EXECUTABLE. A pure offline assertion script.\n" + "set -euo pipefail\n" + "python3 - <<'EOF'\n" + "SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n" + "assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n" + "EOF\n") + c = classify_test({"name": "s", "path": str(f), "layer": "L0"}) + check("a marker in HEADER PROSE does not downgrade an executable check", + c["state"] == "executable" and not c["annotation_conflict"]) + check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)", + c["scaffold_reason"] == "") + + # ... and the protection is intact: a real placeholder body is still caught, in + # each of the three shapes the corpus actually uses. + detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + "echo \"TODO: assert the PVC is bound\"\n") + c = classify_test({"name": "s", "path": str(f), "layer": "L0"}) + check("a TODO carrying a message in check.sh IS still a scaffold", + c["state"] == "scaffold" and c["annotation_conflict"]) + check("the scaffold reason names the file and the offending line", + c["scaffold_reason"].startswith("check.sh:") + and "TODO: assert the PVC is bound" in c["scaffold_reason"]) + detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + "# TODO(T-4.1) write the hermetic sink fixture\ntrue\n") + check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold", + classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"] + == "scaffold") + detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + "echo \"FIXME the extraction before trusting this\"\n") + check("an unpunctuated FIXME carrying trailing prose IS still a scaffold", + classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"] + == "scaffold") + # The dedicated declaration slot outranks a body that would otherwise look real — + # this is how all three genuine scaffolds in the tree declare themselves. + detector.unlink() + f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n" + "# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml) + c = classify_test({"name": "s", "path": str(f), "layer": "L3"}) + check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding", + c["state"] == "scaffold" and c["annotation_conflict"]) + check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration", + scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline " + "assertion script; covers no scaffold.\nset -e\n") == "") + # ---------------------------------------------------------------- PROOF / ledger # These are the assertions that stand between this report and a fake green. check("empty ledger parses to no rows (pre-ledger behaviour is preserved)", @@ -1341,8 +1532,11 @@ def main() -> int: f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n") rc = 1 if h["annotation_conflicts"]: - sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO " - f"bodies: {h['annotation_conflicts']}\n") + sys.stderr.write( + "[status] STRICT: tests annotated executable that declare a placeholder " + "body: " + "; ".join(f"{k} ({v})" for k, v in + sorted(h["annotation_conflict_reasons"].items())) + + "\n") rc = 1 if rep["safety_critical_without_executable_l3"]: sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 " diff --git a/conformance/tests/README.md b/conformance/tests/README.md index 8fcf3a7..dc10f4b 100644 --- a/conformance/tests/README.md +++ b/conformance/tests/README.md @@ -93,9 +93,52 @@ and coverage extraction exact and robust. break the thing and assert fail-closed / no-data-loss, with `finally` restoring state. - **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement. +- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no + Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod, + so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw + binding over the same ConfigMap Flux substitutes from: + + ```yaml + spec: + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) + ``` + + Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it: + `($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a + full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`, + which chainsaw templates with the same bindings. Prefer reading the *actual* observed value + over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP + the Service was assigned) — it is both stricter and environment-independent. Never take the + environment from a CLI flag or an exported variable: a flag someone forgets is how a suite + ends up silently asserting the other cluster's expectations. L1 goldens are the one + exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token. + +### How a scaffold declares itself (and how `status.py` tells one from a verifier) Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a -later roadmap phase delivers the capability under test. `lib/status.py` treats such a -marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as -outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body -can never be reported as an executable test. +later roadmap phase delivers the capability under test. `lib/status.py` classifies each +test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`, +`render-check.sh`, `check.sh` — and either signal below **outranks** +`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as +an executable test. Declare a scaffold in **both** positions: + +1. **`# STATUS: scaffold — `** in the file's header comment block. The + declared state must be the first token after `STATUS:` (executable tests say + `# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`). +2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first + non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`, + or an inline comment on a missing fixture: + `file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`. + +What is **not** a scaffold marker, and why the distinction is load-bearing: a marker +mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does +NOT prove" list), and a marker that is the **entire content of a quoted literal** — a +placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in +`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to +classify that 32-assertion check as a scaffold and, through it, report the safety-critical +R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each +idiom a **standalone quoted literal in a list**; a marker with text attached (a regex +alternation, several idioms in one string) still classifies as a scaffold, because +ambiguity here only ever downgrades. diff --git a/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml new file mode 100644 index 0000000..85117d3 --- /dev/null +++ b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml @@ -0,0 +1,106 @@ +# ============================================================================= +# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an +# operator can prove isolation, egress path, cert validity, and backup integrity)"), +# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator +# must be able to prove": Isolation (denied-probe test), Egress path (egress IP +# check), Kill-switch (fault-injection test), TLS (cert status + external check), +# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not +# one-time checks", with the acceptance gate "The five auditability proofs above can +# be produced on demand." +# +# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement) +# ----------------------------------------------------------------------------------- +# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that +# compiles and would keep passing while every proof rotted into a scaffold. Four distinct +# failure modes are asserted instead, each by DRIVING the real machinery: +# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer +# the spec's own words require (fault injection / a DR drill => L3), taken from +# lib/status.py's own verdict so this check cannot disagree with the status report; +# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements ` +# must resolve to real test dirs AND pass the merge gate's coverage assertion; +# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can +# attribute — a proof that passes but records nothing leaves the operator with a green +# terminal and no evidence; +# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the +# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP +# token, and a ledger row claiming `proven` at the repository's ROOT commit — and +# requires record_run.py to report not-proven and status.py to report `stale-proof`. It +# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in +# ci/ or run.sh ran before this test existed. +# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo +# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly, +# so a sixth proof added to the spec fails here instead of being silently unverified. +# +# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster, +# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE +# anyone needs it. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate, +# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN, +# and two synthetic forgeries are presented to them, +# THEN each of docs/07's five proofs maps to an executable test at the required layer, the +# documented one-command invocation selects and asserts them, each is capturable as a +# ledger row, and neither a self-skip nor a stale row can be recorded as proof. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's +# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run +# it directly, and this Chainsaw Test wraps the SAME script. +# +# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE +# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny, +# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass +# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs +# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The +# assertion here was NOT relaxed to accommodate them (R-TEST-06). +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * That the five proofs PASS. This asserts they are present, strong, runnable by one +# command, capturable, and that the chain refuses forgeries — not that isolation holds +# today (R-NET-01's test) or that the restore works (R-STATE-03's). +# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this +# suite queries Hubble. Only the denied-probe half is automated — a real, named gap. +# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that +# would need signed history. +# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every +# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately +# not re-asserted here. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: auditability-proofs-producible + labels: + conformance.home.k3s/R-SEC-03: "true" + annotations: + conformance.home.k3s/requirements: "R-SEC-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree and the harness's own tooling, when status.py, the requirement + resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence + tools' self-tests are driven and two synthetic forgeries are presented to them, then each + of docs/07's five auditability proofs has an executable test at the layer the spec + requires, is selectable and asserted by the documented one-command invocation, is + capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as + evidence. +spec: + steps: + - name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh +# --------------------------------------------------------------------------------------------- +# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder +# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is +# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof +# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would +# report itself as unexecutable. Observed and worked around here; the detector's crudeness is +# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is +# the wrong trade (R-TEST-06). diff --git a/conformance/tests/l0-static/auditability-proofs-producible/check.sh b/conformance/tests/l0-static/auditability-proofs-producible/check.sh new file mode 100755 index 0000000..bb36f30 --- /dev/null +++ b/conformance/tests/l0-static/auditability-proofs-producible/check.sh @@ -0,0 +1,431 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — auditability-proofs-producible +# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can +# prove isolation, egress path, cert validity, and backup integrity)." +# — spec docs/00-vision-and-scope.md). +# +# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md +# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove": +# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a +# denied-probe test. +# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check. +# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test. +# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert +# status + external check. +# - **Backups**: "irreplaceable data is restorable" — via a DR drill. +# "These proofs are part of the platform acceptance suite, not one-time checks." +# and its acceptance gate: "The five auditability proofs above can be produced on demand." +# +# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY +# ------------------------------------------------------------ +# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01, +# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and +# would keep passing while every proof rotted into a scaffold. Four separate failure modes +# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted +# by DRIVING the real machinery rather than by reading files: +# +# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a +# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and, +# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an +# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this +# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the +# repository's own status report. +# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh +# --requirements ` must resolve to a non-empty test set and must ASSERT +# their coverage (that flag is the merge gate's own path: it passes --require to +# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the +# gate wiring break, R-SEC-03 fails. +# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator +# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must +# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with +# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail +# instead of silently making the requirement unprovable. +# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause +# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the +# real tools, in a throwaway directory: +# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log +# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3 +# test. record_run.py must refuse to call it `proven`. +# * a ledger row claiming `proven` at a commit that PREDATES the requirement's +# implementing code — status.py must degrade it to `stale-proof`, never green. +# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing +# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof +# were themselves unexercised in the merge gate until this check. +# +# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec +# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's +# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather +# than quietly ignoring it. +# +# WHY L0: every clause above is a property of the repository and its own tooling, provable +# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact +# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true +# with one. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one +# documented command, capturable as evidence, and that the evidence chain refuses +# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR +# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the +# proof, which is what is asserted. +# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this +# suite queries Hubble, and this check does not pretend otherwise: it asserts the +# denied-probe half, which is the half that is automated. The Hubble half remains an +# operator-interactive capability — a real, named gap in R-SEC-03's coverage. +# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status +# report's prose, the ledger's provenance fields) is a usability property, not a testable +# one. +# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review; +# nothing here can prove a row was never removed from it (that would need signed history). +# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than +# the five bullets docs/07 enumerates. The five are what the spec made concrete and are +# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim, +# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +: "${SPEC_REPO:=${ROOT}/../home-network-k3s}" +export SPEC_REPO + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(sys.argv[1]) +CONF = ROOT / "conformance" +LIB = CONF / "lib" +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +def die(): + print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} " + "assertions failed", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +def run(cmd, **kw) -> subprocess.CompletedProcess: + return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw) + + +# ================================================= THE FIVE PROOFS docs/07 ENUMERATES +# key = the bullet label docs/07 uses (verified against the spec below when it is mounted) +# reqs = the requirement(s) whose verifying test IS that proof +# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection +# test" / "a DR drill", an L2 happy-path test does not discharge the proof. +PROOFS = { + "Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"}, + "why": '"service X cannot reach service Y" — via Hubble flow logs + a ' + "denied-probe test"}, + "Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"}, + "why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'}, + "Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"}, + "why": '"with VPN down, X leaks nothing" — via a fault-injection test'}, + "TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"}, + "why": '"every endpoint serves a valid, non-self-signed, unexpired cert" ' + "— via cert status + external check"}, + "Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"}, + "why": '"irreplaceable data is restorable" — via a DR drill'}, +} +PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]}) + +# ---- SELF-POLICING: the proof list must still be the SPEC's proof list --------------- +spec = Path(os.environ.get("SPEC_REPO", "")) +sec_doc = spec / "docs" / "07-security-and-secrets.md" +if sec_doc.is_file(): + text = sec_doc.read_text() + m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text, + re.M | re.S) + check(m is not None, + "docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — " + "the proof list this check is coupled to has moved or been removed, so the mapping " + "below can no longer be trusted (R-TEST-06: follow the spec, never the other way)") + if m: + labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M) + check(set(labels) == set(PROOFS), + f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check " + f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying " + "test, and a proof this check does not know about would be silently unverified") + check(len(labels) == 5, + f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance " + "gate says 'The five auditability proofs above can be produced on demand' — the " + "prose and the list have drifted apart") +else: + note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be " + "re-derived from docs/07, so the mapping below was used as written. Every other " + "assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)") + +# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH +# Driven through lib/status.py so this check and the repository's own status report cannot +# disagree about what is executable and what is a scaffold. +tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-")) +try: + out = tmp / "status" + st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT), + "--registry", str(CONF / "requirements" / "registry.yaml"), + "--tests", str(CONF / "tests"), "--out", str(out)]) + check(st.returncode == 0, + f"lib/status.py exited {st.returncode} — the report an operator would use to find " + f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}") + status = {} + sj = out / "status.json" + if sj.is_file(): + status = json.loads(sj.read_text()) + check(bool(status.get("by_requirement")), + "lib/status.py produced no per-requirement report — the audit posture of each " + "requirement is not derivable, so R-SEC-03 cannot be assessed at all") + by_req = status.get("by_requirement", {}) + + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + rec = by_req.get(rid) + check(rec is not None, + f"proof '{label}' maps to {rid}, which the status report does not know — " + "either the registry lost the requirement or this mapping is stale") + if not rec: + continue + tests = (rec.get("verification") or {}).get("tests") or [] + executable = [t for t in tests if t.get("state", "").startswith("executable")] + check(bool(executable), + f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are " + f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: " + f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the " + "operator cannot prove this property") + at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]] + check(bool(at_layer), + f"proof '{label}' ({rid}) has no executable test at layer(s) " + f"{sorted(spec_proof['layers'])} — it has " + f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. " + f"docs/07 requires this proof to be produced {spec_proof['why']}, and a " + "weaker layer does not discharge it (a rendering check is not a drill)") + + # ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND + # The resolver: --requirements must translate the five R-IDs into real test dirs. + resolver = run(["bash", "-c", + f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; ' + f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"']) + dirs = [l for l in resolver.stdout.splitlines() if l.strip()] + check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS), + f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) " + f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's " + "acceptance gate depends on ('produced on demand') cannot select the proofs") + + # The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a + # zero exit here is the repository's own statement that all five proofs are covered. + gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)]) + check(gate.returncode == 0, + f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited " + f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit " + f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}") + + # ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE + # Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass + # token can pass forever and never be recorded as proof of anything. + sys.path.insert(0, str(LIB)) + os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1") + sys.dont_write_bytecode = True + import record_run as rr # noqa: E402 + + smap = rr.sentinel_map(CONF / "tests") + check(len(smap) >= 10, + f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its " + "grip on the test tree, so the assertions below would assert nothing") + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + rec = by_req.get(rid) or {} + tests = (rec.get("verification") or {}).get("tests") or [] + names = [t.get("name") for t in tests + if t.get("state", "").startswith("executable") + and t.get("layer") in spec_proof["layers"]] + for name in names: + ent = smap.get(name) + check(ent is not None, + f"proof '{label}': record_run.py does not know the test {name!r} — its " + "result could never be attributed to a ledger row") + if not ent: + continue + if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS: + continue # offline layers are truthful by JUnit alone + check(bool(ent.get("pass")), + f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, " + "so lib/record_run.py can never record it as proven. It would go green " + "in the terminal and leave the operator with no evidence — exactly the " + "gap the ledger exists to close (docs/07: the proofs are 'part of the " + "platform acceptance suite, not one-time checks')") + check(bool(ent.get("terminal_pass")), + f"proof '{label}': the test {name!r} has no TERMINAL pass token, so " + "record_run.py cannot decide its verdict") + + # =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE + # Both tools' own guards, which NOTHING in ci/ or run.sh drives today. + for tool in ("record_run.py", "status.py"): + r = run([sys.executable, str(LIB / tool), "--selftest"]) + check(r.returncode == 0, + f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a " + "self-skipped or stale run from being recorded as proof are broken, so nothing " + f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}") + + # ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token. + # This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof. + forge_target, forge_skip = None, None + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + for name, ent in smap.items(): + if rid in (ent.get("requirements") or []) and ent.get("skip") \ + and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS: + forge_target, forge_skip = name, ent["skip"][0] + break + if forge_target: + break + if forge_target: + break + check(forge_target is not None, + "no Chainsaw-layer proof test declares a skip token, so the forgery attempt below " + "could not be mounted. That is not a pass: it means this check could not verify " + "that a self-skip is refused as proof, which is the whole point of the ledger") + if forge_target: + run_dir = tmp / "20260101T000000Z-forged" + run_dir.mkdir(parents=True) + (run_dir / "report.junit.xml").write_text( + '\n' + '' + f'' + "\n") + (run_dir / "run.log").write_text( + f"=== RUN chainsaw/{forge_target}\n" + f" === STDOUT\n {forge_skip}\n") + forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir), + "--dry-run", "--ledger", str(tmp / "never-written.jsonl"), + "--tests", str(CONF / "tests"), "--repo", str(ROOT)]) + verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b", + forged.stdout, re.M) + check(verdicts and all(v != "proven" for v in verdicts), + f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: " + "a JUnit saying `passed` with nothing in the log but that test's own SKIP token " + f"({forge_skip}). A self-skip must never become proof — with this hole every " + "audit proof in docs/07 could be produced by a cluster that skipped it") + check(not (tmp / "never-written.jsonl").exists(), + "record_run.py --dry-run WROTE to the ledger — a tool that records while " + "claiming not to cannot be trusted to record only real runs") + + # ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code. + # status.py must degrade it to stale-proof (not green), or an old green run would keep + # vouching for code it never executed. + root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"]) + old_sha = (root_sha.stdout.strip().splitlines() or [""])[0] + check(len(old_sha) == 40, + "the repository's root commit could not be resolved, so the stale-proof guard could " + "not be exercised (that is a gap in this check, not a pass)") + if len(old_sha) == 40 and forge_target: + stale_ledger = tmp / "stale.jsonl" + rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0] + stale_ledger.write_text(json.dumps({ + "ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev", + "chainsaw_version": "v0.0.0", "kube_version": "unknown", + "layer": smap[forge_target].get("layer", "L3"), "test": forge_target, + "requirements": [rid], "junit_result": "passed", + "sentinel": (smap[forge_target].get("terminal_pass") or "X"), + "proof": "proven", "run_id": "20260101T000000Z-forged", + "note": "synthetic row written by the R-SEC-03 conformance check", + }) + "\n") + out2 = tmp / "status-stale" + st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT), + "--registry", str(CONF / "requirements" / "registry.yaml"), + "--tests", str(CONF / "tests"), "--out", str(out2), + "--ledger", str(stale_ledger)]) + check(st2.returncode == 0, + f"lib/status.py failed against a synthetic ledger: " + f"{(st2.stderr or st2.stdout).strip()[:300]}") + s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {} + rec2 = (s2.get("by_requirement") or {}).get(rid) or {} + per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])} + check(per.get(forge_target) == "stale-proof", + f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT " + f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A " + "proof from a run that predates the code it vouches for is not evidence, and " + "honouring it would let one old green run certify every later change") + check((rec2.get("proof") or {}).get("state") != "proven", + f"{rid} is reported PROVEN on the strength of a row from the root commit — the " + "freshness guard is not holding") + + # ---- the ledger itself is machine-readable, and no row proves a test that is gone ---- + ledger = CONF / "runs" / "ledger.jsonl" + if ledger.is_file(): + rows, bad = [], [] + for i, line in enumerate(ledger.read_text().splitlines(), 1): + if not line.strip(): + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + bad.append(i) + check(not bad, + f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record " + "is not machine-readable, so no proof can be derived from it") + for r in rows: + for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof", + "run_id"): + check(key in r, + f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without " + "provenance cannot be audited") + if r.get("proof") == "proven": + check(bool(r.get("sentinel")), + f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — " + "the sentinel is the only thing separating a real pass from a self-skip") + check(r.get("test") in smap, + f"a ledger row claims `proven` for {r.get('test')!r}, which is not a " + "test in the suite any more — a proof of something that no longer exists") + check("-dirty" not in str(r.get("git_sha", "")), + f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree " + f"({r.get('git_sha')}) — what ran is not what is committed") + else: + note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions " + "were skipped; every other assertion ran, and the forgery probes above use " + "synthetic ledgers, not this file") +finally: + shutil.rmtree(tmp, ignore_errors=True) + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 30, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +for n in NOTES: + print(f" note: {n}") +print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 " + f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the " + f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves " + "and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can " + "attribute, both evidence tools' self-tests pass, and two independent forgeries — a " + "self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. " + "R-SEC-03's producibility half only: whether each proof PASSES is its own " + "requirement's test, and the Hubble half of the isolation proof is automated nowhere " + "(see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml new file mode 100644 index 0000000..68f3af5 --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml @@ -0,0 +1,75 @@ +# ============================================================================= +# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: +# created, baselined, cluster-joined, and managed as code"); the four stages +# are enumerated in docs/02 §"Node lifecycle stages". +# +# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live +# Node object cannot answer it — a node built by OpenTofu and one built by hand in a +# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with +# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four +# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the +# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection +# to any node), not by reading files: a chain that does not resolve fails the check. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test. +# The dev group names are read from the playbooks' own default host patterns, the node +# names are generated from the IaC module's prefix/count variables, and the two sets are +# compared against each other — so the assertion survives a rename or a resize. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) driven, +# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted +# provider — a cloud provider fails), from a for_each map rather than a count index, +# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the +# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies +# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid +# declaratively; one k3s role founds the cluster with --cluster-init and joins the +# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at +# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux) +# resolves and targets every dev node at every stage. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in no-connection modes; no cluster, no network); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or +# hypervisor access, neither available offline. +# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is +# parsed, not evaluated; a network-allowed CI stage would close this. +# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check. +# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by +# tests/l3-fault-injection/node-loss (R-STATE-02). +# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the +# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code", +# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the +# divergence rather than hiding it. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: dev-nodes-provisioned-as-code + labels: + conformance.home.k3s/R-ENV-01: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the dev provisioning chain is parsed and driven + offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined + and handed to Flux by one code path that covers all of them — with no node outside + it and no stage missing. +spec: + steps: + - name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh new file mode 100755 index 0000000..0f033ef --- /dev/null +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh @@ -0,0 +1,442 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code +# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created, +# baselined, cluster-joined, and managed as code." — spec +# docs/00-vision-and-scope.md; the four stages are enumerated in +# docs/02-environments-and-provisioning.md §"Node lifecycle stages +# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')": +# 1. Create — OpenTofu instantiates the VM +# 2. Baseline — OS baseline and prerequisites +# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables +# k3s bundled flannel/traefik/servicelb/kube-proxy +# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC +# and re-applied, never hand-edited on the box.) +# +# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY +# ---------------------------------------------------------------------- +# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster +# cannot answer it: a Kubernetes node object looks identical whether it was built by +# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by +# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE +# provisioning path exists for all four stages and covers every dev node — and, for the +# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode +# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that +# does not actually resolve fails here. The IaC half is asserted by parsing the module: +# `tofu init/validate` needs the provider registry, and L0 is offline by contract. +# +# GIVEN the repository as git tracks it, +# WHEN the provisioning chain is parsed and (where offline execution is possible) +# actually driven, +# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider +# is a KVM-family hypervisor provider — no cloud provider may appear), +# instantiates its nodes from a for_each map (never a count index, so one +# node can be replaced without disturbing its siblings), and gives each +# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a +# cloud-init seed carrying the login user + SSH key; +# AND the node set it creates is EXACTLY the dev node set Ansible manages — +# names derived from the module's own prefix/count, so no node can be +# hand-added to the inventory or hand-created in the hypervisor; +# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel +# modules persisted AND loaded, sysctls, the Longhorn data disk formatted +# and mounted, swap off, iscsid enabled), driven by variables with +# defaults rather than per-host special cases; +# JOIN: one role installs k3s for both roles of every node — `--cluster-init` +# on the primary, `--server` join on the secondaries — with flannel, +# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium +# installed as the CNI afterwards; +# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap; +# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and +# `--list-hosts` shows every one of those stages targeting EVERY dev +# node, so no node is outside the code path. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here. +# The dev group names come from the playbooks' own default host patterns, the node names +# from the IaC module's prefix/count variables, and the two sets are compared to each +# other. This check therefore keeps working when the estate is renamed or resized. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can: +# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor +# access. `tofu plan` against the live estate is the operator's own check. +# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads +# the provider from the registry; L0 is offline by contract, so the module is parsed, +# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI +# with a network-allowed stage, not here. +# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the +# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property. +# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof +# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02). +# - That the baseline actually ran on a node (cluster/host state), and that the +# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as +# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a +# network topology fact. +# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as +# cloud-init applying the OS baseline. The implementation moved the rich baseline into +# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network), +# because Proxmox provisioning is API-only and a custom cloud-init snippet needs +# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be +# "baselined ... as code", not that cloud-init be the tool, so this check asserts the +# STAGE, wherever it lives, and records the divergence rather than hiding it. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +TOFU = ROOT / "infra" / "tofu" +ANS = ROOT / "infra" / "ansible" +check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)") +check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code") +if not TOFU.is_dir() or not ANS.is_dir(): + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + +TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf")))) +check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes") + +# ============================================================ STAGE 1 — CREATE (IaC) +# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving +# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught +# here rather than being noticed later. Each entry is justified, not just listed: +# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts. +# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate). +KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"} +providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and + [(m.group(1), m.group(2)) for m in + re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)]) +check(len(providers) > 0, + "infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' " + "cannot be checked at all") +for alias, source in sorted(providers.items()): + check(source.lower() in KVM_PROVIDERS, + f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor " + f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs") + +# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates. +vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}', + TF, re.S) +check(len(vm_blocks) >= 1, + "no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node") +for rtype, rname, body in vm_blocks: + check("for_each" in body, + f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node ' + "replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so " + "'created ... as code' would not survive a rebuild") + check(not re.search(r"^\s*count\s*=", body, re.M), + f'resource "{rtype}" "{rname}" uses a positional count index') + for want, why in ( + (r"\bcpu\b", "CPU"), + (r"\bmemory\b", "RAM"), + (r"network_device|network_interface", "a NIC"), + (r"initialization|cloud_init|cloudinit", "a cloud-init seed"), + ): + check(re.search(want, body) is not None, + f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, ' + "disks, NICs and a cloud-init seed to come from IaC") + disks = len(re.findall(r"^\s*disk\s*\{", body, re.M)) + check(disks >= 2, + f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS ' + "disk PLUS a dedicated data disk for Longhorn") + check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None, + f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be ' + "reached by the BASELINE stage without a hand step") + +# Node identity comes from variables, and those variables define the whole node set. +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r'default\s*=\s*(.+)', m.group(1)) + return d.group(1).strip() if d else None + + +prefix_raw = tf_default("node_name_prefix") +count_raw = tf_default("node_count") +ips_raw = tf_default("node_ips") +check(prefix_raw is not None and count_raw is not None, + "infra/tofu has no node_name_prefix/node_count variables — the node set is not derived " + "from code, so the created nodes cannot be compared with the managed ones") +prefix = (prefix_raw or "").strip('"') +try: + node_count = int((count_raw or "0").strip()) +except ValueError: + node_count = 0 +check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created") +if ips_raw is not None: + n_ips = len(re.findall(r'"[^"]+"', ips_raw)) + check(n_ips >= node_count, + f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module " + "cannot address every node it creates") +check((TOFU / "outputs.tf").is_file() and + re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None, + "infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be " + "told the node facts by hand instead of consuming them from IaC") + +# ------------------------------------------------- the dev node set, derived twice over +# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here +# restates the estate's naming. +PLAYBOOKS = ANS / "playbooks" + + +def default_patterns(playbook: str) -> list[str]: + p = PLAYBOOKS / playbook + if not p.is_file(): + return [] + return re.findall(r"default\('([^']+)'\)", p.read_text()) + + +dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml")) +check(len(dev_groups) > 0, + "no playbook declares a default host pattern — the dev target is not expressed in code") + +INV = ANS / "inventory" / "hosts.ini" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code") +ini_groups: dict[str, list[str]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in (INV.read_text().splitlines() if INV.is_file() else []): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + (children if cur.endswith(":children") else ini_groups).setdefault( + cur.split(":")[0] if cur.endswith(":children") else cur, []) + continue + if cur is None or cur.endswith(":vars"): + continue + host = s.split()[0] + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(host) + else: + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> set[str]: + seen = seen or set() + if group in seen: + return set() + seen.add(group) + out = set(ini_groups.get(group, [])) + for child in children.get(group, []): + out |= hosts_in(child, seen) + return out + + +dev_hosts: set[str] = set() +for g in sorted(dev_groups): + dev_hosts |= hosts_in(g) +check(len(dev_hosts) > 0, + f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed") +expected = {f"{prefix}{i + 1}" for i in range(node_count)} +check(dev_hosts == expected, + f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC " + f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)") + +# ============================================================ STAGE 2 — BASELINE (code) +BASE_ROLE = ANS / "roles" / "node_baseline" +check(BASE_ROLE.is_dir(), + "no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)") +base_tasks = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +base_defaults = "\n".join(p.read_text(errors="replace") + for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else "" +# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it — +# a declarative, idempotent task, not a shell one-liner. +for pattern, what in ( + (r"ansible\.builtin\.apt:", "a package baseline (apt module)"), + (r"modprobe:", "kernel modules loaded (modprobe module)"), + (r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"), + (r"sysctl", "sysctl tuning"), + (r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"), + (r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"), + (r"swap", "swap disabled"), + (r"iscsid", "iscsid enabled (Longhorn prerequisite)"), +): + check(re.search(pattern, base_tasks) is not None, + f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and " + "k3s/Longhorn/KubeVirt prerequisites to be applied as code") +check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and + re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None, + "node_baseline's package/module sets are not variables with defaults — a per-host special " + "case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)") + +# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s) +K3S_ROLE = ANS / "roles" / "k3s_server" +k3s_text = "\n".join(p.read_text(errors="replace") + for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else "" +check(K3S_ROLE.is_dir() and len(k3s_text) > 200, + "no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)") +check("--cluster-init" in k3s_text, + "the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded") +check(re.search(r"--server\s+https", k3s_text) is not None, + "the k3s role never joins secondaries with --server — nodes would not form one cluster") +check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None, + "the k3s role never supplies a join token — the join could only be completed by hand") +# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the +# flag that actually disables it. +for pattern, what in ( + (r"--flannel-backend=none", "flannel (Cilium is the CNI)"), + (r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"), + (r"traefik", "traefik (Gateway API via Cilium)"), + (r"servicelb", "servicelb"), +): + check(re.search(pattern, k3s_text) is not None, + f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at " + "install, or the bundled component fights the platform's own") +check((ANS / "roles" / "cilium").is_dir(), + "no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap") +check((ANS / "roles" / "flux_bootstrap").is_dir(), + "no flux_bootstrap role — stage 4's hand-off to GitOps has no code") + +# ============================================== STAGE 4 — MANAGE + the chain is REAL +SITE = PLAYBOOKS / "site.yml" +check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path") +site_txt = SITE.read_text() if SITE.is_file() else "" +order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)] +check(order[:1] == ["node-baseline.yml"], + f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)") +for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"): + check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain") +if "k3s.yml" in order and "cilium.yml" in order: + check(order.index("k3s.yml") < order.index("cilium.yml"), + "site.yml installs Cilium before k3s — the CNI cannot precede the cluster") + +# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------ +# --syntax-check and --list-hosts open NO connection to any node: they resolve the +# playbook, roles, collections and inventory only. Running them is the difference between +# "the chain is written down" and "the chain resolves". +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it") +else: + # cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory) + # is the one in force — the same resolution a real run gets. + def ansible(*args) -> subprocess.CompletedProcess: + return subprocess.run(["ansible-playbook", *args], cwd=ANS, + capture_output=True, text=True) + + syn = ansible("--syntax-check", "playbooks/site.yml") + syn_out = (syn.stderr or "") + (syn.stdout or "") + lh = None + if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out): + # NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts " + "were not driven; every parse assertion above still ran") + else: + check(syn.returncode == 0, + "`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain " + f"does not even load: {syn_out.strip()[:400]}") + lh = ansible("--list-hosts", "playbooks/site.yml") + check(lh.returncode == 0, + "`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not " + f"resolve against the tracked inventory: " + f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}") + if lh is not None and lh.returncode == 0: + # Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values. + plays: list[tuple[str, set[str]]] = [] + name, hosts, in_hosts = None, set(), False + for line in lh.stdout.splitlines(): + m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line) + if m: + if name is not None: + plays.append((name, hosts)) + name, hosts, in_hosts = m.group(1), set(), False + continue + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if in_hosts and line.strip() and not line.strip().startswith("pattern:"): + hosts.add(line.strip()) + if name is not None: + plays.append((name, hosts)) + check(len(plays) >= 4, + f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand") + # Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three + # nodes means the third was finished by hand. + STAGES = { + "baseline": r"baseline", + "cluster-join": r"^k3s (primary|secondar)", + "cni": r"cilium", + "manage (flux hand-off)": r"flux", + } + for stage, rx in STAGES.items(): + covered: set[str] = set() + matched = 0 + for pname, phosts in plays: + if re.search(rx, pname, re.I): + matched += 1 + covered |= phosts + check(matched >= 1, f"no play in site.yml implements the {stage} stage") + check(covered == dev_hosts, + f"the {stage} stage targets {sorted(covered)}, not every dev node " + f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned " + "node (R-ENV-01)") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 30, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; " + f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), " + f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path " + "clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT " + "proven here (see the header's gap list; rebuild is L3 node-loss).") +PY diff --git a/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml new file mode 100644 index 0000000..7870a99 --- /dev/null +++ b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml @@ -0,0 +1,94 @@ +# ============================================================================= +# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by +# routing rules ... realized as two paths — the local `internet` path and the +# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress +# routing: two paths, one physical WAN": "the `internet` path is simply the default +# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and +# the nodes themselves — egress the local/private WAN by default and **never** the +# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the +# kill-switch blackholes **only** that class." +# +# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here) +# ------------------------------------------------------------------------ +# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class +# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and +# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES +# that select among them, which nothing else asserts: +# * exactly TWO paths, the same closed set in the schema, the admission policy and the +# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope); +# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one +# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one +# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because +# absence of a matching rule IS the default path; +# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a +# vpn-class pod's cluster/LAN traffic still takes the local path; +# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing +# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored; +# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every +# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN. +# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.) +# +# WHY L0: every clause above is a statement about ALL rules — including rules that must be +# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default +# route in the main table). A cluster can show which path two pods took; it cannot show that +# no third rule exists or that the vocabulary is closed. +# +# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are +# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod +# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked +# against its own values. +# +# GIVEN the repository as git tracks it, +# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy +# routing / OpenVPN / nftables templates, and every environment's overlay addresses are +# parsed and cross-checked, +# THEN there are exactly two paths with `internet` as the default; exactly one steering rule +# selects exactly the vpn class by label; the gateway is selected by node label with a +# named egress interface; all RFC1918 space and every address any environment assigns +# itself is excluded from the tunnel; the tunnel's default route and its fail-closed +# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN +# cannot take the main table; and no nftables DROP is unscoped. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this +# Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by +# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled +# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test. +# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates +# that generate them; a hand-added rule on the box is invisible here. +# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no +# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default". +# * IPv6 egress. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-paths-selected-by-rules + labels: + conformance.home.k3s/R-NET-04: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-04" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the egress vocabulary, every steering rule and the gateway + node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers + exactly two egress paths with the WAN as the default, exactly one rule selects exactly + the vpn class, local destinations stay on the local path in every environment, the + tunnel never becomes the default route, and nothing unscoped can blackhole the default + path. +spec: + steps: + - name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh b/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh new file mode 100755 index 0000000..87986c1 --- /dev/null +++ b/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh @@ -0,0 +1,513 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — egress-paths-selected-by-rules +# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing +# rules. Per OQ-1 the environment has **one physical commercial WAN and no +# multi-ISP plans**, so this is realized as two paths — the local `internet` path and +# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP +# uplinks) is **out of scope** unless a future ADR reopens it." +# — spec docs/00-vision-and-scope.md). +# +# The elaborating spec text this check quotes and asserts against — docs/03 +# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)": +# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge +# router -> internet | +# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN +# egress gateway (ADR-0013) | +# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing +# to policy-route across; the `internet` path is simply the default route out." +# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes +# themselves — egress the local/private WAN by default and **never** the VPN. Only +# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch +# blackholes **only** that class." +# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default +# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN — +# only what Cilium points at `tun0` uses the VPN." +# +# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST +# --------------------------------------------------------------------------------- +# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it, +# and the class can be flipped via GitOps. Those are proven by +# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class +# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod). +# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label. +# +# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its +# distinctive content — none of which any existing test asserts — is: +# +# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is +# one closed set of two values agreed by the schema, the admission policy and the +# renderer. A third value appearing in one of them but not the others would be a path +# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third +# class would contradict OQ-1's resolution. +# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn` +# path is entered only via rules that select the vpn class — a Cilium +# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped +# to the tunnel source address. Nothing selects the `internet` class, because +# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS +# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN. +# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn +# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class +# pod's cluster/LAN traffic still takes the local path at the same time as its +# internet traffic takes the tunnel. +# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route +# lives in a DEDICATED routing table reached only through the scoped rule, the +# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN +# from touching the main table. Otherwise there would be one path (the VPN), not two. +# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the +# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified +# by the vpn class's own source set / fwmark. The `internet` path therefore fails +# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails +# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check +# asserts the OTHER direction, which that test does not.) +# +# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration: +# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting +# exactly one class"), and scoping ("no unqualified drop, no default route in the main +# table") are statements about ALL rules, including the ones that would have to be absent. +# A running cluster can show you which path two pods took; it cannot show you that no third +# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural +# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in +# tests/l2-behavioral/egress-default-path (same requirement label). +# +# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must +# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay +# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different +# LAN is checked against its own addresses, and the check keeps working when they change. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled +# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled +# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default +# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3. +# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that +# generate them, driven by no node. A hand-added `ip rule` on the gateway node is +# invisible here (and to every other test); drift on the box is an Ansible check-mode +# exercise for the operator. +# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no +# honest repository signal that would distinguish "one WAN, correctly" from "one WAN, +# because nobody tried" — and inventing one would be a hollow assertion. +# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform. +# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and +# this check would NOT catch its absence. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 +LBL = "homelab.keithjackson.dev" +CLASS_LABEL = f"{LBL}/egress-class" + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def die(): + print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} " + "assertions failed", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +def docs(path: Path): + """Every YAML document in a file, skipping empties.""" + try: + return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)] + except yaml.YAMLError: + return [] + + +TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml") + if ".git" not in p.parts and "_artifacts" not in p.parts) +SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json" +RENDER = ROOT / "catalog" / "render.py" +VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway" +CLUSTERS = ROOT / "clusters" + +for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS): + check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be " + "read at all (R-NET-04)") +if FAILURES: + die() + + +# ======================================================= (1) EXACTLY TWO PATHS, AGREED +import json # noqa: E402 — only needed for the schema + +schema = json.loads(SCHEMA.read_text()) +egress_prop = ((schema.get("properties") or {}).get("egress") or {}) +enum = egress_prop.get("enum") +check(isinstance(enum, list) and len(enum) == 2, + f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is " + "realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path " + "nothing routes; one value is not 'multiple egress paths'") +PATHS = set(enum or []) +check(PATHS == {"internet", "vpn"}, + f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as " + "`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)") +check(egress_prop.get("default") == "internet", + f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: " + "'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply " + "the default route out'. A service that says nothing must get the WAN") + +# The admission policy must constrain the pod label to the SAME closed set, or a workload +# could carry a class the schema never allowed (and that no rule routes). +pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml")) +check(len(pol_files) >= 1, + "no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod " + "carrying an egress class outside the declared path set") +pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files) +for cls in sorted(PATHS): + check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None, + f"the egress admission policy never names the {cls!r} path — a class the policy " + "does not know cannot be validated against the path set") +# No tracked object may carry a class outside the closed set — with ONE self-policing +# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the +# admission test can assert the policy REJECTS it; that pod is exempt only while a +# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the +# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist. +def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool: + tests_root = fixture.parent + while tests_root != ROOT and tests_root.name != "tests": + tests_root = tests_root.parent + if tests_root == ROOT: + return False + for tf in sorted(tests_root.rglob("kyverno-test.yaml")): + for d in docs(tf): + for case in (d.get("results") or []): + if case.get("result") == "fail" and resource_name in (case.get("resources") or []): + return True + return False + + +stray: list[str] = [] +_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)") +for f in TRACKED_YAML: + for d in docs(f): + labels = ((d.get("metadata") or {}).get("labels") or {}) + cls = labels.get(CLASS_LABEL) + # Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs). + tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {} + for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)): + if value is None or value in PATHS or value == "CLASS": + continue + name = ((owner.get("metadata") or {}).get("name") or "?") + if rejected_by_a_policy_test(name, f): + continue + stray.append(f"{f.relative_to(ROOT)}:{name}={value}") + # Values written as raw text (templates, docs-as-code) are caught too. + for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))): + if value in PATHS or value == "CLASS": + continue + if any(value in s for s in stray): + continue + # Raw-text hit with no parsed owner: exempt only if EVERY object in the file that + # carries this value is a policy-test fixture proven to be rejected. + owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f) + if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value] + if owners and all(rejected_by_a_policy_test(o, f) for o in owners): + continue + stray.append(f"{f.relative_to(ROOT)}:={value}") +check(not stray, + f"tracked YAML assigns egress-class value(s) outside the declared path set " + f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission " + "policy allows is a workload on no path at all (the only exemption is a Kyverno " + "policy-test fixture that a kyverno-test.yaml still expects to FAIL)") + +# The renderer must project the DECLARED class onto the pod label, unmodified: that label is +# what the routing rule selects on, so a renderer that rewrote it would break the coupling +# between the declaration and the path. +rtext = RENDER.read_text() +check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None + or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None, + "catalog/render.py does not project the ServiceDefinition's own `egress` value onto the " + f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be " + "two independent facts that could disagree") + + +# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT +policies: list[tuple[Path, dict]] = [] +for p in TRACKED_YAML: + for d in docs(p): + if d.get("kind") == "CiliumEgressGatewayPolicy": + policies.append((p, d)) +check(len(policies) == 1, + f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored " + f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE " + "rule. Two rules can overlap, and which one wins for an overlapping pod is not " + "something the repository states") + +for path, pol in policies: + rel = str(path.relative_to(ROOT)) + spec = pol.get("spec") or {} + selectors = spec.get("selectors") or [] + check(len(selectors) == 1, + f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional " + "selector widens the set of pods steered off the default path, and R-NET-04's " + "selection must be exactly 'the vpn class'") + for sel in selectors: + pod_sel = sel.get("podSelector") + check(isinstance(pod_sel, dict) and pod_sel, + f"{rel}: a selector has no podSelector — an empty selector in Cilium matches " + "EVERY pod, which would put the whole cluster on the VPN path and leave no " + "default path at all") + ml = (pod_sel or {}).get("matchLabels") or {} + me = (pod_sel or {}).get("matchExpressions") or [] + check(not me, + f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a " + "single label equality; expressions (In/NotIn/Exists) can select pods that " + "never declared a class, so the rule set would no longer be disjoint from the " + "default path") + check(set(ml) == {CLASS_LABEL}, + f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly " + f"{CLASS_LABEL}. Any other key makes path selection depend on something other " + "than the declared class (and a namespace/app key would silently exempt pods)") + check(ml.get(CLASS_LABEL) == "vpn", + f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only " + "the `vpn` class may be steered: docs/03 — 'Only services labeled " + "egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would " + "route the default path through the tunnel, i.e. leave one path, not two") + + # The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a + # rule, so the gateway can move (or gain a standby) with no rule rewrite. + gw = spec.get("egressGateway") or {} + node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {} + check(bool(node_sel), + f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a " + "named host means the vpn path cannot fail over (docs/03 requires two gateway " + "nodes + Egress Gateway HA)") + check("nodeName" not in gw, + f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by " + "label so the path survives losing that node") + check(isinstance(gw.get("interface"), str) and gw.get("interface"), + f"{rel}: the egressGateway names no egress interface — without it the steered class " + "is not actually sent down a different path from the default") + + # ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION ------------- + dests = spec.get("destinationCIDRs") or [] + excl = spec.get("excludedCIDRs") or [] + check([str(c) for c in dests] == ["0.0.0.0/0"], + f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations " + "('Routed through the in-cluster VPN tunnel'), expressed as the whole internet " + "minus the exclusions below — narrowing it here would silently leave public " + "destinations on the WAN for a vpn-class service") + try: + excluded = [ipaddress.ip_network(str(c)) for c in excl] + except ValueError as e: + excluded = [] + check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})") + check(len(excluded) >= 3, + f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, " + "pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918") + + def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool: + return any(net.subnet_of(e) for e in excluded if e.version == net.version) + + # RFC1918 in full — computed, not string-matched, so a typo'd mask is caught. + for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"): + net = ipaddress.ip_network(rfc) + check(covered(net), + f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class " + "pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not " + "coexist for the same pod and east-west traffic would break") + + # Every address each ENVIRONMENT declares for itself must fall in the excluded set: that + # is what makes 'local stays local' true per environment rather than by coincidence. + def overlay_addresses() -> list[tuple[str, str]]: + found: list[tuple[str, str]] = [] + base = CLUSTERS / "base" / "cluster-vars.yaml" + data = {} + if base.is_file(): + for d in docs(base): + if d.get("kind") == "ConfigMap": + data = {k: str(v) for k, v in (d.get("data") or {}).items()} + for f in sorted(CLUSTERS.glob("*/kustomization.yaml")): + env = {**data} + for m in re.finditer( + r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P[A-Z0-9_]+)\s*\n" + r"\s*value:\s*(?P\S.*?)\s*$", f.read_text(), re.M): + env[m.group("k")] = m.group("v").strip().strip('"').strip("'") + label = f.parent.name + for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START", + "CILIUM_LB_POOL_STOP", "K3S_API_HOST"): + v = env.get(key, "") + if v and not v.startswith("TBD"): + found.append((f"{label}/{key}", v)) + return found + + addrs = overlay_addresses() + check(len(addrs) >= 5, + f"only {len(addrs)} environment address(es) could be read from the Flux overlays — " + "the 'local traffic stays local' assertion would be checking almost nothing") + for label, value in addrs: + try: + net = ipaddress.ip_network(value, strict=False) + except ValueError: + continue + check(covered(net), + f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is " + "NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment " + "would try to reach its own cluster/LAN through the VPN") + + +# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ======== +TPL = VPN_ROLE / "templates" +up = (TPL / "vpn-egress-up.sh.j2") +down = (TPL / "vpn-egress-down.sh.j2") +conf = (TPL / "vpn-egress.conf.j2") +nft = (TPL / "vpn-egress-killswitch.nft.j2") +for f in (up, down, conf, nft): + check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing " + "rules are not authored") +if FAILURES: + die() + +defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {} + + +def render(text: str) -> str: + """Substitute `{{ var }}` with the role's own default value, for every default. + + The templates reference their knobs by NAME, so an analysis that looked for the VALUE + (e.g. the nft set name) would silently match nothing and pass vacuously — which is + exactly what an earlier draft of this check did. Rendering first means the assertions + below examine the text the node actually gets. Only simple `{{ name }}` expressions are + substituted; anything else is left as-is (and the assertions are written so an + unsubstituted expression fails rather than passes). + """ + out = text + for k, v in defaults.items(): + if isinstance(v, (str, int)): + out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out) + return out + + +up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text()) +table_id = str(defaults.get("vpn_egress_gateway_route_table_id", "")) +check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"), + f"the tunnel's routing table id is {table_id!r} — the second path must live in a " + "DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and " + "putting the tunnel's default route in any of them makes the VPN the node's default " + "route instead of a selectable second path") +check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(), + "no ip-rule priority is declared for the tunnel table — 'selected by routing rules' " + "needs a rule, at a defined precedence") + +# Every `ip route ... default` the up-script installs must name the dedicated table. +route_lines = [l.strip() for l in up_t.splitlines() + if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)] +check(len(route_lines) >= 2, + f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own " + "default route via the tunnel AND the fail-closed blackhole behind it") +for l in route_lines: + check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None, + f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route " + "landing in the main table changes the path every other pod and the node itself " + "uses — the VPN would stop being opt-in") + check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None, + f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the " + "kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never " + "be blackholed") + +# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup `) sends +# everything into the tunnel table, which would leave one path instead of two. +rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)] +check(len(rule_lines) >= 1, + "the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second " + "routing table would never be consulted (R-NET-04's 'selected by routing rules')") +for l in rule_lines: + check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None, + f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector " + "EVERY packet is looked up in the tunnel table, so the WAN path disappears and the " + "VPN is no longer opt-in") + check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None, + f"an ip rule looks up a table other than the tunnel's: {l!r}") +check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None, + "the steering rule is not scoped to ${SRC}, the tunnel's own local address — that " + "address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that " + "distinguishes the steered class from every other flow on the gateway node") +check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None, + "the up-script does not refuse to run when the tunnel address is unknown — installing " + "a partial rule set is how a path ends up half-selected") + +# The provider must not be able to make the tunnel the default route either. +check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None, + "the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed " + "routes itself, in the MAIN table, and the VPN would become the node's default route " + "(docs/03: 'does **not** set the node's default route (`route-noexec` + policy " + "routing)')") +check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None, + "the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN " + "provider could redirect the node's whole default route into the tunnel, collapsing " + "the two paths into one") +check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None, + "the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the " + "default route") + + +# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN) +src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", "")) +fwmark = str(defaults.get("vpn_egress_gateway_fwmark", "")) +check(bool(src_set), + "the role declares no nftables source set — the kill-switch would have no way to name " + "'the steered class' and could only match broadly") +policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)] +check(len(policies_lines) >= 1, + "the kill-switch nft template declares no base chain — nothing would be enforced") +for l in policies_lines: + check(re.search(r"policy\s+accept\s*;", l) is not None, + f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here " + "blackholes the `internet` path too, and docs/03 requires the two classes to fail " + "in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'") +drop_lines = [l.strip() for l in nft_t.splitlines() + if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")] +check(len(drop_lines) >= 1, + "the kill-switch nft template contains no drop rule at all — there is no kill-switch, " + "and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope") +for l in drop_lines: + scoped = (src_set and src_set in l) or (fwmark and "mark" in l) + check(scoped, + f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel " + f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole " + "the default `internet` path, which must never be blackholed") + + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 35, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; " + f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), " + f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route " + f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP " + f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR " + "is L2 (egress-class for the two labelled classes, egress-default-path for the " + "unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).") +PY diff --git a/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml new file mode 100644 index 0000000..97bb685 --- /dev/null +++ b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml @@ -0,0 +1,88 @@ +# ============================================================================= +# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with +# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST +# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA, +# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)", +# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs"). +# +# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the +# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments +# from one code path, with exactly one --cluster-init founder, no external datastore, and a +# floating API endpoint that is SAN'd into the serving cert and cannot collide with the +# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live +# cluster cannot tell you what the NEXT environment brought up from this repo would look +# like. The observed half is asserted separately at L2 by +# tests/l2-behavioral/ha-control-plane, which carries the same requirement label. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down. +# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group / +# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay +# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name. +# +# GIVEN the repository as git tracks it, +# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each +# environment's Flux overlay are parsed and cross-checked against each other, +# THEN k3s is the pinned distro and nothing is installed as an agent; every environment +# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY +# ONE --cluster-init founder and every other server joining it with --server; +# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd); +# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip +# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is +# in the apiserver serving cert's SANs together with every server's own address, and +# it is neither a node address nor the ingress Gateway VIP nor inside that +# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME +# roles and plays retargeted by variables only. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this +# Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating, +# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane. +# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves) +# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss +# drains a node for R-STATE-02's data claim and asserts nothing about the control plane. +# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of +# infra/PROD-PROVISIONING.md). +# +# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible): +# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both +# environments rather than the floating VIP, contradicting the prod overlay's own comment. +# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06). +# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity] +# contract and per environment. +# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and +# the order is asserted — not as a tidiness preference but because the CNI's endpoint is +# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its +# API server. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: ha-control-plane-as-code + labels: + conformance.home.k3s/R-PLAT-01: "true" + annotations: + conformance.home.k3s/requirements: "R-PLAT-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each + environment's Flux overlay are parsed and cross-checked, then every environment is + authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one + embedded-etcd founder, no external datastore, and a leader-elected control-plane-only + VIP that is in the serving cert's SANs and collides with no other allocator — built by + one code path for both environments. +spec: + steps: + - name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/ha-control-plane-as-code/check.sh b/conformance/tests/l0-static/ha-control-plane-as-code/check.sh new file mode 100755 index 0000000..03e44f1 --- /dev/null +++ b/conformance/tests/l0-static/ha-control-plane-as-code/check.sh @@ -0,0 +1,729 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — ha-control-plane-as-code +# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade +# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the +# same multi-node topology." — spec docs/00-vision-and-scope.md). +# +# The elaborating spec text this check quotes and asserts against: +# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)" +# docs/02-environments-and-provisioning.md, table: +# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd +# (k3s server on all 3)" (the SAME cell for dev/test and prod) +# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes" +# docs/02 §"k3s topology decisions": +# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)." +# "Nodes are both control-plane and workload (3-node converged cluster); with only 3 +# nodes we do not isolate control-plane nodes." +# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs" +# +# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives) +# --------------------------------------------------------------------- +# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers: +# +# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control +# plane in BOTH environments, from ONE code path": provable with no cluster, and +# ONLY provable here. A live cluster cannot tell you whether the *next* environment +# brought up from this repo will also be HA, whether a second `--cluster-init` +# founder could exist, or whether an external datastore is configured anywhere. That +# is this check. +# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all +# etcd members, with the API answering through the floating VIP": that is cluster +# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label). +# +# Neither half is sufficient alone, and this file claims only the first. +# +# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod" +# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the +# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every +# group_vars//vars.yml that overrides them — and identifies WHICH environment is +# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by +# its group name. Add a fourth prod node, rename a group, or add a third environment and +# the assertions follow; nothing here restates an address, a hostname or a node count. +# +# GIVEN the repository as git tracks it, +# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's +# Flux overlay are parsed and cross-checked against each other, +# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is +# installed as an `agent` (an agent is not a control-plane member); +# SERVERS: every environment's server group resolves to a quorum-capable, ODD +# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to +# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while +# every other server joins it with `--server` (so a second founder — the +# split-brain shape — cannot be authored); +# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no +# `--disable-etcd` anywhere in the provisioning code, so every server is +# an etcd member (docs/02's "no external datastore", "on all 3"); +# ENDPOINT: the API has a floating endpoint that survives losing the founder — the +# kube-vip VIP is control-plane-only (never Service LB, or it would fight +# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and +# baked into the apiserver serving cert's SAN list ALONG WITH every +# server's own address (so TLS validates whichever node answers); +# ADDRESSES: each environment's control-plane VIP is not one of its node addresses, +# is not that environment's ingress Gateway VIP, and lies OUTSIDE that +# environment's Cilium LoadBalancer pool — the two allocators can never be +# handed the same address; +# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The +# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only +# the cold `--server` join, k3s_apiserver_endpoint derives from +# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's +# K3S_API_HOST equals that environment's VIP and is none of its node +# addresses ([parity], both halves), the HelmRelease keeps a per-environment +# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and +# Cilium with roles/kube_vip blocking until it answers; +# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays, +# retargeted by variables only (docs/02 "the *same* Ansible bootstrap +# runs"), so "exercise the same multi-node topology" is a property of the +# code rather than a coincidence of two hand-written inventories. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with +# a healthy quorum, that kube-vip is actually floating the VIP, and that the API +# answers through it. All of that is asserted at L2 by +# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test. +# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds, +# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing +# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's +# data-availability claim and does NOT assert control-plane survival. A dedicated +# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and +# is NOT claimed here or at L2. +# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged +# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator +# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This +# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict. +# - IPv6 / dual-stack, and any claim about the edge network. +# +# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06): +# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST` +# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each +# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod +# overlay's own comment. Making the check fail over it would have turned the merge gate +# red on a design question this workstream may not decide (R-TEST-06), so it was reported +# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a +# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE +# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there +# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at. +# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline -> +# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation +# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now +# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an +# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The +# ordering is no longer a preference about tidiness, it is a dependency of the control +# plane coming up at all. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +def die(): + print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +ANS = ROOT / "infra" / "ansible" +INV = ANS / "inventory" / "hosts.ini" +GV = ANS / "inventory" / "group_vars" +K3S_ROLE = ANS / "roles" / "k3s_server" +VIP_ROLE = ANS / "roles" / "kube_vip" +PLAYBOOKS = ANS / "playbooks" +CLUSTERS = ROOT / "clusters" + +for p, why in ((INV, "the inventory that defines the node set"), + (K3S_ROLE, "the role that installs the k3s control plane"), + (VIP_ROLE, "the role that floats the control-plane VIP"), + (CLUSTERS, "the Flux overlays that identify each environment")): + check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA " + "control plane is authored at all (R-PLAT-01)") +if FAILURES: + die() + +K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml"))) +VIP_TEXT = "\n".join(p.read_text(errors="replace") + for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2")))) + + +# ===================================================================== inventory parse +# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a +# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]` +# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'. +ini_hosts: dict[str, list[str]] = {} +ini_children: dict[str, list[str]] = {} +host_vars: dict[str, dict[str, str]] = {} +cur = None +for line in INV.read_text().splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + continue + if cur is None or cur.endswith(":vars"): + continue + host = s.split()[0] + if cur.endswith(":children"): + ini_children.setdefault(cur.split(":")[0], []).append(host) + else: + ini_hosts.setdefault(cur, []).append(host) + hv = host_vars.setdefault(host, {}) + for kv in s.split()[1:]: + if "=" in kv: + k, v = kv.split("=", 1) + hv[k] = v + + +def hosts_in(group: str, seen=None) -> set[str]: + seen = seen or set() + if group in seen: + return set() + seen.add(group) + out = set(ini_hosts.get(group, [])) + for child in ini_children.get(group, []): + out |= hosts_in(child, seen) + return out + + +def load_yaml(p: Path) -> dict: + try: + d = yaml.safe_load(p.read_text()) or {} + except yaml.YAMLError: + return {} + return d if isinstance(d, dict) else {} + + +ALL_VARS = load_yaml(GV / "all.yml") + + +# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM +# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds +# the default pair; every group_vars//vars.yml that overrides `k3s_primary_group` is +# another environment. Nothing below knows the words "dev" or "prod" as group names. +class Env: + def __init__(self, label, vars_, path): + self.label = label + self.vars = vars_ + self.path = path + + def var(self, name, default=None): + # group-scoped value wins, else the cluster-wide default from all.yml. + if name in self.vars: + return self.vars[name] + return ALL_VARS.get(name, default) + + +envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")] +for vf in sorted(GV.glob("*/vars.yml")): + v = load_yaml(vf) + if "k3s_primary_group" in v or "k3s_servers_group" in v: + envs.append(Env(str(vf.relative_to(ROOT)), v, vf)) + +check(len(envs) >= 2, + "only ONE environment is derivable from the inventory's R-ENV-02 seam " + "(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires " + "dev/test to exercise the same multi-node topology, so two must be authored") + +# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay. +# roles/flux_bootstrap's own default supplies the value for an environment that does not +# override it, so this is read from code, never guessed from a group name. +fb_default = None +fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml" +if fb_defaults_file.is_file(): + fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path") +check(isinstance(fb_default, str) and fb_default.startswith("clusters/"), + "roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from " + "an inventory environment to its Flux overlay is not expressed in code, so this check " + "cannot tell which environment is prod") + +prod_envs = [] +for e in envs: + overlay = e.var("flux_bootstrap_env_path", fb_default) + e.overlay = overlay if isinstance(overlay, str) else "" + if e.overlay.rstrip("/").endswith("/prod"): + prod_envs.append(e) +check(len(prod_envs) == 1, + f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay " + f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in " + "R-PLAT-01 has no unambiguous referent") + + +# ============================================== overlay knob values (per environment) +# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map; +# each clusters//kustomization.yaml overrides values with JSON6902 patches written as +# - op: replace +# path: /data/ +# value: "" +# (verified shape across both overlays). A key not patched keeps the base value. +BASE_VARS: dict[str, str] = {} +base_cm = CLUSTERS / "base" / "cluster-vars.yaml" +if base_cm.is_file(): + d = load_yaml(base_cm) + BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()} +check(len(BASE_VARS) > 5, + "clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface " + "cannot be read, so the VIP/pool separation cannot be checked") + +_PATCH_RE = re.compile( + r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P[A-Z0-9_]+)\s*\n\s*value:\s*(?P\S.*?)\s*$", + re.M) + + +def overlay_vars(overlay: str) -> dict[str, str]: + out = dict(BASE_VARS) + f = ROOT / overlay / "kustomization.yaml" + if not f.is_file(): + return out + for m in _PATCH_RE.finditer(f.read_text()): + out[m.group("k")] = m.group("v").strip().strip('"').strip("'") + return out + + +# ============================================================ PER-ENVIRONMENT TOPOLOGY +def as_ip(s): + try: + return ipaddress.ip_address(str(s).strip()) + except ValueError: + return None + + +for e in envs: + L = e.label + pg = e.var("k3s_primary_group") + sg = e.var("k3s_servers_group") + check(isinstance(pg, str) and isinstance(sg, str) and pg and sg, + f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot " + "resolve 'my environment's primary/servers' and would only ever build dev") + if not (isinstance(pg, str) and isinstance(sg, str)): + continue + primaries = hosts_in(pg) + servers = hosts_in(sg) + + # --- SERVERS: quorum-capable, one founder, prod exactly three --------------------- + check(len(primaries) == 1, + f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run " + "`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no " + "cluster at all") + check(primaries and primaries <= servers, + f"{L}: the primary {sorted(primaries)} is not a member of the servers group " + f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or " + "it is not part of the HA control plane it founds") + n = len(servers) + check(n >= 3, + f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes " + "'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd " + "cluster of fewer than 3 members tolerates ZERO losses — it is not HA") + check(n % 2 == 1, + f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd " + f"member count adds a member without adding fault tolerance and introduces a tie: " + f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}") + if n: + tolerance = n - (n // 2 + 1) + check(tolerance >= 1, + f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate " + f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one") + if e in prod_envs: + check(n == 3, + f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: " + "'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'") + else: + check(n >= 2, + f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer " + "nodes than prod but requires dev/test to 'exercise the same multi-node " + "topology' — one node exercises no topology") + + # --- ADDRESSES: the VIP can collide with nothing ---------------------------------- + vip_raw = e.var("kube_vip_address") + vip = as_ip(vip_raw) + check(vip is not None, + f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has " + "no floating endpoint, so losing the founder takes the API endpoint with it") + node_ips = {} + for h in sorted(servers): + ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", "")) + check(ip is not None, + f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join " + "target and the serving-cert SAN list all derive from it") + if ip is not None: + node_ips[h] = ip + if vip is not None: + check(vip not in set(node_ips.values()), + f"{L}: kube_vip_address {vip} is ALSO a node address " + f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a " + "node's own address cannot float off that node") + ov = overlay_vars(getattr(e, "overlay", "")) + gw = as_ip(ov.get("GATEWAY_VIP", "")) + lo = as_ip(ov.get("CILIUM_LB_POOL_START", "")) + hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", "")) + check(gw is not None and lo is not None and hi is not None, + f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + " + "CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and " + "the Service-LB allocator cannot be checked") + if gw is not None: + check(vip != gw, + f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME " + f"address ({vip}) — kube-vip and Cilium would both ARP for it") + if lo is not None and hi is not None: + check(not (lo <= vip <= hi), + f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium " + f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a " + "Service, and the two allocators would fight over ARP for it") + check(lo <= hi, + f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be " + "used to prove the VIP is outside it") + +# ==================================================================== DISTRO: k3s only +check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M) + is not None, + "roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane " + "version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)") +check("get.k3s.io" in K3S_TEXT, + "roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the " + "distro (ADR-0001) and no other distro may stand in for it") +check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None, + "roles/k3s_server never composes the `server` subcommand — without it k3s installs an " + "AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')") +for bad, why in ( + (r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 " + "nodes (no external datastore)')"), + (r"datastore-endpoint\s*:", "an external datastore via config.yaml"), + (r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not " + "an etcd member (docs/02: etcd 'across all 3 nodes')"), + (r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"), +): + hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*")) + if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini") + and re.search(bad, p.read_text(errors="replace"))] + check(not hits, + f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA " + f"mode, and {bad!r} contradicts it") + +# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH +check(re.search(r"--cluster-init", K3S_TEXT) is not None, + "roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never " + "founded, so there is no HA datastore to join") +init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'", + K3S_TEXT) +check(init_when is not None, + "the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an " + "ungated founder flag would re-found the cluster on every server (split brain)") +join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT) +check(join is not None, + "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` " + "— without a join target the servers are separate single-node clusters") +join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'", + K3S_TEXT, re.S) +check(join_when is not None, + "the join install task is not gated on k3s_server_mode == 'secondary' — the founder " + "would also try to join itself") +k3s_play = PLAYBOOKS / "k3s.yml" +check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint") +if k3s_play.is_file(): + play_txt = k3s_play.read_text() + modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt) + check(sorted(modes) == ["primary", "secondary"], + f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and " + "one secondary play must exist, or the founder/joiner split is not authored") + check(re.search(r"serial:\s*1", play_txt) is not None, + "playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — " + "converging two of three servers at once drops embedded etcd below quorum, which " + "is exactly the outage an HA control plane exists to avoid") + hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M) + check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns), + f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal " + "group name means this play can only ever build ONE environment, so prod cannot " + "be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')") + +# =========================================== SERVING-CERT SANs: VIP + every server +sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", "")) +check("kube_vip_address" in sans, + "roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's " + "serving cert would carry no SAN for the floating VIP, so every client reaching the " + "API through it fails TLS validation and the HA endpoint is unusable") +check("k3s_servers_group" in sans and "k3s_node_ip" in sans, + "k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the " + "cert must be valid for WHICHEVER server answers behind the VIP, and per-environment " + "resolution of that set is what keeps one role serving both environments (R-ENV-02)") +check(re.search(r"tls-san:", K3S_TEXT) is not None, + "the SAN list is passed only as an install flag and never persisted to " + "/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a " + "no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert") + +# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06) +# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's +# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control +# plane the CNI's control-plane path depended on one machine. Measured on live prod +# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 +# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the +# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or +# policy update lands at all. +# +# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs +# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what +# the configuration in git WOULD build, in BOTH environments, including the next one — a live +# cluster can only tell you about itself. +# +# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting +# "Cilium uses the HA address" requires an HA address to name; before the split there was +# only k3s_api_host, which had to be the founder for the cold join. So the assertions below +# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and +# the Flux half agrees with the Ansible half per environment. +CIL_ROLE = ANS / "roles" / "cilium" +cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2" +check(cil_values_tpl.is_file(), + "roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the " + "API endpoint its kube-proxy replacement dials, are not authored") +if cil_values_tpl.is_file(): + # Parse the rendered-shape template, never its prose: strip comment lines first, exactly + # as the kube-vip parser above does, so a reassuring comment cannot satisfy the check. + cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines() + if not l.lstrip().startswith("#")) + m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M) + check(m is not None, + "roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled " + "(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up") + if m is not None: + host_expr = m.group(1).strip() + check("k3s_apiserver_endpoint" in host_expr, + f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint " + "variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued " + "expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's " + "control-plane path depend on ONE server of an HA control plane, and the failure " + "is silent: the datapath keeps running from Cilium's cache while no service or " + "policy update ever lands (measured on prod 2026-09-06)") + check("k3s_api_host" not in host_expr, + f"roles/cilium renders k8sServiceHost={host_expr!r}, which references " + "k3s_api_host — the FOUNDER's own address. That variable exists only for the " + "cold-bootstrap `--server` join and one tls-san entry; it is a single point of " + "failure by construction and must never be a steady-state endpoint") + +# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather +# than restated: an environment overrides one line (kube_vip_address) and both the floating +# address and the CNI's endpoint move together. A literal here could disagree with the VIP the +# same environment actually floats, which is the two-places-one-fact bug this repo keeps +# hitting. +endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", "")) +check("kube_vip_address" in endpoint_def, + f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive " + "from kube_vip_address, so that overriding the VIP for an environment also moves what " + "that environment's CNI dials. A restated literal can silently disagree with the VIP " + "kube-vip is actually floating") +check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")), + "group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API " + "addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD " + "build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable") + +# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux +# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the +# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree, +# Flux and the bootstrap describe different control-plane paths and fight over the CNI. +for e in envs: + L = e.label + ov = overlay_vars(getattr(e, "overlay", "")) + k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", "")) + vip = as_ip(e.var("kube_vip_address")) + check(k3s_api_host_knob is not None, + f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that " + "token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and " + "kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one " + "fails the whole platform Kustomization") + if k3s_api_host_knob is not None and vip is not None: + check(k3s_api_host_knob == vip, + f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} " + f"but this environment's control-plane VIP (kube_vip_address, which is what " + f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve " + f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API " + "endpoints into the same CNI and fight over it. If they differ because the " + "overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of " + "an HA control plane must not dial one server") + check(k3s_api_host_knob not in set( + as_ip(host_vars.get(h, {}).get("k3s_node_ip", "")) + for h in hosts_in(str(e.var("k3s_servers_group")))), + f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this " + "environment's own NODE addresses. Cilium's kube-proxy replacement would dial a " + "single machine on a control plane built specifically to survive losing one") + +# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease +# would ship one environment's control-plane address to every environment (R-ENV-02), and an +# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config +# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent +# being replaced could translate: deadlocked on 2 of 3 nodes). +hr = ROOT / "platform" / "cilium" / "helmrelease.yaml" +check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI " + "config after the bootstrap, so the parity contract has only one side") +if hr.is_file(): + hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines() + if not l.lstrip().startswith("#")) + hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M) + check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}", + f"platform/cilium/helmrelease.yaml sets k8sServiceHost=" + f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token " + "\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane " + "address to every environment (R-ENV-02), and an empty value is the 2026-09-01 " + "incident verbatim") + +# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the +# one-command bring-up, so this is a property of the authored sequence, not of an operator's +# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with +# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.) +site_play = PLAYBOOKS / "site.yml" +check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up " + "whose ORDER can be checked") +if site_play.is_file(): + imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text()) + for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"): + check(needed in imports, + f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's " + "k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the " + "CNI, not an optional extra step — leaving it out of the bring-up means a cold " + "build installs a CNI pointed at an address nothing announces") + if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")): + check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"), + f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> " + "kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an " + "unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server " + "on a cold cluster); kube-vip before k3s cannot work at all, because the VIP " + "ships as a k3s auto-deploy manifest and needs a cluster to apply it") +check(re.search(r"wait_for", VIP_TEXT) is not None + and re.search(r"kube_vip_address", VIP_TEXT) is not None, + "roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the " + "same as having a floating endpoint (k3s still has to apply it, the image still has to " + "pull, leader election still has to settle). Since the CNI is installed immediately " + "afterward against this exact address, an unverified VIP turns a missing endpoint into " + "unreachable cilium-agents diagnosed from the far side") + +# ============================================ FLOATING ENDPOINT: kube-vip on every server +# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose. +# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with +# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the +# thing that ships.) Parser contract: the template writes each env var as an adjacent +# `- name: X` / `value: "Y"` pair; comment lines are dropped first. +VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2")) +check(len(VIP_TEMPLATES) >= 1, + "roles/kube_vip ships no manifest template — nothing deploys the API VIP") +vip_manifest = "\n".join( + "\n".join(l for l in p.read_text(errors="replace").splitlines() + if not l.lstrip().startswith("#")) + for p in VIP_TEMPLATES) +vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?", + vip_manifest)) +check(len(vip_env) >= 6, + f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the " + "parser lost its grip on the file, so the assertions below would assert nothing") +check(vip_env.get("cp_enable") == "true", + f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane " + "mode must be ON or it floats no API VIP at all") +check(vip_env.get("svc_enable") == "false", + f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be " + "false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM " + "for the same addresses (ADR-0002 gives Service LB to Cilium)") +check(vip_env.get("vip_leaderelection") == "true", + f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} " + "— without leader election the VIP has no defined owner to fail over FROM, which is " + "the whole HA mechanism") +check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}", + f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the " + "environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on " + "prod (R-ENV-02) and would not match the serving-cert SAN list") +check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None, + "the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its " + "node, which is the opposite of a floating endpoint") +vip_play = PLAYBOOKS / "kube-vip.yml" +check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP") +if vip_play.is_file(): + vp = vip_play.read_text() + vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M) + check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0], + f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers " + "group through the same environment-scoped variable the other plays use, so the " + "manifest exists on EVERY server and the VIP survives losing any one of them") +check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None + and re.search(r"k3s_data_dir", VIP_TEXT) is not None, + "roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies " + "/server/manifests, so an environment that moves --data-dir would write the " + "manifest somewhere k3s never reads and come up with no VIP at all") + +# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS +# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs". A prod-only role or playbook would be a fork of the control-plane path. +role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()} +play_names = {p.name for p in PLAYBOOKS.glob("*.yml")} +env_words = re.compile(r"prod|dev", re.I) +forked = sorted(n for n in (role_names | play_names) + if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml")) +check(not forked, + f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by " + "the SAME code retargeted by variables (docs/02), not by a per-environment fork") +for e in envs: + if e.path.name != "vars.yml": + continue + # Every key a non-default environment sets must be a variable something already reads, + # or it is an invented knob nothing consumes (a silent no-op that looks like config). + consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*")) + if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh")) + for k in sorted(e.vars): + if k.startswith("ansible_"): + continue + check(consumers.count(k) > 1, + f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an " + "environment override that nothing reads is not a scoped override, it is dead " + "configuration (and would make prod silently inherit a dev default)") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 40, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +for n in NOTES: + print(f" note: {n}") +summary = "; ".join( + f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]=" + f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers" + for e in envs) +print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. " + "R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, " + "embedded etcd on every server with no external datastore, a leader-elected " + "control-plane-only VIP on every server that is SAN'd into the serving cert and " + "collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI " + "dials that VIP, not the founder, in both environments and on both sides of the " + "[parity] contract), from one code path for both environments. The LIVE " + "cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane " + "SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml b/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml new file mode 100644 index 0000000..1b34f35 --- /dev/null +++ b/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml @@ -0,0 +1,63 @@ +# ============================================================================= +# L0 STATIC — Keycloak client secrets are declared consistently, ordered by a REAL +# Flux edge, and healed by a reconciler whose behaviour is proven offline +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# R-IDP-03 (identity config is declarative) and R-IDP-04 (IdP secrets SOPS-encrypted) +# are both implemented through ONE mechanism that FAILS OPEN: the KeycloakRealmImport +# resolves each client's secret from a `(env:NAME)` placeholder mapped to a key of the +# SOPS-encrypted keycloak-realm-clients Secret. When the operator cannot resolve a +# placeholder it stores the LITERAL placeholder text as the client secret AND THE IMPORT +# JOB STILL REPORTS SUCCESS — then rests at Done=True / Started=False and never re-runs. +# +# Measured 2026-09-06: prod had all 12 clients holding the literal placeholder (every +# OIDC login failed with "Invalid client or Invalid client credentials"); dev had exactly +# one (oauth2-proxy), because that Secret key was added AFTER dev's import completed. +# Declarative config that silently does not apply is not declarative config. +# +# GIVEN the repository as authored, +# THEN every confidential client declares a placeholder, every placeholder is defined, +# resolves to the keycloak-realm-clients Secret, and names a key that EXISTS in the +# SOPS file under the `-client-secret` contract the reconciler derives +# from; public clients declare no secret; the Flux Kustomization applying the Secret +# is a transitive `dependsOn` ancestor (wait: true) of the ones applying the import +# and the reconciler — a real edge, not a comment; the reconciler is reconciled from +# BOTH cluster entrypoints, mounts only what it needs, carries no ServiceAccount +# token and contains no dollar sign; and its own `--selftest` (extracted from the +# shipped ConfigMap and executed) proves compare-then-write, idempotence, +# no-secret-logging, the endpoint allow-list and fail-closed behaviour. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML + +# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw +# Test wraps the SAME script. +# +# NOT proven here: that the LIVE realm's stored secrets match git (that is +# tests/l2-behavioral/keycloak-client-secrets-match-git), that the SOPS ciphertext +# decrypts to a usable value (key PRESENCE is offline-checkable; plaintext is not), or +# that the CronJob's schedule fires. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: keycloak-client-secret-reconciler + labels: + conformance.home.k3s/R-IDP-03: "true" + conformance.home.k3s/R-IDP-04: "true" + annotations: + conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every OIDC client's secret is declared once, resolvable, ordered behind a real Flux + dependency, and continuously reconciled — so an unresolved import placeholder can + neither be merged nor silently persist in the realm. +spec: + steps: + - name: audit-declaration-ordering-and-reconciler + description: >- + Assert the realm import, the SOPS Secret and the reconciler agree; that the Flux + ordering is a real dependsOn edge; and run the reconciler's own selftest. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh b/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh new file mode 100755 index 0000000..185e9d5 --- /dev/null +++ b/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh @@ -0,0 +1,529 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — keycloak-client-secret-reconciler +# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative." +# — spec docs/15-identity-and-access.md) +# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available." +# — spec docs/15-identity-and-access.md) +# +# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN +# --------------------------------------------------------------- +# platform/keycloak/keycloak-realm-import.yaml declares 12 confidential OIDC clients +# whose secrets are `(env:NAME)` import placeholders, mapped by spec.placeholders to +# keys of the SOPS-encrypted keycloak-realm-clients Secret. Measured on both clusters +# on 2026-09-06: +# +# * when the Keycloak Operator cannot resolve a placeholder it stores the LITERAL +# placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; +# * the KeycloakRealmImport then rests at Done=True / Started=False and NEVER +# re-runs, so it never self-heals. +# +# Consequence: on prod all 12 clients held the literal placeholder and every OIDC +# login on the cluster failed ("Invalid client or Invalid client credentials"); on dev +# exactly one (oauth2-proxy) was broken, because that Secret key was added AFTER dev's +# import had already completed and a completed import is never re-run. +# +# Nothing in the repo asserted the declaration was self-consistent, and nothing +# corrected the realm afterwards. This check is the static half of the fix (the +# reconciler CronJob is the runtime half, and this check proves that too). +# +# GIVEN the repository as authored, +# THEN (1) DECLARATION IS COMPLETE: every confidential client in the realm import +# declares a secret placeholder; every placeholder it uses is defined in +# spec.placeholders; every placeholder resolves to the keycloak-realm-clients +# Secret; and every referenced key EXISTS in the SOPS file. A client whose +# key is missing is exactly the dev failure, and it is now unmergeable. +# (2) THE NAMING CONTRACT HOLDS: each key is `-client-secret`. That is +# what lets the reconciler derive the client set from the mounted Secret with +# no second declaration — so it must be enforced, not assumed. No orphan +# placeholder, and no orphan Secret key (an orphan key would make every +# reconciler run fail with NOT FOUND). +# (3) PUBLIC CLIENTS DECLARE NO SECRET (the `kubernetes` client is PKCE-public; +# giving it a secret would be a silent downgrade). +# (4) ORDERING IS A REAL FLUX EDGE, NOT A COMMENT: the Flux Kustomization that +# renders the Secret is a transitive `dependsOn` ancestor — with `wait: true` +# — of the one that renders the realm import and the reconciler. The repo's +# hard-won rule: a dependency that is only DOCUMENTED is not a dependency. +# (5) THE RECONCILER IS WIRED AND SAFE: it is reconciled from BOTH cluster +# entrypoints, mounts exactly the two Secrets it needs at the paths its code +# defaults to, carries no ServiceAccount token, and contains no dollar sign +# (it ships inside a substituted Flux path, where an unescaped dollar-brace +# expansion is rewritten before any container runs). +# (6) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the +# script from the ConfigMap and executes its `--selftest`, which drives the +# real code paths against an in-process stub Keycloak and asserts +# compare-then-write, idempotence (zero writes on a converged realm), +# no-secret-logging, the endpoint allow-list (users/groups/roles are +# unreachable) and fail-closed behaviour. What CI proves is byte-for-byte +# what ships. +# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh +# blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps +# platform/keycloak/ to these requirement IDs, and an L2 test exists that +# audits the LIVE realm. An unreferenced test is precisely the "looks applied +# and isn't" failure this repo keeps hitting. +# +# NOT proven here (named honestly, in the style of the sibling L0 checks): +# - that the LIVE realm's stored secrets match git. That is cluster state; it is +# conformance/tests/l2-behavioral/keycloak-client-secrets-match-git. +# - that the SOPS ciphertext decrypts to a usable secret. Key PRESENCE is checkable +# offline; the plaintext is not, and must not be. +# - that the CronJob's image can be pulled or its schedule fires. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml" +SOPS_FILE="platform/keycloak/keycloak-realm-clients.sops.yaml" +RECON_FILE="platform/keycloak/client-secret-reconciler.yaml" + +# Honest SKIP (never green) if the thing under test is absent entirely. +for f in "$IMPORT_FILE" "$SOPS_FILE" "$RECON_FILE"; do + if [ ! -f "${ROOT}/${f}" ]; then + echo "check SKIPPED: ${f} absent — nothing to audit" >&2 + exit 3 + fi +done + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit of the Keycloak client-secret declaration, its Flux ordering, and the +reconciler that heals it. PyYAML (the same dependency tests/l0-static/no-plaintext-secrets +and least-privilege-rbac already take) parses the manifests; lib/status.py is IMPORTED for +the Flux graph walk rather than forked, so there is one definition of "what a Flux path +renders" in the harness.""" +from __future__ import annotations + +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree +sys.path.insert(0, str(ROOT / "conformance" / "lib")) +import status as st # noqa: E402 — reuse the walk, don't fork it + +IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml" +SOPS_FILE = "platform/keycloak/keycloak-realm-clients.sops.yaml" +RECON_FILE = "platform/keycloak/client-secret-reconciler.yaml" +SECRET_NAME = "keycloak-realm-clients" +ADMIN_SECRET = "keycloak-admin-credentials" +KEY_SUFFIX = "-client-secret" +SECRETS_MOUNT = "/etc/keycloak-realm-clients" +DOLLAR = chr(36) + +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def load(rel): + return list(yaml.safe_load_all((ROOT / rel).read_text())) + + +# ================================================================= (1)-(3) declaration +realm_import = next((d for d in load(IMPORT_FILE) + if d and d.get("kind") == "KeycloakRealmImport"), None) +check(realm_import is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document found") +if realm_import is None: + print("CHECK FAIL: cannot parse the realm import", file=sys.stderr) + raise SystemExit(1) + +placeholders = realm_import["spec"].get("placeholders") or {} +clients = realm_import["spec"]["realm"].get("clients") or [] +check(len(clients) > 0, f"{IMPORT_FILE}: parsed zero clients — the file shape changed and " + f"this check would silently audit nothing") + +# The operator's placeholder syntax, spelled without a literal dollar so this file is +# itself safe to grep for one. Matches `(env:NAME)`. +PLACEHOLDER_RE = re.compile(re.escape(DOLLAR) + r"\(env:([A-Za-z_][A-Za-z0-9_]*)\)") + +used_placeholders: set[str] = set() +declared_pairs: dict[str, str] = {} # clientId -> secret key in the Secret + +for c in clients: + cid = c.get("clientId", "") + secret = c.get("secret") + public = bool(c.get("publicClient")) + if public: + # (3) a PKCE public client must not carry a secret at all. + check(secret is None, + f"{IMPORT_FILE}: client '{cid}' is publicClient but declares a secret — a " + f"public client holds no secret; declaring one is a silent downgrade") + continue + # (1) every confidential client must declare one. + check(isinstance(secret, str) and secret, + f"{IMPORT_FILE}: confidential client '{cid}' declares NO secret. The operator " + f"would generate a random one that exists nowhere in git, so nothing could " + f"ever reconcile it and the consuming app's mirrored credential would be wrong") + if not isinstance(secret, str): + continue + m = PLACEHOLDER_RE.fullmatch(secret.strip()) + check(m is not None, + f"{IMPORT_FILE}: client '{cid}' has a secret that is not an (env:NAME) " + f"placeholder. A literal secret in git violates R-IDP-04/R-SEC-01; the only " + f"permitted form is a placeholder resolved from the {SECRET_NAME} Secret") + if m is None: + continue + name = m.group(1) + used_placeholders.add(name) + # (1) the placeholder must be DEFINED. An undefined one is stored literally by the + # operator, with a successful-looking import — the whole incident. + ph = placeholders.get(name) + check(ph is not None, + f"{IMPORT_FILE}: client '{cid}' uses placeholder '{name}' but spec.placeholders " + f"does not define it. The operator stores the LITERAL placeholder as the client " + f"secret and STILL reports the import successful (measured 2026-09-06)") + if ph is None: + continue + ref = (ph or {}).get("secret") or {} + check(ref.get("name") == SECRET_NAME, + f"{IMPORT_FILE}: placeholder '{name}' reads Secret '{ref.get('name')}', not " + f"'{SECRET_NAME}'. The reconciler mounts only {SECRET_NAME}, so a second source " + f"would be declared but never healed") + key = ref.get("key") + check(isinstance(key, str) and key, + f"{IMPORT_FILE}: placeholder '{name}' names no Secret key") + if not isinstance(key, str): + continue + # (2) the naming contract the reconciler derives its client set from. + check(key == cid + KEY_SUFFIX, + f"{IMPORT_FILE}: client '{cid}' reads Secret key '{key}', but the reconciler " + f"derives the clientId from the key name, so it MUST be " + f"'{cid}{KEY_SUFFIX}'. Rename the key (and the SOPS entry) or the reconciler " + f"would look up a client that does not exist") + declared_pairs[cid] = key + +# (2) no orphan placeholder: a defined-but-unused placeholder is dead config that will +# quietly rot out of sync with the Secret. +for name in sorted(set(placeholders) - used_placeholders): + check(False, + f"{IMPORT_FILE}: spec.placeholders defines '{name}' but no client uses it — " + f"dead declaration; remove it or wire it to the client that needs it") + +# ---- the SOPS Secret: keys are cleartext in a SOPS-encrypted file; values are not. +sops_doc = next((d for d in load(SOPS_FILE) if d and d.get("kind") == "Secret"), None) +check(sops_doc is not None, f"{SOPS_FILE}: no Secret document found") +sops_keys = set((sops_doc or {}).get("stringData") or {}) +check(len(sops_keys) > 0, + f"{SOPS_FILE}: parsed zero stringData keys — the file shape changed and this check " + f"would silently audit nothing") +check((sops_doc or {}).get("metadata", {}).get("name") == SECRET_NAME, + f"{SOPS_FILE}: the Secret is not named {SECRET_NAME}") + +for cid, key in sorted(declared_pairs.items()): + # (1) THE DEV FAILURE, made unmergeable: a client declared in the import whose key + # is absent from the Secret imports "successfully" with a literal placeholder. + check(key in sops_keys, + f"{SOPS_FILE}: client '{cid}' needs key '{key}' and it is NOT in the Secret. " + f"The realm import would still report SUCCESS and store the literal placeholder " + f"as that client's secret — this is exactly how dev's oauth2-proxy broke") + +# (2) no orphan Secret key: the reconciler treats every `-client-secret` key as a +# client that MUST exist, and fails loudly if it does not. Catch it here instead. +for key in sorted(k for k in sops_keys if k.endswith(KEY_SUFFIX)): + cid = key[: -len(KEY_SUFFIX)] + check(cid in declared_pairs, + f"{SOPS_FILE}: key '{key}' has no matching client in {IMPORT_FILE}. The " + f"reconciler would look up client '{cid}', not find it, and fail every run") + +# ===================================================== (4) ordering is a real Flux edge +# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`; a doc +# is an entrypoint only if it carries the kustomize.toolkit apiVersion, `kind: +# Kustomization` and a spec-level `path:`. Overlay `patch: |-` bodies carry Kustomizations +# WITHOUT a path and are correctly skipped (they are patches, not entrypoints). +_DEPENDSON_RE = re.compile(r"^(\s*)dependsOn:\s*$", re.M) +_DEP_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)") +_WAIT_RE = re.compile(r"^\s{2,6}wait:\s*true\s*(?:#.*)?$", re.M) + + +def depends_on(doc: str) -> list[str]: + m = _DEPENDSON_RE.search(doc) + if not m: + return [] + indent = len(m.group(1)) + out: list[str] = [] + for raw in doc[m.end():].splitlines(): + if raw.strip() == "" or raw.lstrip().startswith("#"): + continue + if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"): + break # dedented out of the list + nm = _DEP_NAME_RE.match(raw) + if nm: + out.append(nm.group(1)) + return out + + +kustomizations: dict[str, dict] = {} +for yfile in sorted((ROOT / "clusters").rglob("*.yaml")): + text = yfile.read_text() + if st.FLUX_API not in text: + continue + for doc in st._DOC_SPLIT_RE.split(text): + if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M): + continue + pm = st._PATH_RE.search(doc) + if not pm: + continue + nm = st._NAME_RE.search(doc) + name = nm.group(1) if nm else "?" + kustomizations[name] = { + "file": str(yfile.relative_to(ROOT)), + "path": pm.group(1), + "dependsOn": depends_on(doc), + "wait": bool(_WAIT_RE.search(doc)), + } + +check(len(kustomizations) > 0, + "no Flux Kustomization with a spec.path found under clusters/ — the parser or the " + "repo layout changed; this check would silently audit nothing") + + +def direct_render(path_spec: str) -> set[str]: + """Files a Flux `spec.path` renders ITSELF — a kustomize walk that deliberately does + NOT expand nested Flux Kustomizations. + + lib/status.py's reconciled_files() follows every `spec.path` it discovers, which is + right for "is this file applied to the cluster at all" but wrong for OWNERSHIP: the + root Kustomization's path is clusters/, so a transitive walk makes the root the + owner of every file in the repo and the ordering assertion below becomes vacuous + (measured: it reported the root as an owner of all three keycloak files). `resources:` + parsing itself is st.kustomize_resources, not forked. + """ + rel = path_spec.lstrip("./") or "." + out: set[str] = set() + seen: set[str] = set() + + def walk(d: str) -> None: + if d in seen: + return + seen.add(d) + kfile = ROOT / d / "kustomization.yaml" + if not kfile.is_file(): + return + out.add(str(kfile.relative_to(ROOT))) + for item in st.kustomize_resources(kfile): + if "://" in item: + continue # remote base: not repo state + target = (ROOT / d / item).resolve() + try: + trel = target.relative_to(ROOT).as_posix() + except ValueError: + continue # outside the repo + if target.is_dir(): + walk(trel) + elif target.is_file(): + out.add(trel) + + if (ROOT / rel).is_dir(): + walk(rel) + return out + + +rendered_by = {n: direct_render(k["path"]) for n, k in kustomizations.items()} + + +def owners(rel_file: str) -> list[str]: + return sorted(n for n, files in rendered_by.items() if rel_file in files) + + +def ancestors(name: str, seen=None) -> set[str]: + """Transitive dependsOn closure of `name` (its dependencies, direct and indirect).""" + seen = seen if seen is not None else set() + for dep in kustomizations.get(name, {}).get("dependsOn", []): + if dep in seen: + continue + seen.add(dep) + ancestors(dep, seen) + return seen + + +secret_owners = owners(SOPS_FILE) +import_owners = owners(IMPORT_FILE) +recon_owners = owners(RECON_FILE) + +check(bool(secret_owners), + f"{SOPS_FILE} is rendered by NO Flux Kustomization — the client secrets are in git " + f"but never applied") +check(bool(import_owners), + f"{IMPORT_FILE} is rendered by NO Flux Kustomization") +# (5) the reconciler must actually be reconciled, or it is decoration. +check(bool(recon_owners), + f"{RECON_FILE} is rendered by NO Flux Kustomization — the healer would sit in git " + f"and never run, which is the 'looks applied and isn't' failure it exists to end") + +for consumer_file, consumer_owners in ((IMPORT_FILE, import_owners), + (RECON_FILE, recon_owners)): + for co in consumer_owners: + for so in secret_owners: + if so == co: + continue # same Kustomization: one atomic apply, ordering moot + anc = ancestors(co) + check(so in anc, + f"ORDERING IS ONLY DOCUMENTED: '{co}' renders {consumer_file} but does " + f"not (transitively) dependsOn '{so}', which renders the " + f"{SECRET_NAME} Secret. Without that edge the consumer can reconcile " + f"before the secret exists, and the Keycloak Operator's response to a " + f"missing placeholder is to store it LITERALLY and report success") + if so in anc: + check(kustomizations[so]["wait"], + f"'{so}' (which renders the {SECRET_NAME} Secret) does not set " + f"wait: true, so '{co}' can proceed before it is applied") + +# ------ both environments must reconcile the reconciler, not just dev. +for env in ("dev", "prod"): + entry = f"clusters/{env}" + if not (ROOT / entry).is_dir(): + continue + applied, _ = st.reconciled_files(ROOT, entry) + check(RECON_FILE in applied, + f"{RECON_FILE} is not reachable from the {entry} entrypoint — the healer would " + f"exist on one cluster only, and prod is where all 12 clients were broken") + check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}") + +# ============================================ (5) the reconciler manifest is safe + wired +recon_docs = [d for d in load(RECON_FILE) if d] +cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None) +cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None) +check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script") +check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler") + +script = "" +if cm is not None: + data = cm.get("data") or {} + check(len(data) == 1, + f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key " + f"(found {sorted(data)}) — a second copy would be untested by this gate") + script = next(iter(data.values()), "") + check(len(script.splitlines()) > 50, + f"{RECON_FILE}: the extracted script is implausibly short " + f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the " + f"selftest below would prove nothing") + # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with + # postBuild.substituteFrom, so envsubst rewrites it before any container runs. The + # script is written with NO dollar sign at all, so the hazard is structural rather + # than escaped and no future edit has to reason about escaping. + check(DOLLAR not in script, + f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside a " + f"SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at " + f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python " + f"never needs one; chr(36) spells it where a literal is genuinely required") + +if cj is not None: + pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"] + ctr = pod["containers"][0] + vols = {v["name"]: v for v in pod.get("volumes", [])} + mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])} + secret_vols = [v for v in vols.values() + if (v.get("secret") or {}).get("secretName") == SECRET_NAME] + check(len(secret_vols) == 1, + f"{RECON_FILE}: the CronJob must mount the {SECRET_NAME} Secret — that mount IS " + f"the desired state it reconciles towards") + if secret_vols: + mp = mounts.get(secret_vols[0]["name"], {}).get("mountPath") + check(mp == SECRETS_MOUNT, + f"{RECON_FILE}: {SECRET_NAME} is mounted at {mp}, but the script defaults " + f"to {SECRETS_MOUNT} and no env var overrides it") + cm_vols = [v for v in vols.values() + if (v.get("configMap") or {}).get("name") == (cm or {}).get( + "metadata", {}).get("name")] + check(len(cm_vols) == 1, + f"{RECON_FILE}: the CronJob does not mount the reconciler ConfigMap, so it " + f"would run nothing") + env_srcs = {e["name"]: e for e in ctr.get("env", [])} + for var in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"): + ref = (env_srcs.get(var, {}).get("valueFrom") or {}).get("secretKeyRef") or {} + check(ref.get("name") == ADMIN_SECRET, + f"{RECON_FILE}: {var} must come from the {ADMIN_SECRET} Secret " + f"(got {ref.get('name')!r}) — the admin credential is never inlined") + for var in ("KEYCLOAK_BASE_URL", "KEYCLOAK_REALM"): + check(env_srcs.get(var, {}).get("value"), + f"{RECON_FILE}: {var} is not set; the script fails closed without it, but " + f"the CronJob should not ship broken") + # R-SEC-02: it talks only to Keycloak's REST API and needs no Kubernetes API rights. + check(pod.get("automountServiceAccountToken") is False, + f"{RECON_FILE}: the reconciler needs NO Kubernetes API access (its inputs are " + f"mounted Secrets) — set automountServiceAccountToken: false") + check(cj["spec"].get("concurrencyPolicy") == "Forbid", + f"{RECON_FILE}: set concurrencyPolicy: Forbid so a run stalled on an " + f"unavailable Keycloak is not lapped by the next tick") + img = ctr.get("image", "") + check(re.search(r":[0-9]", img), + f"{RECON_FILE}: container image {img!r} is not pinned to an explicit version") + +# ============================== (6) the reconciler's BEHAVIOUR, proven by its own selftest +if script: + with tempfile.TemporaryDirectory() as td: + p = Path(td) / "reconcile.py" + p.write_text(script) + proc = subprocess.run([sys.executable, str(p), "--selftest"], + capture_output=True, text=True, timeout=180) + tail = (proc.stdout + proc.stderr).strip().splitlines() + check(proc.returncode == 0, + "the reconciler's --selftest FAILED (compare-then-write / idempotence / " + "no-secret-logging / endpoint allow-list / fail-closed):\n " + + "\n ".join(tail[-12:])) + if proc.returncode == 0: + print(" reconciler --selftest: " + (tail[-1] if tail else "(no output)")) + +# ======================================================= (7) this check is load-bearing +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text() + if (ROOT / "ci" / "touched-requirements.sh").is_file() else "") +check("keycloak-client-secret-reconciler/check.sh" in L0, + "ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the " + "requirement-scoped conformance selection can filter this check out of a gate run, " + "so l0.sh must invoke it directly and unconditionally or the gate is not enforcing " + "it. An unreferenced test is the failure this whole change exists to end") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") +check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M), + "ci/touched-requirements.sh has no platform/keycloak/* entry in its path->requirement " + "map, so editing the realm import or this reconciler attributes NO requirement and " + "the L2 keycloak tests are filtered out of the merge gate's touched subset") +l2 = ROOT / "conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml" +check(l2.is_file(), + "the LIVE-realm counterpart (conformance/tests/l2-behavioral/" + "keycloak-client-secrets-match-git) is missing — this static check cannot see a " + "realm that drifted after apply, which is the whole failure mode") +if l2.is_file(): + check("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in l2.read_text(), + "the L2 test does not assert the reconciler's audit sentinel, so it could pass " + "on a Job that never ran the audit") + +# ------------------------------------------------------------------------------ report +print(f"realm import: {len(clients)} client(s), {len(declared_pairs)} confidential with a " + f"declared secret, {len(placeholders)} placeholder(s)") +print(f"{SECRET_NAME} Secret: {len(sops_keys)} key(s) in git (names only; values stay " + f"SOPS-encrypted)") +print("flux ordering: " + ", ".join( + f"{f.split('/')[-1]} <- {owners(f)}" for f in (SOPS_FILE, IMPORT_FILE, RECON_FILE))) + +print() +if not FAILURES: + print(f"check OK: keycloak-client-secret-reconciler — {CHECKS} assertions " + f"(R-IDP-03, R-IDP-04)") + raise SystemExit(0) +print(f"CHECK FAIL (keycloak-client-secret-reconciler): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml new file mode 100644 index 0000000..54b6b2e --- /dev/null +++ b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml @@ -0,0 +1,68 @@ +# ============================================================================= +# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in +# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in +# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the +# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to +# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob +# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three +# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on +# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST. +# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the +# COUPLING was untested, which is what makes the class reintroducible. +# +# GIVEN the repository as authored, +# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash +# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"), +# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an +# INDEPENDENT detector is actually refused by that guard, in base and in every +# environment overlay; +# (2) no environment overlay carries a placeholder-shaped value for a knob no guard +# refuses; +# (3) every function in standup.sh that reads a guarded knob applies the guard to it +# (the "three guards died at once" regression); +# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix +# — the exact anchoring defect above — and still accepts a real endpoint; +# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently +# testing a copy if the guard stops being a self-contained predicate. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + +# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test +# wraps the SAME script. +# +# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts — +# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves +# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct; +# and that the guard runs at all in a real standup (the dispatch path is shell, executed +# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the +# authored backup target cannot be a placeholder that the guard fails to recognise. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: placeholder-guard-coupling + labels: + conformance.home.k3s/R-ENV-02: "true" + conformance.home.k3s/R-STATE-03: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every placeholder-shaped value authored in cluster-vars, for any environment + overlay, is actually recognised as a placeholder by the real bootstrap guard that + is supposed to refuse it — so a value change in one file cannot silently disarm a + guard in another and enable velero against an endpoint that does not exist. +spec: + steps: + - name: couple-authored-placeholders-to-the-real-guard + description: >- + Extract the real placeholder predicate from bootstrap/standup.sh, run it against + every authored cluster-vars value the repo actually carries, and assert the guard + refuses each one an independent detector calls a placeholder. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/placeholder-guard-coupling/check.sh b/conformance/tests/l0-static/placeholder-guard-coupling/check.sh new file mode 100755 index 0000000..d3814e6 --- /dev/null +++ b/conformance/tests/l0-static/placeholder-guard-coupling/check.sh @@ -0,0 +1,516 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — placeholder-guard-coupling +# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)." +# — spec docs/02-environments-and-provisioning.md) +# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)." +# — spec docs/04-storage-and-state.md) +# +# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE +# -------------------------------------------------------------------- +# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a +# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is +# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed +# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` +# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match — +# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped +# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname +# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and +# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident. +# +# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was +# never tested: a guard in one file keying on a value in another, with nothing tying them +# together. This check is that tie, and it is derived from BOTH real files — no current +# string is hardcoded here, so it does not rot when the placeholder or the guard changes. +# +# HOW THE REAL GUARD IS EXERCISED (and why not a copy) +# --------------------------------------------------- +# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when +# --kubeconfig/--expect-server/ are absent, `cd`s to the repo root, exports +# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster. +# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY +# those, then calls the real function. That is the real bytes, not a reimplementation: +# * every function whose name ends `_is_placeholder` is discovered, not named here; +# * a duplicate definition of one is a failure (which of the two would run?); +# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing +# else, no command substitution, no call to another function. If the guard ever grows +# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the +# remedy (move the predicate into a small sourceable library that standup.sh sources) +# rather than quietly falling back to testing a copy of the logic; +# * a CANARY pins the extraction itself: the extracted function must refuse the empty +# string and accept a realistic endpoint. An extraction that grabbed the wrong lines, +# or a predicate degraded to always-true / always-false, fails there — this repo has +# already been bitten by a check that passed when perturbed because it asked a question +# that was always satisfiable. +# There is deliberately NO second implementation of the matching logic anywhere in this file. +# +# THE INDEPENDENT DETECTOR (the other half of the comparison) +# ---------------------------------------------------------- +# Asking "does the guard agree with the guard?" proves nothing, so the set of +# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the +# UNION of two independent sources: +# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME, +# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `` templates and the empty +# string. These are NOT derived from the guard, so DELETING a guard arm cannot make a +# value stop being placeholder-shaped — it makes assertion (1) fail, which is the point. +# A value using an idiom the guard does not know is a finding either way: fix it by +# giving the knob a real value, or by teaching the guard the idiom. +# * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED. +# Stripping the anchors is what makes the comparison non-circular in the dimension that +# actually broke: the detector matches a word ANYWHERE, case-insensitively, and the +# guard must agree. A prefix-anchored guard cannot pass. +# +# GIVEN the repository as authored, +# WHEN the real predicate(s) are extracted from bootstrap/standup.sh and executed, +# THEN (1) VALUES: for every knob a guard is responsible for, every authored value — the +# base default AND each environment overlay's value — that the independent +# detector calls placeholder-shaped is REFUSED by the real guard, and every value +# it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero +# suspended forever, which is the same class of silent failure); +# (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a +# placeholder-shaped value for a knob that no guard refuses. (Base alone is +# exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/` — +# and its loud sentinels exist precisely to be overridden. A base placeholder an +# overlay forgets to override IS caught, because it becomes that overlay's +# effective value.) +# (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`, +# following one hop of variable aliasing) applies a placeholder guard to it. This +# is the "all three guards died at once" regression stated as a property: the +# guard being correct is worthless if a consumer stops calling it. +# (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a +# value carrying that word in the MIDDLE (not at the start) — the precise defect +# of 2026-09-05 — while still accepting a realistic endpoint. +# (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim +# from the file and self-contained, the canary holds, no divergent second copy of +# the matching logic exists elsewhere in standup.sh, and a floor on the number of +# assertions run keeps a pass from degenerating into a no-op. +# AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and +# ci/gate.sh sets the merge block when that stage fails. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps) +# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not +# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator- +# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore. +# - It does not execute standup.sh's dispatch path, so it cannot prove the guard is +# REACHED in a real run — only that, when reached, it refuses what it must. The +# consumer property (3) is a static approximation of reachability: it is a text-level +# taint trace over one file, not an interpreter, and a guard whose result is computed +# and then ignored would satisfy it. +# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the +# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values +# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are +# asserted by the node_baseline role itself, at play time). +# - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit +# to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so +# the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh +# load_touched_requirements) — this check runs. An edit to clusters/* maps to +# R-ENV-02, which this check carries, so a cluster-vars value change also selects it. +# A change that touches standup.sh ALONGSIDE some unrelated mapped path (say +# platform/cilium/) would scope the run away from this check; closing that needs a +# `bootstrap/*` entry in the path map, which is not this check's file to edit. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}" + +cd "${ROOT}" + +if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then + echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2 + exit 3 +fi + +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT + +# --------------------------------------------------------------------------------------- +# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files, +# and run every assertion that does not need a shell. Writes: +# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else +# ${WORK}/corpus.tsv \t\t\t +# ${WORK}/findings.txt one line per failed assertion +# ${WORK}/counts.txt +# --------------------------------------------------------------------------------------- +"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY' +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3] +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def bail(msg): + """A failure that makes the rest of the check meaningless: report and stop.""" + print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr) + (WORK / "findings.txt").write_text(msg + "\n") + (WORK / "counts.txt").write_text("0\n") + (WORK / "corpus.tsv").write_text("") + (WORK / "guards.sh").write_text("") + raise SystemExit(1) + + +TEXT = (ROOT / STANDUP_REL).read_text() +LINES = TEXT.splitlines() + +# ------------------------------------------------------------------ (5) extract the guard +DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{") +defs: dict[str, list[int]] = {} +for i, line in enumerate(LINES): + m = DEF_RE.match(line) + if m: + defs.setdefault(m.group(1), []).append(i) + +if not defs: + bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed " + "(update this check's discovery pattern in the SAME change) or it was DELETED, which " + "is the 2026-09-05 failure with the guard removed instead of disarmed.") + +# An "argument-shaped" call: "" +CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"") + +guard_src: list[str] = [] +guard_words: dict[str, list[str]] = {} +ARM_RE = re.compile(r"^(?P[^()]*?)\)\s*return\s+[01]\s*;;$") +ALLOWED = ( + re.compile(r"^$"), + re.compile(r"^#"), + re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"), + re.compile(r"^esac$"), + re.compile(r"^return\s+[01]$"), + ARM_RE, +) + +for name, at in sorted(defs.items()): + check(len(at) == 1, + f"{STANDUP_REL} defines {name}() {len(at)} times (lines " + f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and " + "this check can no longer say which one it extracted. Keep exactly one definition.") + if len(at) != 1: + continue + start = at[0] + end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None) + if end is None: + bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the " + "extraction boundary this check relies on is gone. Restore the conventional " + "layout, or move the predicate into a sourceable library and point this check at it.") + body = LINES[start:end + 1] + block = "\n".join(body) + check(block in TEXT, + f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; " + "the harness would be testing something the script does not contain") + + words: list[str] = [] + for raw in body[1:-1]: + s = raw.strip() + if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s: + bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED " + f"case/return predicate (offending line: {s!r}). This check exercises the REAL " + "function by sourcing its own lines; a body that calls out to anything else " + "cannot be extracted faithfully, and silently testing a COPY of the logic is " + "exactly the failure mode this check exists to prevent. REMEDY: move the " + "predicate into a tiny library that standup.sh sources (e.g. " + "bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do " + "NOT reimplement the matching here.") + m = ARM_RE.match(s) + if not m: + continue + for pat in m.group("pats").split("|"): + pat = pat.strip() + if pat in ('""', "''", ""): + continue # the empty-value arm: covered by the canary + words.append(pat.strip("'\"").strip("*")) + check(len(words) >= 1, + f"{name}() has no non-empty placeholder pattern left — every arm that recognises a " + "loud placeholder string has been removed, so the guard now only refuses the empty " + "value and any 'TBD'-style endpoint would sail straight through it") + guard_words[name] = words + guard_src.append(block) + +(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n") + +# ------------------------------------------------- no divergent second copy of the logic +# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means +# a second, independently-drifting implementation of the same test (which is how three +# guards came to disagree with one value in the first place). +guard_line_nums = {j for name, at in defs.items() if len(at) == 1 + for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES)) + if LINES[k] == "}") + 1)} +all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w}) +divergent: list[str] = [] +for j, line in enumerate(LINES): + if j in guard_line_nums or line.strip().startswith("#"): + continue + code = line.split("#", 1)[0] + hit = [w for w in all_words if w in code.upper()] + if hit: + divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}") +check(not divergent, + "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the " + f"predicate — a SECOND implementation of the same test, which is precisely how a value " + f"change disarmed three guards at once. Call the predicate instead. Sites: " + + " | ".join(divergent)) + +# ------------------------------------------------------------- cluster-vars, as authored +CV = ROOT / "clusters" / "base" / "cluster-vars.yaml" +if not CV.is_file(): + bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is " + "missing and this check cannot mean anything") + +base: dict[str, str] = {} +for d in yaml.safe_load_all(CV.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and (d.get("metadata") or {}).get("name") == "cluster-vars": + base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()} +check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the " + "ConfigMap parse failed, so every value assertion below would be vacuous") + +overlays: dict[str, dict[str, str]] = {} +for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + eff = dict(base) + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception as exc: # noqa: BLE001 + check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this " + "check cannot compute that overlay's effective values") + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: # noqa: BLE001 + continue + for op in ops if isinstance(ops, list) else []: + if not isinstance(op, dict): + continue + m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", ""))) + if m and str(op.get("op", "")) in ("replace", "add"): + eff[m.group(1)] = "" if op.get("value") is None else str(op["value"]) + overlays[env] = eff +check(len(overlays) >= 1, + "no clusters//kustomization.yaml overlay was parsed — assertion (2) would be vacuous") + +# ------------------------------------------------ which knobs is a guard responsible for? +# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a +# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here. +FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{") +funcs: dict[str, list[str]] = {} +i = 0 +while i < len(LINES): + m = FUNC_RE.match(LINES[i]) + if m: + end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1) + funcs[m.group(1)] = LINES[i + 1:end] + i = end + 1 + continue + i += 1 +check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the " + "consumer assertion (3) would be vacuous") + +CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"") +ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]") + +guarded_keys: set[str] = set() +consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func) +for fname, body in funcs.items(): + text = "\n".join(body) + keys_of_var: dict[str, set[str]] = {} + for m in CVAR_RE.finditer(text): + keys_of_var.setdefault(m.group(1), set()).add(m.group(2)) + # one hop of aliasing: ep="${live}" carries live's taint + for _ in range(3): + for m in ALIAS_RE.finditer(text): + src = keys_of_var.get(m.group(2)) + if src: + keys_of_var.setdefault(m.group(1), set()).update(src) + called_on = {m.group(2) for m in CALL_RE.finditer(text)} + called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on} + # Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its + # value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts. + guarded_here: dict[str, bool] = {} + for var, keys in keys_of_var.items(): + for key in keys: + guarded_here[key] = guarded_here.get(key, False) or (var in called_vars) + for key, g in guarded_here.items(): + consumers.append((fname, key, g)) + if g: + guarded_keys.add(key) + +if not guarded_keys: + bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. " + "Either the guards were removed, or the way the script reads cluster-vars changed and " + "this check's derivation must be updated in the SAME change — it must never silently " + "verify an empty set.") + +# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it. +for fname, key, guarded in sorted(consumers): + if key not in guarded_keys: + continue # nothing claims responsibility for this knob at all + check(guarded, + f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a " + "*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers " + "of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips " + "the guard is that failure in one function.") + +# ---------------------------------------------------------------- independent detector +# Deliberately NOT derived from the guard alone (see the header): SEED is universal +# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose +# ANCHORING is wrong still has its vocabulary tested against real values. +SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME", + "CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"] +VOCAB = sorted({w.upper() for w in SEED} | set(all_words)) +ANGLE = re.compile(r"<[^>\s]+>") + + +def placeholder_shaped(value: str) -> bool: + if value.strip() == "": + return True # an unset endpoint is not a target + up = value.upper() + return any(w in up for w in VOCAB) or bool(ANGLE.search(value)) + + +rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value +GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None + +# (1) VALUES: base default and every overlay's value, for every guarded knob. +for key in sorted(guarded_keys): + seen: list[tuple[str, str]] = [] + if key in base: + seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key])) + for env, eff in sorted(overlays.items()): + seen.append((f"clusters/{env} effective:{key}", eff.get(key, ""))) + check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard " + "protects a knob nothing authors") + for origin, value in seen: + expect = "placeholder" if placeholder_shaped(value) else "usable" + for name in sorted(guard_words): + rows.append((name, expect, origin, value)) + +# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob. +for env, eff in sorted(overlays.items()): + for key, value in sorted(eff.items()): + if key in guarded_keys or not placeholder_shaped(value): + continue + check(False, + f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, " + f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it " + "reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay " + "that ships a placeholder no guard recognises is the 2026-09-05 failure with a " + "different knob's name on it.") + +# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix, +# and a realistic endpoint must still be accepted. +for name, words in sorted(guard_words.items()): + for w in sorted(set(words)): + rows.append((name, "placeholder", + f"anchoring:{name}:mid-string({w})", + f"https://account-{w}-id.r2.cloudflarestorage.com")) + rows.append((name, "placeholder", f"anchoring:{name}:tail({w})", + f"https://r2-endpoint.example.invalid/{w}")) + rows.append((name, "placeholder", f"canary:{name}:empty", "")) + rows.append((name, "usable", f"canary:{name}:real-r2", + "https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com")) + rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000")) + +# Every guarded knob must reach the oracle once per authored source (base + each overlay), +# so a knob silently dropping out of the corpus is a failure rather than a quiet pass. +authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))] +expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words) +check(len(authored) == expected_rows, + f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} " + f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x " + f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from " + "an overlay's effective values, and the coupling is only partly exercised") + +# ------------------------------------------------------- (6) the check is load-bearing +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +check("--layer l0" in L0, + "ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate") +check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# --------------------------------------------------------------------------- handoff +with (WORK / "corpus.tsv").open("w") as fh: + for name, expect, origin, value in rows: + fh.write(f"{name}\t{expect}\t{origin}\t{value}\n") +(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES)) +(WORK / "counts.txt").write_text(f"{CHECKS}\n") + +print(f"guards: {', '.join(sorted(guard_words))} " + f"vocabulary: {', '.join(VOCAB)}") +print(f"guarded knobs: {', '.join(sorted(guarded_keys))} " + f"consumers traced: {len(consumers)} overlays: {', '.join(sorted(overlays))}") +for name, expect, origin, value in rows: + if not origin.startswith(("anchoring:", "canary:")): + print(f" authored {origin} = {value!r} -> expect {expect}") +PY + +# --------------------------------------------------------------------------------------- +# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them. +# --------------------------------------------------------------------------------------- +# shellcheck disable=SC1090,SC1091 +source "${WORK}/guards.sh" + +oracle_failures=0 +oracle_rows=0 +while IFS=$'\t' read -r guard expect origin value; do + [ -n "${guard}" ] || continue + oracle_rows=$((oracle_rows + 1)) + if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi + if [ "${verdict}" != "${expect}" ]; then + oracle_failures=$((oracle_failures + 1)) + if [ "${expect}" = placeholder ]; then + printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt" + else + printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt" + fi + fi +done < "${WORK}/corpus.tsv" + +# ------------------------------------------------------------------------------ verdict +py_checks="$(cat "${WORK}/counts.txt")" +total=$((py_checks + oracle_rows)) +findings="$(cat "${WORK}/findings.txt")" + +# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it +# asked a question that was always satisfiable. The floors below are the minimum evidence +# for the verdict to mean anything, not a style rule. +if [ "${oracle_rows}" -lt 6 ]; then + findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised +" +fi +if [ "${total}" -lt 20 ]; then + findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless +" +fi + +if [ -n "${findings}" ]; then + echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2 + printf '%s' "${findings}" | sed 's/^/ - /' >&2 + exit 1 +fi + +echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header." diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml new file mode 100644 index 0000000..532aa7e --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml @@ -0,0 +1,77 @@ +# ============================================================================= +# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local +# storage; it MUST NOT be an active development surface until the roadmap +# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to +# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step +# differs", "Never develop against prod"); roadmap/phases.md Phase 7. +# +# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a +# governance rule is verified through the property that ENFORCES it, never by observing +# that nobody misbehaved today. Here that property is structural: prod receives the base +# configuration plus environment-scoped overrides and nothing else, from the same branch +# and the same Ansible code as dev, so there is no place to put prod-only work at all. +# +# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose +# group_vars retargets the k3s roles away from the global default), the prod overlay is +# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the +# node count that inventory resolves to. No hostname, address or environment name is +# written into the test. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and +# the shared provisioning chain is actually retargeted at prod with --list-hosts, +# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC +# resource creates any of them (bare metal is imaged, not instantiated); prod's +# Longhorn path is node-local with one replica per node and the default +# StorageClass is the local driver; prod runs the same playbooks/roles as dev, +# retargeted only by variables the shared code already defines; the prod overlay +# carries only the base plus environment-scoped patches on the same Git ref dev +# tracks, with no prod-only manifest anywhere in the shared configuration; and prod +# artifacts exist only alongside the operator's recorded opening of the prod phase. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and — +# where present — ansible-playbook in --list-hosts mode, which opens no connection); +# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test +# wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local +# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the +# node-health/BIOS surveys that establish it write to gitignored artifacts, and the +# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it. +# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered +# HA test; asserting it here would also make this test environment-specific, since +# the same suite must pass against dev. +# * Git HISTORY: this is the working tree at HEAD. +# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by +# the recorded declaration. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: prod-baremetal-not-dev-surface + labels: + conformance.home.k3s/R-ENV-03: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are + inspected, then prod is authored as three bare-metal nodes with node-local storage, + bootstrapped by dev's own code retargeted by variables, receiving only + environment-scoped overrides on the same branch — so there is nowhere to develop + against prod — and it exists only because the roadmap phase was opened. +spec: + steps: + - name: then-prod-is-bare-metal-and-carries-no-development-of-its-own + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh new file mode 100755 index 0000000..1fd041a --- /dev/null +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh @@ -0,0 +1,528 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface +# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it +# MUST NOT be an active development surface until the roadmap declares it ready." +# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC +# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn", +# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs", and "**Never develop against prod.** Prod hardware is a target, not +# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry +# "operator declares prod hardware ready"; note "Do not begin building on prod +# hardware before this phase is explicitly opened".) +# +# HOW A "MUST NOT" IS MADE TESTABLE +# -------------------------------- +# The second clause is a governance rule, and the same discipline applies as to R-TEST-06 +# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today" +# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can +# be. In this repo that property is structural — prod receives the base configuration plus +# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work +# even if someone wanted to. This check asserts that, together with the first clause's +# authored facts (three nodes, no VM IaC behind them, node-local storage, the same +# bootstrap code as dev). +# +# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really +# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable — +# see the gap list. What IS checkable is that the repository describes prod that way and +# that no IaC creates prod nodes as virtual machines. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are +# inspected — and, where offline execution is possible, the provisioning chain is +# actually retargeted at prod with `--list-hosts`, +# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd +# (no NAT indirection, i.e. real machines on the LAN); +# (2) NO IaC resource creates any prod node: the VM module's generated node names are +# disjoint from the prod host set and no prod hostname appears anywhere under +# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated; +# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute +# local path, and the replica count equals the node count (one replica per node's +# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and +# is not counted against this; +# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted +# by variables only — every prod group_vars key is an existing role/global +# variable (an invented prod-only knob would be a fork), and no prod-only role +# or playbook exists; +# (5) prod is NOT a development surface: clusters/prod carries only the base plus +# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object +# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source), +# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on), +# and no prod-only manifest exists anywhere in the rendered configuration; +# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap +# carries the phase that opens prod build (checked when the spec repo is mounted), +# and the repository records that the operator opened it. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local +# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the +# node-health and BIOS surveys that establish it write to gitignored artifacts, so +# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no +# "am I virtual" fact either — a bare-metal check would need a privileged host-path +# probe of DMI data, which is a heavier and more invasive test than the claim warrants. +# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn +# disks at the local path). That is cluster state and belongs to R-PLAT-01's +# (currently uncovered) HA test; asserting it here would also make this test +# environment-specific, since the same suite must pass against dev. +# - That prod's HISTORY contains no development. This is the working tree at HEAD. +# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that +# prod work is accompanied by the operator's recorded declaration. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +: "${SPEC_REPO:=${ROOT}/../home-network-k3s}" +export SPEC_REPO + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept +# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from +# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect, +# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately +# narrow: every other non-zero exit is a real failure. The parse assertions always run. +ENV_ONLY_FAILURE = re.compile( + r"vault password|vault-id|Attempting to decrypt" + r"|No module named 'ansible_collections|couldn't resolve module/action", re.I) + +ANS = ROOT / "infra" / "ansible" +INV = ANS / "inventory" / "hosts.ini" +GV = ANS / "inventory" / "group_vars" +TOFU = ROOT / "infra" / "tofu" +check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere") +if not INV.is_file(): + print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr) + raise SystemExit(1) + +# --------------------------------------------------------------- inventory parsing +ini_groups: dict[str, list[dict]] = {} +children: dict[str, list[str]] = {} +cur = None +for line in INV.read_text().splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + continue + if cur is None or cur.endswith(":vars"): + continue + if cur.endswith(":children"): + children.setdefault(cur.split(":")[0], []).append(s.split()[0]) + continue + parts = s.split() + host = {"name": parts[0]} + for kv in parts[1:]: + if "=" in kv: + k, v = kv.split("=", 1) + host[k] = v + ini_groups.setdefault(cur, []).append(host) + + +def hosts_in(group: str, seen=None) -> list[dict]: + seen = seen or set() + if group in seen: + return [] + seen.add(group) + out = list(ini_groups.get(group, [])) + for child in children.get(group, []): + out += hosts_in(child, seen) + return out + + +def load_gv(group: str) -> dict: + d: dict = {} + for p in [GV / f"{group}.yml", GV / group / "vars.yml"]: + if p.is_file(): + try: + d.update(yaml.safe_load(p.read_text()) or {}) + except Exception: + pass + return d + + +# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars +# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global +# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod. +ALL_VARS = load_gv("all") +dev_primary_group = str(ALL_VARS.get("k3s_primary_group", "")) +candidates = [] +for entry in sorted(GV.iterdir()) if GV.is_dir() else []: + g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name + if g == "all": + continue + v = load_gv(g) + if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group: + candidates.append((g, v)) +check(len(candidates) == 1, + f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found " + f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through " + "the same roles as dev") +if not candidates: + print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +PROD_GROUP, PROD_VARS = candidates[0] +PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"]) +prod_hosts = hosts_in(PROD_GROUP) +prod_names = sorted(h["name"] for h in prod_hosts) + +# ------------------------------------------------- (1) exactly three, directly reachable +check(len(prod_names) == 3, + f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 " + "says prod is a 3× mini-PC cluster") +dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers"))) +for h in prod_hosts: + # Real machines on the LAN answer their own sshd. Dev's hosts are reached through a + # NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the + # indirection a hypervisor-hosted estate needs and bare metal does not. + check(str(h.get("ansible_port", "22")) == "22", + f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port " + "implies an indirection (NAT to a virtual estate), not a machine on the LAN") + check(h.get("k3s_node_ip") == h.get("ansible_host"), + f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but " + f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a " + "bare-metal node's SSH address IS its cluster address") + +# ------------------------------------------------------ (2) nothing creates prod as a VM +TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \ + if TOFU.is_dir() else "" +check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod") + + +def tf_default(var: str): + m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S) + if not m: + return None + d = re.search(r"default\s*=\s*(.+)", m.group(1)) + return d.group(1).strip() if d else None + + +prefix = (tf_default("node_name_prefix") or "").strip('"') +try: + node_count = int((tf_default("node_count") or "0").strip()) +except ValueError: + node_count = 0 +iac_names = {f"{prefix}{i + 1}" for i in range(node_count)} +check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod") +check(iac_names.isdisjoint(set(prod_names)), + f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes " + "— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the " + "hypervisor module") +for name in prod_names: + check(name not in TF, + f"prod node {name} is named inside infra/tofu — prod hardware must not be described by " + "the VM module at all") +# The prod host addresses must not appear as IaC node addresses either (a renamed VM is +# still a VM). +tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or "")) +prod_ips = {str(h.get("ansible_host")) for h in prod_hosts} +check(tf_ips.isdisjoint(prod_ips), + f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses") + +# --------------------------------------------------------------- (3) node-local storage +# Every environment's value for a cluster-vars knob (base default + each overlay patch). +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and (d.get("metadata") or {}).get("name") == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default. +cluster_envs = env_values("CLUSTER_ENV") +prod_overlays = [e for e, v in cluster_envs.items() + if e != "base" and v != cluster_envs.get("base")] +check(len(prod_overlays) >= 1, + "no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own") +prod_overlay = None +for e in prod_overlays: + if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP: + prod_overlay = e +check(prod_overlay is not None, + f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the " + "cluster config and the machine inventory disagree about which estate is prod") + +data_paths = env_values("LONGHORN_DATA_PATH") +replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT") +if prod_overlay: + p = data_paths.get(prod_overlay, data_paths.get("base", "")) + check(p.startswith("/"), + f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute " + "node-local filesystem path") + check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p), + f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe") + r = replicas.get(prod_overlay, replicas.get("base", "")) + check(r.isdigit() and int(r) == len(prod_names), + f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — " + "'local storage' on a 3-node estate means one replica per node's own disk") +# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier +# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not +# counted here. +default_scs = [] +for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"], + capture_output=True, text=True, check=True).stdout.split("\0"): + if not f: + continue + try: + for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")): + if isinstance(d, dict) and d.get("kind") == "StorageClass" and \ + str(((d.get("metadata") or {}).get("annotations") or {}) + .get("storageclass.kubernetes.io/is-default-class", "")) == "true": + default_scs.append((f, str(d.get("provisioner", "")))) + except Exception: + continue +check(len(default_scs) == 1, + f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads " + "land on by default must be unambiguous") +for f, prov in default_scs: + check(prov == "driver.longhorn.io", + f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default " + "tier would not be its own local disks") + +# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted +PLAYBOOKS = ANS / "playbooks" +ROLES = ANS / "roles" +prod_token = PROD_GROUP.replace("k3s_", "") +for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")): + for entry in sorted(d.iterdir()) if d.is_dir() else []: + check(prod_token not in entry.name.lower(), + f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires " + "the SAME Ansible bootstrap for prod, differing only in the machine-creation step") +# Every prod group_vars key must be a variable the shared code already knows: a role default, +# a global, or an ansible built-in. An invented key is a fork of the role by another name. +known: set[str] = set(ALL_VARS) +for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []: + for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []: + try: + known |= set((yaml.safe_load(p.read_text()) or {}).keys()) + except Exception: + pass +check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing") +for k in sorted(PROD_VARS): + check(k.startswith("ansible_") or k in known, + f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only " + "configuration that the dev path never exercises is developing against prod") +# ...and it must retarget, not re-implement: the group it names must exist in the inventory. +check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children, + f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define") + +# ---------------------------------------- (5) prod is not a development surface (overlay) +if prod_overlay: + odir = ROOT / "clusters" / prod_overlay + files = sorted(p.name for p in odir.iterdir() if p.is_file()) + check(files == ["kustomization.yaml", "sync.yaml"], + f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its " + "kustomization and its own root sync; anything else is configuration authored FOR PROD") + k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {} + check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"], + f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base " + "plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')") + # Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs. + ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)} + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + body = p.get("patch") or "" + kind = tgt.get("kind") + if kind is None: # inline patch: kind comes from the body + try: + doc = yaml.safe_load(body) or {} + except Exception: + doc = {} + kind = doc.get("kind") + name = (doc.get("metadata") or {}).get("name") + if kind == "Kustomization": + # Flux Kustomization patches are allowed ONLY to wire sops decryption — + # anything else (path, interval, images, values) changes what prod RUNS. + keys = set((doc.get("spec") or {}).keys()) + check(keys <= {"decryption"}, + f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} " + "— only `decryption` (the per-cluster sops key) is an environment-scoped " + "override; changing anything else is developing against prod") + continue + check((kind, None) in ALLOWED or (kind, name) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an " + "environment-scoped object — prod must not carry configuration of its own") + else: + check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED, + f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not " + "an environment-scoped object — prod must not carry configuration of its own") + + # Same Git ref as the other environment(s): a prod-only branch is a development line + # that only prod runs, which is exactly what R-ENV-03 forbids. + def git_ref(overlay: str): + kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {} + for p in (kk.get("patches") or []): + body = p.get("patch") or "" + try: + doc = yaml.safe_load(body) or {} + except Exception: + continue + if doc.get("kind") == "GitRepository": + return (doc.get("spec") or {}).get("ref") + return None + + others = [o.name for o in (ROOT / "clusters").iterdir() + if o.is_dir() and o.name not in ("base", prod_overlay)] + prod_ref = git_ref(prod_overlay) + check(prod_ref is not None, + f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base " + "points at, with no record of what it runs") + for o in others: + check(git_ref(o) == prod_ref, + f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a " + "development line only prod runs (R-ENV-03: prod is a target, not a sandbox)") + +# No prod-only manifest anywhere in the rendered configuration. +tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True + ).stdout.split("\0") if f] +RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/") +for f in tracked: + if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", + f, re.I): + check(False, + f"{f} is a prod-specific file inside the shared configuration — every environment " + "renders these paths, so prod-only content here is development against prod") +check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout") + +# --------------------------- (6) prod work exists only with a declaration of readiness +# In-repo: the operator's recorded decision to open the prod phase. +log = ROOT / "DECISIONS-LOG.md" +check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I), + "DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod " + "inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready") +# In the spec (read-only, may not be mounted): the roadmap phase that opens prod. +spec = Path(os.environ.get("SPEC_REPO", "")) +phases = spec / "roadmap" / "phases.md" +if phases.is_file(): + txt = phases.read_text(errors="replace") + check(re.search(r"##\s*Phase\s*7\b", txt) is not None and + re.search(r"prod", txt, re.I) is not None, + "the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers " + "to does not exist") + check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I) + is not None, + "the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 " + "names would be unenforceable") +else: + note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked " + "(the in-repo declaration still was)") + +# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts +if shutil.which("ansible-playbook") is None: + note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the " + "prod retarget was PARSED but not DRIVEN; the merge gate's container has it") +else: + sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY] + args = ["--list-hosts", "playbooks/site.yml", + "-e", f"k3s_hosts={PROD_GROUP}", + "-e", f"k3s_primary_hosts={PROD_PRIMARY}"] + if sec_groups: + args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"] + lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True) + out = (lh.stderr or "") + (lh.stdout or "") + if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out): + # See the same narrow, self-policing skip in dev-nodes-provisioned-as-code. + note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault " + "passphrase and/or the gitignored Galaxy collections), so the prod retarget was not " + "driven; every parse assertion above still ran") + else: + check(lh.returncode == 0, + f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires " + "the same Ansible bootstrap to run for prod") + seen: set[str] = set() + in_hosts = False + for line in lh.stdout.splitlines(): + if re.match(r"\s*hosts \(\d+\):", line): + in_hosts = True + continue + if re.search(r"play #\d+", line): + in_hosts = False + continue + if in_hosts and line.strip(): + seen.add(line.strip()) + check(seen == set(prod_names), + f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod " + f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path") + +# --------------------------------------------------------------------------- verdict +check(CHECKS >= 25, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +for n in NOTES: + print(f" note: {n}") +print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = " + f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base " + f"plus environment-scoped patches only, local Longhorn path " + f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with " + f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses " + "only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the " + "live cluster's shape is NOT asserted here (see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..f137268 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,69 @@ +# ============================================================================= +# L0 STATIC — certificate lifecycle needs no manual step (R-TLS-02) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and +# revoked without manual steps"); mechanism in docs/08 + ADR-0006. +# +# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the +# clause — "without manual steps" — which in a GitOps platform is a property of the +# REPOSITORY: a human step in the certificate path leaves evidence in git (committed +# cert material, a Gateway serving a Secret nothing declares, an issuer with no +# automated solver, a credential never delivered, auto-approval switched off, or CRs +# nothing reconciles). The three verbs are observed by the companion L2 test. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`, +# so the check resolves it against the BASE cluster-vars default AND every overlay's value and +# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from +# one authored assertion, with no hostname, address or environment name written down. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# is inspected offline, +# THEN every Gateway listener that terminates TLS serves a Secret a tracked Certificate +# produces; no certificate material is committed anywhere; every Certificate's +# issuer resolves — in EVERY environment — to a tracked ACME issuer with an +# automated dns01/http01 solver whose credential is SOPS-delivered from git; +# cert-manager's automatic request approval is not disabled; and every one of +# those objects is rendered by a Flux Kustomization, so nothing in the chain +# depends on a human running kubectl. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and +# this Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them): +# * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated). +# * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation, +# and the spec's own reading (docs/08: "removing/rotating a certificate is a +# declarative change") needs `enableCertificateOwnerRef`, which the tracked +# cert-manager HelmRelease does not set — so a removed Certificate leaves its +# serving Secret behind. The L2 test asserts the withdrawal behaviour and is +# EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06). +# * That the live cluster is serving the tracked Certificate (cluster state). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: tls-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the certificate path is inspected offline, then no + manual step exists anywhere in it: serving Secrets are cert-manager-produced, no + cert material is committed, every environment's issuer is a tracked ACME issuer + with an automated solver and a git-delivered credential, auto-approval is on, and + the whole chain is Flux-reconciled. +spec: + steps: + - name: then-no-manual-step-exists-in-the-certificate-path + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh + and conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/check.sh b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh new file mode 100755 index 0000000..294b4b1 --- /dev/null +++ b/conformance/tests/l0-static/tls-lifecycle-automated/check.sh @@ -0,0 +1,478 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — tls-lifecycle-automated +# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked** +# without manual steps." — spec docs/00-vision-and-scope.md; mechanism in +# docs/08-tls-and-certificates.md, ADR-0006.) +# +# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT +# ------------------------------------------------------ +# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs +# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps* +# clause IS a repository property, and it is the half that can be proven with no +# cluster: if a human had to mint, install, or approve a serving certificate, the +# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway +# pointing at a Secret nothing declares, an issuer that no automated solver can +# satisfy, a per-cert credential handed over out of band, or an admission path that +# waits for a manual CertificateRequest approval. This check asserts that none of +# those exist and that the automated path is completely wired instead. +# +# The three VERBS are runtime events. They are asserted by the companion L2 test +# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an +# issuance, an unattended renewal, and the withdrawal of a removed certificate. This +# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below. +# +# GIVEN the repository as git tracks it, +# WHEN every serving-TLS reference, Certificate, Issuer and the cert-manager release +# are inspected offline, +# THEN (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo +# with no Certificate cannot pass this check by having nothing to check); +# (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets +# that a tracked Certificate declares as its `secretName` — so the serving +# credential is produced by the controller, never dropped in by hand; +# (3) NO certificate material is tracked anywhere: no Secret manifest of type +# kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret +# (plain OR SOPS-encrypted), and no PEM certificate body in any tracked +# YAML — i.e. issuance is not, and cannot silently become, a human step; +# (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY +# ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the +# base cluster-vars default AND every overlay's value, so an environment +# whose issuer does not exist is caught here rather than at prod bring-up; +# (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list +# whose every solver is an automated challenge type (dns01 / http01) — the +# same code path serves issuance AND renewal, so a solver a human must +# complete would defeat both; +# (6) each dns01 solver's provider credential is referenced BY NAME and that +# Secret is delivered SOPS-encrypted from git via a kustomization — the +# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step; +# (7) cert-manager's automatic CertificateRequest approval is NOT disabled +# (`disableAutoApproval`), and any `approveSignerNames` narrowing still +# covers the ClusterIssuers actually used — a request that waits for a +# human approver is the definition of a manual step; +# (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the +# ClusterIssuers and the Certificate are each included in a kustomization +# that a Flux Kustomization renders — so a deleted Certificate is +# re-created and re-issued with no human action; +# (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration` +# would mean the renewal instant never arrives). +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of +# l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path, +# evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert. +# - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate +# is explicit that this must be "observed across a renewal" — L2 step 3 does that +# by fast-forwarding a certificate's renewal window; nothing static can. +# - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct +# readings, neither established: (a) ACME revocation — cert-manager does not +# revoke certificates at all, so no repo state could show it; (b) the spec's own +# reading in docs/08 ("removing/rotating a certificate is a declarative change") +# — withdrawal of a REMOVED certificate's serving material requires cert-manager's +# `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart +# default false). L2 step 4 asserts the withdrawal behaviour and is expected to +# FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06). +# - That the tracked Certificate is the one the live Gateway is actually serving +# (cluster state), or that the ACME account/DNS credential is still valid. +# - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see +# l3-fault-injection/alerting). +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def tracked() -> list[str]: + out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout + return [p for p in out.split("\0") if p] + + +FILES = tracked() +check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout") +YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)] + + +def docs_of(rel: str): + """Parsed YAML documents of a tracked file, or [] when it is not parseable YAML.""" + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)] + + +def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]: + out = [] + for rel, d in DOCS: + if d.get("kind") != kind: + continue + if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix): + continue + out.append((rel, d)) + return out + + +def ns_of(d: dict, default: str = "default") -> str: + return (d.get("metadata") or {}).get("namespace") or default + + +def name_of(d: dict) -> str: + return (d.get("metadata") or {}).get("name") or "" + + +# ------------------------------------------------------------ cluster-vars resolution +# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902 +# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This +# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an +# issuer name can be resolved per environment instead of being asserted for dev only. +def env_values(key: str) -> dict[str, str]: + vals: dict[str, str] = {} + base = ROOT / "clusters" / "base" / "cluster-vars.yaml" + if base.is_file(): + for d in yaml.safe_load_all(base.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and name_of(d) == "cluster-vars": + v = (d.get("data") or {}).get(key) + if v is not None: + vals["base"] = str(v) + for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception: + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: + continue + for op in ops if isinstance(ops, list) else []: + if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}": + vals[env] = str(op.get("value")) + return vals + + +TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$") + + +def resolve_per_env(value: str) -> dict[str, str]: + """A literal resolves to itself in every environment; a ${TOKEN} to each env's value.""" + m = TOKEN_RE.match(value.strip()) + if not m: + return {"(literal)": value.strip()} + return env_values(m.group(1)) + + +# --------------------------------------------- what Flux actually renders (reachability) +# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through +# the kustomize resource graph: /kustomization.yaml -> resources -> (file | directory +# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by +# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere": +# the loose form silently passes when an entry is deleted from the very kustomization.yaml +# that the directory reference renders (verified — it made two assertions here no-ops). +KUST_RESOURCES: dict[str, list[str]] = {} +for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]: + refs: list[str] = [] + for d in docs_of(rel): + for r in (d.get("resources") or []) + (d.get("components") or []): + if not isinstance(r, str) or "://" in r: # skip remote bases + continue + try: + p = (ROOT / rel).parent.joinpath(r).resolve() + refs.append(str(p.relative_to(ROOT))) + except Exception: + continue + KUST_RESOURCES[rel] = refs + +FLUX_PATHS: set[str] = set() +for _rel, _d in DOCS: + if _d.get("kind") == "Kustomization" and \ + str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"): + p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/") + if p: + FLUX_PATHS.add(p) + + +def _walk(entry: str, seen: set[str], out: set[str]) -> None: + if entry in seen: + return + seen.add(entry) + kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml" + if kust in KUST_RESOURCES: # a directory (or a kustomization) + for r in KUST_RESOURCES[kust]: + _walk(r, seen, out) + return + out.add(entry) # a leaf manifest + + +RENDERED: set[str] = set() +for _p in FLUX_PATHS: + _walk(_p, set(), RENDERED) + + +def delivered(rel: str) -> bool: + """True iff Flux renders this file from one of its Kustomization paths.""" + return rel in RENDERED + + +# --------------------------------------------------- (1) there is something to check +CERTS = of_kind("Certificate", "cert-manager.io/") +check(len(CERTS) >= 1, + "no cert-manager.io Certificate is authored anywhere in the tracked tree — every " + "assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)") + +ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/") +check(len(ISSUERS) >= 1, + "no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically") + +# ---------------------------- (2) every terminating listener serves a controller cert +CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName) -> file + (ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel + for rel, d in CERTS +} +listeners_checked = 0 +for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"): + gw_ns = ns_of(gw) + for lis in ((gw.get("spec") or {}).get("listeners") or []): + tls = lis.get("tls") or {} + if str(tls.get("mode", "")) != "Terminate": + continue + listeners_checked += 1 + refs = tls.get("certificateRefs") or [] + check(len(refs) > 0, + f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates " + "TLS with NO certificateRefs — it can only serve a certificate someone puts " + "there by hand") + for r in refs: + kind = str(r.get("kind", "Secret")) + key = (str(r.get("namespace") or gw_ns), str(r.get("name"))) + check(kind == "Secret", + f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only " + "Secret refs can be traced to a cert-manager Certificate") + check(key in CERT_SECRETS, + f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which " + "NO tracked cert-manager Certificate declares as its secretName — that " + "credential would have to be created by hand (R-TLS-02 'without manual steps')") +check(listeners_checked > 0, + "no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no " + "serving-TLS reference to trace, so (2) proved nothing") + +# --------------------------------- (3) no certificate material is tracked, anywhere +TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"} +for rel, d in DOCS: + if d.get("kind") != "Secret": + continue + stype = str(d.get("type", "")) + check(stype != "kubernetes.io/tls", + f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name=" + f"{name_of(d)}) — a serving certificate authored by a human, not issued") + for field in ("data", "stringData"): + for k in (d.get(field) or {}): + check(str(k) not in TLS_KEYS, + f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate " + "material in git means a manual issuance step exists (R-TLS-02)") + +PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----") +for rel in YAML_FILES: + body = (ROOT / rel).read_text(errors="replace") + # Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header + # in an example, and a .md file is not applied to any cluster. A manifest cannot. + check(not PEM_RE.search(body), + f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest") + +# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env +# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace` +# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the +# HelmRelease, never written down here, so this holds if the release namespace ever changes. +cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"] +CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager" + +ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS} +solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field) +issuers_verified: set[str] = set() +for rel, cert in CERTS: + spec = cert.get("spec") or {} + ref = spec.get("issuerRef") or {} + kind = str(ref.get("kind", "Issuer")) + per_env = resolve_per_env(str(ref.get("name", ""))) + check(len(per_env) > 0, + f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in " + "clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue") + for env, iname in sorted(per_env.items()): + hit = ISSUER_INDEX.get((kind, iname)) + check(hit is not None, + f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in " + f"environment {env!r}, but no such issuer is tracked — that environment cannot " + "issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)") + if hit is None: + continue + irel, issuer = hit + issuers_verified.add(iname) + ispec = issuer.get("spec") or {} + acme = ispec.get("acme") + check(isinstance(acme, dict), + f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME " + f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate " + "ACME so both are unattended") + if not isinstance(acme, dict): + continue + check(str(acme.get("server", "")).startswith("https://"), + f"{irel}: {kind}/{iname} has no https ACME directory endpoint") + check(bool((acme.get("privateKeySecretRef") or {}).get("name")), + f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key " + "would have to be provisioned by hand") + solvers = acme.get("solvers") or [] + check(len(solvers) > 0, + f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved " + "until a human completed the challenge") + for s in solvers: + types = [t for t in ("dns01", "http01") if t in s] + check(len(types) == 1, + f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only " + "the automated dns01/http01 solvers satisfy 'without manual steps'") + for provider, cfg in (s.get("dns01") or {}).items(): + if not isinstance(cfg, dict): + continue + for field, val in cfg.items(): + if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")): + n = str(val.get("name", "")) + check(bool(n), + f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name") + if n: + # A ClusterIssuer resolves secret refs in the controller's + # cluster-resource-namespace; a namespaced Issuer in its own. + want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS) + solver_creds.append((iname, want_ns, n, f"{provider}.{field}")) + +check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing") + +# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand. +check(len(solver_creds) > 0, + "no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped " + "Cloudflare token to drive DNS-01, so this check would otherwise be vacuous") +# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential. +# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy +# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.) +SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d) + for rel, d in DOCS if d.get("kind") == "Secret"} +for iname, sns, sname, field in solver_creds: + hit = SECRET_DOCS.get((sns, sname)) + check(hit is not None, + f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest " + f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the " + "credential would be created by hand in-cluster (R-TLS-02)") + if hit is None: + continue + srel, sdoc = hit + check("sops" in sdoc, + f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)") + check(delivered(srel), + f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would " + "never deliver it, so issuance would wait on a manual kubectl apply") + +# ------------------------------- (7) automatic CertificateRequest approval is intact +check(len(cm_releases) == 1, + f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — " + "the controller that issues and renews must itself be GitOps-managed") +for rel, hr in cm_releases: + values = ((hr.get("spec") or {}).get("values") or {}) + check(values.get("disableAutoApproval") in (None, False), + f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} " + "— every CertificateRequest would then wait for an approver, which is a manual step") + signers = values.get("approveSignerNames") + if signers is not None: + pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])] + for iname in sorted(issuers_verified): + ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and + (p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats) + check(ok, + f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — " + "requests for it would never be auto-approved") + # The chart's own CRD install must be on, else the CRs below cannot even be applied. + check((values.get("crds") or {}).get("enabled") is True, + f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD " + "would have to be installed by hand") + +# ------------------------------- (8) the whole chain is continuously reconciled from git +check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20, + f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} " + "Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing") +for rel, _ in cm_releases + CERTS + ISSUERS: + check(delivered(rel), + f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a " + "deleted Certificate/issuer/controller would stay deleted until a human re-applied it") + +# --------------------------------------------- (9) renewal is not defeated arithmetically +def to_seconds(v) -> int | None: + m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v)) + if not m: + return None + return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)] + + +renew_pairs = 0 +for rel, cert in CERTS: + spec = cert.get("spec") or {} + dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore")) + if dur is None or rb is None: + continue + renew_pairs += 1 + check(rb < dur, + f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= " + f"duration={spec.get('duration')} — the renewal instant never arrives") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op. +# The floor below is a count of assertions that MUST have run for the verdict to mean +# anything; it is not a style rule. +check(CHECKS >= 20, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), " + f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate " + f"across every environment, {listeners_checked} terminating listener(s), " + f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). " + "R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs " + "(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).") +PY diff --git a/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml new file mode 100644 index 0000000..14fca5e --- /dev/null +++ b/conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml @@ -0,0 +1,391 @@ +# ============================================================================= +# L2 BEHAVIORAL — certificate lifecycle is automatic (R-TLS-02) +# [RUNNABLE / NOT YET EXECUTED — see STATUS] +# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" +# Spec: docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed, +# and revoked** without manual steps." +# docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well +# before expiry automatically; no human action." / "Revocation: removing/ +# rotating a certificate is a declarative change; cert-manager handles +# issuance of replacements." +# docs/08 acceptance gate — "Certificates renew automatically ahead of expiry +# with no manual step (**observed across a renewal**)." +# Impl: platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml. +# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps" +# clause statically (in the merge gate). This test observes the three VERBS. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The +# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow +# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name` +# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no +# address: dev proves dev's cert and prod proves prod's from the same authored assertions. +# +# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS, +# WHEN the serving Certificate's provenance and renewal schedule are read, and a NEW +# Certificate is then requested, left to renew itself, and removed, +# THEN 1. the serving credential was produced by the CONTROLLER, not a human: its +# Secret carries cert-manager's certificate-name annotation, its +# CertificateRequest was submitted by cert-manager's own ServiceAccount and +# was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer; +# 2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and +# strictly before status.notAfter, so no human is in the renewal path; +# 3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears +# with certificate material, with no step between declaring and having it; +# 4. RENEWED — that certificate, given a renewal window that opens one minute +# after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and +# the serving bytes actually change, with no human action. This is docs/08's +# "observed across a renewal", fast-forwarded; +# 5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material. +# +# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS +# ------------------------------------------------------------------- +# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every +# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order +# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate +# limits for evidence that is not about ACME: cert-manager's renewal scheduler and +# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created +# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2 +# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls +# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03). +# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the +# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while +# steps 1-2 prove the real serving certificate is on exactly that automated path. +# +# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an +# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's +# per-test namespace — the same class of mutation every other L2 test here performs, +# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored +# them against live dev/prod read-only inspection under an explicit instruction not to +# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the +# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true. +# +# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06) +# ------------------------------------------------------ +# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects +# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`; +# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the +# live wildcard Secret on both clusters has NO ownerReferences (verified read-only, +# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked +# without manual steps" does not hold. That is a finding about the platform, not a +# defect in this test: the assertion stays as the requirement reads and is escalated to +# the PM. (ACME revocation proper is not implemented by cert-manager at all.) +# +# KNOWN, DELIBERATE GAPS (this test claims none of these): +# * Not proven: that the CA would refuse a revoked certificate — there is no +# revocation request to make, so nothing observable exists to assert. +# * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a +# 90-day expiry; a conformance run cannot wait, and forcing it would rotate the +# platform's serving credential). Step 2 asserts the schedule; step 4 asserts the +# mechanism that will act on it. +# * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04 +# (l3-fault-injection/alerting). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cert-lifecycle-automated + labels: + conformance.home.k3s/R-TLS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given cert-manager and a TLS-terminating Gateway, when the serving certificate's + provenance and renewal schedule are read and a new certificate is requested, left + to renew, and removed, then issuance, renewal and withdrawal all happen with no + human step — the serving cert was controller-issued and auto-approved, its + renewal is scheduled ahead of expiry, and a declared certificate issues and + reissues itself unattended. +spec: + timeouts: + apply: 60s + assert: 3m + exec: 10m + steps: + # ---------------------------------------------------------------- READ-ONLY HALF + - name: given-cert-manager-is-reconciled-and-healthy + description: >- + The controller that issues and renews must be running, and must have been + installed by Flux rather than by hand — otherwise "automatic" has no engine. + try: + - script: + content: | + set -u + # Discover cert-manager's Deployments by chart label, in whatever namespace + # the release lives (nothing about the environment is assumed). + sel='app.kubernetes.io/instance=cert-manager' + rows="$(kubectl get deploy -A -l "$sel" \ + -o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \ + --no-headers 2>/dev/null || true)" + echo "$rows" + [ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; } + n=0 + echo "$rows" | while read -r ns name rdy; do + [ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; } + done || exit 1 + n="$(echo "$rows" | wc -l)" + # controller + webhook + cainjector: without the webhook a Certificate is + # rejected, without cainjector the webhook has no CA — all three or nothing. + [ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; } + # GitOps-managed, not hand-installed: Flux labels what it applies. + ns="$(echo "$rows" | head -1 | awk '{print $1}')" + lbl="$(kubectl get deploy -n "$ns" -l "$sel" \ + -o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)" + [ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; } + echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)" + check: + (contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true + + - name: then-the-serving-certificate-was-issued-by-the-controller + description: >- + Follow the Gateway's terminating listener to the Secret it serves, and prove + that Secret's provenance is cert-manager — annotation, CertificateRequest + requester, automatic approval, and a Ready ACME issuer. + try: + - script: + content: | + set -u + # --- discover the serving Secret from the Gateway itself ------------------- + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \ + 2>/dev/null | head -1)" + [ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; } + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \ + 2>/dev/null | awk '{print $1}')" + [ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; } + echo "gateway=$gns/$gname serving-secret=$sec" + + # --- provenance (1): only the controller writes this annotation ------------ + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)" + [ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; } + crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + [ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; } + + # --- provenance (2): the Certificate is Ready and has a revision ----------- + ready="$(kubectl -n "$gns" get certificate "$cert" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)" + echo "certificate=$gns/$cert ready=$ready revision=$rev" + [ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; } + [ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; } + + # --- provenance (3): the request was made BY THE CONTROLLER and auto-approved + # spec.username is the identity that created the CertificateRequest. A human + # operator would appear as a user/OIDC identity, not a ServiceAccount. + cr="$(kubectl -n "$gns" get certificaterequest \ + -o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \ + 2>/dev/null | tail -1)" + [ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; } + crname="${cr% *}"; requester="${cr#* }" + echo "certificaterequest=$crname requester=$requester" + case "$requester" in + system:serviceaccount:*) : ;; + *) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;; + esac + appr="$(kubectl -n "$gns" get certificaterequest "$crname" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + [ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; } + + # --- provenance (4): an automated ACME issuer, and it is Ready ------------- + ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')" + iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')" + echo "issuer=$ikind/$iname" + if [ "$ikind" = "ClusterIssuer" ]; then + iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + else + iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)" + iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)" + fi + [ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; } + [ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; } + echo "SERVING_CERT_CONTROLLER_ISSUED_OK" + check: + (contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true + + - name: then-an-unattended-renewal-is-already-scheduled + description: >- + cert-manager writes status.renewalTime only after computing when it will renew. + Asserting it is in the future and before expiry proves the renewal is scheduled + and that nothing is waiting on a human (docs/08 "renews well before expiry"). + try: + - script: + content: | + set -u + gw="$(kubectl get gateway -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)" + gns="${gw% *}"; gname="${gw#* }" + sec="$(kubectl -n "$gns" get gateway "$gname" \ + -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')" + cert="$(kubectl -n "$gns" get secret "$sec" \ + -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')" + renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')" + after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')" + [ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; } + [ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; } + now_s="$(date -u +%s)" + r_s="$(date -u -d "$renew" +%s)" + a_s="$(date -u -d "$after" +%s)" + echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after" + echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d" + [ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; } + [ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; } + echo "RENEWAL_SCHEDULED_OK" + check: + (contains($stdout, 'RENEWAL_SCHEDULED_OK')): true + + # ------------------------------------------------- MUTATING HALF (hermetic namespace) + - name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step + description: >- + Declare an Issuer and a Certificate in this test's own namespace, with a renewal + window that opens one minute after issuance. Nothing else is done to them for the + rest of the test — every later transition is the controller acting on its own. + try: + - apply: + resource: + # selfSigned: no external CA, no rate limit, no DNS write. See the header for + # what this proves (controller automation) and what it does not (the ACME path). + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: conformance-lifecycle + spec: + selfSigned: {} + - apply: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + spec: + secretName: conformance-lifecycle-tls + # 1h is cert-manager's minimum duration; renewing 59m before expiry puts the + # renewal instant ~1 minute after each issuance, so an unattended renewal is + # observable inside a test run instead of 30 days from now. + duration: 1h + renewBefore: 3540s + commonName: conformance-lifecycle.invalid + dnsNames: + - conformance-lifecycle.invalid + privateKey: + rotationPolicy: Always + issuerRef: + kind: Issuer + name: conformance-lifecycle + - assert: + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: conformance-lifecycle + status: + conditions: + - type: Ready + status: "True" + revision: 1 + - script: + content: | + set -u + crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)" + key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.key}' 2>/dev/null || true)" + [ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; } + # Record the serving bytes so the renewal step can prove they CHANGED. + echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum + echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)" + echo "CERT_ISSUED_OK" + check: + (contains($stdout, 'CERT_ISSUED_OK')): true + + - name: then-it-renews-itself-before-expiry-with-no-human-step + description: >- + Nothing touches the Certificate here. Poll until the controller has reissued it + at least twice on its own and the serving bytes differ from revision 1 — docs/08's + "observed across a renewal", and every request auto-approved. + try: + - script: + content: | + set -u + want=3 # revision 1 = issuance; >=3 means TWO unattended renewals + for i in $(seq 1 30); do + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + echo "poll $i: revision=${rev:-0}" + [ "${rev:-0}" -ge "$want" ] && break + sleep 20 + done + rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \ + -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)" + [ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; } + + # The serving material must actually have rotated, not just the revision count. + now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.data.tls\.crt}' | cksum)" + was="$(cat /tmp/conformance-cert-r1.cksum)" + echo "r1=$was now=$now" + [ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; } + + # Every renewal request was approved by the controller, never by a person. + bad=0 + for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do + st="$(kubectl -n "$NAMESPACE" get "$cr" \ + -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)" + who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)" + echo " $cr approved=$st requester=$who" + [ "$st" = "True" ] || bad=1 + case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac + done + [ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; } + echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)" + check: + (contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true + + - name: then-removing-the-certificate-withdraws-the-serving-material + description: >- + docs/08 reads revocation as "removing/rotating a certificate is a declarative + change". The declarative change is the deletion; withdrawal of the credential + must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs + `enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the + tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06). + try: + - delete: + ref: + apiVersion: cert-manager.io/v1 + kind: Certificate + name: conformance-lifecycle + - script: + content: | + set -u + for i in $(seq 1 15); do + if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then + echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)" + exit 0 + fi + sleep 4 + done + owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \ + -o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)" + echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still" + echo " holds usable certificate material (ownerReferences=${owners:-none})." + echo " Removing a certificate therefore does NOT withdraw the credential, so" + echo " R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:" + echo " set enableCertificateOwnerRef: true in platform/cert-manager values" + echo " (owner decision — WS-TEST does not weaken the assertion instead)." + exit 1 + check: + (contains($stdout, 'CERT_WITHDRAWN_OK')): true + finally: + # Namespace teardown removes both objects anyway; explicit so a leftover Secret + # from the expected failure above is never left holding key material. + - script: + content: | + kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true + kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true + rm -f /tmp/conformance-cert-r1.cksum || true diff --git a/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml index 62a892b..36e9be7 100644 --- a/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml @@ -153,3 +153,20 @@ spec: finally: - script: content: 'echo "cleanup handled by chainsaw namespace teardown"' + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo DEFAULT_DENY_PROVEN_OK + check: + (contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true diff --git a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml index 9b3eee0..244d542 100644 --- a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml @@ -3,11 +3,18 @@ # Catalog: docs/14 "DNS/exposure (R-DNS-*)" # Spec: docs/09 (ExternalDNS auto-manages records; split-horizon foundation) # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER +# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the +# expected answer is the LB IP that THIS Service was actually assigned (read back from its own +# status) — so neither the name nor the address is authored per environment, and no run-time flag +# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two +# environments from ever describing the same public name (the 2026-09-04 incident precondition). +# # GIVEN a LoadBalancer Service annotated with an external-dns hostname, # WHEN ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync), # THEN a PUBLIC DNS record for that hostname is auto-created, pointing at the -# Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the -# cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint. +# Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that +# Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint. # # CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown; # ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass. @@ -38,6 +45,13 @@ spec: apply: 90s assert: 5m exec: 6m + # -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST. + # flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this + # binding is the same source of truth the platform is rendered from — dev asks dev, prod + # asks prod, and there is no per-environment fork and no flag to forget. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: given-a-service-requesting-a-dns-name try: @@ -48,7 +62,8 @@ spec: metadata: name: dns-probe-svc annotations: - external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev + # conformance-dns. — never a literal apex. + external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain])) spec: type: LoadBalancer # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and @@ -59,7 +74,27 @@ spec: - name: http port: 80 targetPort: 80 + # Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will + # publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal + # keeps the expression nil-safe while the address is still absent, so the assert POLLS + # (fails and retries) instead of erroring out on the first evaluation. + - assert: + resource: + apiVersion: v1 + kind: Service + metadata: + name: dns-probe-svc + status: + (length(loadBalancer.ingress || `[]`)): 1 - name: then-externaldns-creates-the-public-record + # Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's + # assert guarantees it exists by now — step bindings are evaluated when the step runs). + # Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60 + # range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is + # 192.168.2.20-.30, and neither is written down here. + bindings: + - name: lbIp + value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip) try: - apply: resource: @@ -76,20 +111,27 @@ spec: containers: - name: probe image: busybox:1.36.1 + # Both values come from the cluster under test: the suffix from + # flux-system/cluster-vars, the address from this Service's own status. + env: + - name: BASE_DOMAIN + value: ($baseDomain) + - name: LB_IP + value: ($lbIp) command: - sh - -c - | set -u - host="conformance-dns.home.keithjackson.dev" + host="conformance-dns.${BASE_DOMAIN}" # Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and - # points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it. + # points at THIS Service's assigned LB IP. Proves ExternalDNS created it. for i in $(seq 1 28); do - ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)" + ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)" if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi - echo "poll $i: record not yet published"; sleep 10 + echo "poll $i: record not yet published as ${LB_IP}"; sleep 10 done - echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1 + echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1 - assert: resource: apiVersion: batch/v1 diff --git a/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml index ee68a36..ecb2e1a 100644 --- a/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml @@ -2,17 +2,24 @@ # L2 BEHAVIORAL — split-horizon INTERNAL resolution (R-DNS-01 / R-DNS-03) [RUNNABLE] # Catalog: docs/14 "DNS/exposure (R-DNS-*)" # Spec: docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve -# .home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin. +# .${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin. # Impl: platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin -# answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}). +# answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}). +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own +# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux +# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is +# per-environment and no flag is passed at run time, so the suite cannot be run against the +# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's. # # GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS, # WHEN an in-cluster pod resolves a platform hostname over its PUBLIC name -# (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10), -# THEN the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a -# public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the -# Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates -# (R-DNS-01 "services reachable at .home..."; R-DNS-03 split-horizon). +# (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10), +# THEN the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is +# templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster +# IP. This is what lets pod->public-hostname traffic hairpin through the +# Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates +# (R-DNS-01 "services reachable at .${BASE_DOMAIN}"; R-DNS-03 split-horizon). # # Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the # split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP @@ -34,13 +41,23 @@ metadata: conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- - An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal - Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution. + An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's + internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution. spec: timeouts: apply: 60s assert: 2m exec: 2m + # -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored + # here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from + # (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the + # same source of truth the platform is rendered from. Test-level => in scope for every step + # (unlike step `outputs`, which are step-scoped). + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) + - name: gatewayVip + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP) steps: - name: then-cluster-dns-answers-with-the-internal-gateway-vip try: @@ -59,23 +76,32 @@ spec: containers: - name: probe image: busybox:1.36.1 + # The cluster's OWN identity, resolved from flux-system/cluster-vars by the + # test-level bindings — never a literal (R-ENV-02). + env: + - name: BASE_DOMAIN + value: ($baseDomain) + - name: GATEWAY_VIP + value: ($gatewayVip) command: - sh - -c - | set -u - host="keycloak.home.keithjackson.dev" + host="keycloak.${BASE_DOMAIN}" # Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf -> - # 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP - # (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway. + # 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP + # (the address coredns-custom is templated with), proving pods hairpin + # through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the + # old hardcoded 192.168.1.53-.60 range AND correct in both environments. out="$(nslookup "$host" 2>/dev/null)" echo "$out" - ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)" + ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)" if [ -n "$ip" ]; then echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)" exit 0 fi - echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);" + echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});" echo " split-horizon coredns-custom not in effect (got a public/off-cluster answer)." exit 1 - assert: diff --git a/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml b/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml new file mode 100644 index 0000000..d2f6801 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml @@ -0,0 +1,199 @@ +# ============================================================================= +# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE] +# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster") +# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by +# routing rules ... realized as two paths — the local `internet` path and the +# in-cluster `vpn` path"). +# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is +# simply the default route out". +# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.** +# Every pod — and the nodes themselves — egress the local/private WAN by default and +# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the +# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0` +# uses the VPN." +# +# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT +# -------------------------------------------------------------- +# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their +# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing +# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds +# that pod, and asserts the two claims that make a rule-selected path set complete: +# +# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the +# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is +# the local one. If the steering rule ever widened (an empty selector, a +# matchExpression, a second rule), this pod's IP would move to the tunnel and +# this assertion is what would notice. +# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one +# pass: three pods, two distinct public IPs. "Multiple egress paths" is a +# simultaneity claim, not a claim that one switch has two positions. +# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918 +# destination over the LOCAL path while its public traffic is tunnelled. The +# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod +# uses BOTH paths at once. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a +# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike +# and survives the WAN or the VPN provider rotating addresses. +# +# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three +# times, differing only in the class they declare (or do not), +# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod +# also resolves a name through the in-cluster resolver, +# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out +# is the WAN, and the VPN is strictly opt-in); +# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path); +# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not +# swallowed by the tunnel), so the same pod is using both paths at once. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one +# steering rule is authored, that it selects only the vpn class, that no `ip rule` is +# unscoped and no nftables DROP unqualified is the L0 half, in +# tests/l0-static/egress-paths-selected-by-rules (same requirement label). +# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo +# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need +# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in +# this suite does. +# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn` +# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically +# by the L0 half, not here. +# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That +# is a host-level fact this test cannot see from a pod. +# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster, +# so a WAN outage or the echo being down fails this test. That is the same dependency +# tests/l2-behavioral/egress-class already carries; the alternative — trusting the +# cluster's own account of which path it used — would not be a behavioural test at all. +# +# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside +# that namespace is created or changed — no label on a platform object, no policy edit. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-default-path + labels: + conformance.home.k3s/R-NET-04: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-04" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public + IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the + default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two + paths live at once — and the vpn-class pod still reaches in-cluster destinations over the + local path, so one pod uses both paths simultaneously. +spec: + timeouts: + apply: 60s + assert: 2m + exec: 3m + steps: + # ----------------------------------------------------------------------------------- + - name: given-three-pods-differing-only-in-their-declared-egress-class + try: + - apply: + file: fixtures/three-egress-pods.yaml + # Bare pods (restartPolicy: Never) — Ready == Running here. + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-unlabelled } + status: { phase: Running } + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-internet } + status: { phase: Running } + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-vpn } + status: { phase: Running } + # ----------------------------------------------------------------------------------- + - name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too + try: + - script: + content: | + set -eu + ns="$NAMESPACE" + get_ip() { + # Echo the pod's observed public IP, or empty on failure. + kubectl -n "$ns" exec "$1" -c curl -- \ + curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true + } + + # Poll until all three answer AND the vpn class has diverged from the WAN: + # Cilium may still be programming the egress datapath just after the pods go + # Ready. Bounded by the loop, never a bare sleep-and-hope. + none="" inet="" vpn="" + for i in $(seq 1 12); do + [ -n "$none" ] || none="$(get_ip egress-unlabelled)" + [ -n "$inet" ] || inet="$(get_ip egress-internet)" + [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)" + if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then + [ "$vpn" != "$inet" ] && break + # Steering may not have converged; re-read the vpn pod only. + vpn="" + fi + sleep 5 + done + echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn" + + [ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; } + [ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; } + [ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; } + + # TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN — + # the same WAN the explicitly-internet pod uses. + if [ "$none" != "$inet" ]; then + echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2 + exit 1 + fi + # COEXISTENCE: and the vpn path is a genuinely different path, live right now. + if [ "$vpn" = "$inet" ]; then + echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2 + exit 1 + fi + echo "EGRESS_DEFAULT_PATH_IS_WAN_OK" + check: + (contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-the-vpn-class-pod-still-reaches-private-destinations-locally + try: + - script: + # PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all + # RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the + # local datapath. The cluster resolver is an in-cluster (RFC1918) address reached + # from the pod's own /etc/resolv.conf — no literal is written here — so a + # successful lookup from the vpn-class pod proves the tunnel did not swallow its + # local traffic, i.e. ONE pod is using BOTH paths at the same time. + content: | + set -eu + ns="$NAMESPACE" + resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ + sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)" + [ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; } + echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)" + + # A cluster-internal name, resolvable only by the in-cluster resolver: reaching + # it proves the private-destination path works from a vpn-class pod. + out="" + for i in $(seq 1 6); do + out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ + nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)" + case "$out" in *"Address"*) break ;; esac + sleep 5 + done + case "$out" in + *Address*) : ;; + *) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;; + esac + echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK" + check: + (contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true diff --git a/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml b/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml new file mode 100644 index 0000000..0815781 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml @@ -0,0 +1,67 @@ +# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing +# ONLY in how (or whether) they declare an egress class: +# +# egress-unlabelled NO egress-class label at all -> selected by NO routing rule +# egress-internet egress-class: internet -> the declared default path +# egress-vpn egress-class: vpn -> the only class any rule steers +# +# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress +# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no +# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must +# come out the same WAN as the explicitly-`internet` pod. That is the totality half of +# R-NET-04's "selected by routing rules" — absence of a rule is itself a path. +# +# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an +# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in +# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee +# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe +# the datapath's own default at all. +apiVersion: v1 +kind: Pod +metadata: + name: egress-unlabelled + labels: + app: egress-default-path-test +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never +--- +apiVersion: v1 +kind: Pod +metadata: + name: egress-internet + labels: + app: egress-default-path-test + homelab.keithjackson.dev/egress-class: internet +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never +--- +apiVersion: v1 +kind: Pod +metadata: + name: egress-vpn + labels: + app: egress-default-path-test + homelab.keithjackson.dev/egress-class: vpn +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never diff --git a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml index 151aa03..ccf8e95 100644 --- a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml @@ -68,6 +68,15 @@ spec: timeouts: apply: 60s assert: 180s + # -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener + # (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST + # (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than + # authored here. Test-level => in scope for the fixture apply AND every probe step below + # (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments). + # fixtures/app-and-probe.yaml consumes it via chainsaw resource templating. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: # -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend + # Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared @@ -139,12 +148,14 @@ spec: env: - name: GWVIP value: ($gwvip) + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -u - host="gw-declared-conformance.home.keithjackson.dev" + host="gw-declared-conformance.${BASE_DOMAIN}" last=000 i=0 while [ "$i" -lt 30 ]; do @@ -207,12 +218,14 @@ spec: env: - name: GWVIP value: ($gwvip) + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -u - host="gw-undeclared-conformance.home.keithjackson.dev" + host="gw-undeclared-conformance.${BASE_DOMAIN}" reached_gw=no i=0 while [ "$i" -lt 8 ]; do @@ -300,3 +313,20 @@ spec: apiVersion: v1 kind: Namespace name: conformance-gw-probe + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK + check: + (contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true diff --git a/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml b/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml index 02f83a8..01afc0b 100644 --- a/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml +++ b/conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml @@ -10,6 +10,13 @@ # The namespace opts into the shared Gateway (gateway-access: shared) exactly like a # rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the # probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in. +# +# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They +# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which +# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the +# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's +# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a +# hardcoded suffix silently broke this test on any cluster that did not hold the apex. --- apiVersion: v1 kind: Namespace @@ -134,7 +141,8 @@ spec: namespace: gateway sectionName: https hostnames: - - gw-declared-conformance.home.keithjackson.dev + # gw-declared-conformance. — see the header. + - (join('.', ['gw-declared-conformance', $baseDomain])) rules: - matches: - path: @@ -158,7 +166,8 @@ spec: namespace: gateway sectionName: https hostnames: - - gw-undeclared-conformance.home.keithjackson.dev + # gw-undeclared-conformance. — see the header. + - (join('.', ['gw-undeclared-conformance', $baseDomain])) rules: - matches: - path: diff --git a/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml b/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml new file mode 100644 index 0000000..925d7b2 --- /dev/null +++ b/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml @@ -0,0 +1,379 @@ +# ============================================================================= +# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane (R-PLAT-01) [RUNNABLE] +# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster") +# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with +# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST +# exercise the same multi-node topology.") +# docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02" +# docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA +# embedded etcd (k3s server on all 3)" +# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes" +# docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across +# all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload +# (3-node converged cluster)". +# +# WHY L2 — AND WHAT L0 ALREADY DID +# -------------------------------- +# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED +# half offline: the repo builds a 3-server embedded-etcd control plane in both environments, +# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving +# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory +# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test: +# every assertion below is read from the cluster the suite is pointed at. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written +# here. The expected node count comes from the cluster's OWN declared identity +# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same +# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip +# DaemonSet the cluster is actually running; the server addresses come from the Node +# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars +# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY +# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the +# environment from the cluster, never from a flag or a literal, which is precisely the +# mechanism the tests/README.md convention prescribes. +# +# GIVEN the cluster this suite is pointed at, +# WHEN its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected, +# THEN DISTRO: every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s; +# SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node +# topology docs/02 fixes — no agent-only node, no control-plane-only node), +# the count is ODD and quorum-capable (tolerates >= 1 loss), and it is +# EXACTLY three when the cluster declares itself prod; +# ETCD: every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps +# `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so +# the datastore is embedded and replicated across all of them; +# VIP: kube-vip is Ready on EVERY control-plane node, is control-plane-only +# (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and +# the address it floats is neither a node address, nor the ingress Gateway +# VIP, nor inside the Cilium LoadBalancer pool; +# ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that +# validates against the cluster CA for the VIP itself — i.e. the VIP really +# fronts the API and the serving cert really carries it; +# EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own +# address, again with a CA-valid cert for that address. This is the assertion +# that separates a real HA control plane from one server plus two joiners: +# if only the founder ran an apiserver, the other addresses would refuse. +# +# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node +# addresses from the RUNNER. That works from a host on the node LAN (verified against prod) +# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a +# forwarded port — i.e. it would have made the test a property of where it was launched +# from. A Job on the cluster is on the node network in both environments, so the probe is +# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than +# as a fixture file because its address list is DISCOVERED from the cluster at run time — +# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP +# migrates, and that the API stays reachable when a server dies is an L3 fault-injection +# claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss +# drains a node for R-STATE-02's data-availability claim and asserts nothing about the +# control plane). This is the largest remaining R-PLAT-01 gap. +# - THE NEXT environment. That the repo would build an HA control plane again is the L0 +# half (tests/l0-static/ha-control-plane-as-code). +# - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of +# infra/PROD-PROVISIONING.md). +# - THAT THE RUNNING cilium-agents ACTUALLY DIAL THIS VIP. As of 2026-09-06 the repository +# authors `K3S_API_HOST` = the control-plane VIP in both environments (asserted at L0 by +# tests/l0-static/ha-control-plane-as-code, on both sides of the [parity] contract), but +# this test does not read the agents' live `KUBERNETES_SERVICE_HOST`. Until the operator +# applies that change the LIVE agents still carry the PRIMARY NODE's address — measured +# 2026-09-06: all three prod agents on 192.168.2.10 while kube-vip held 192.168.2.9. An +# in-cluster assertion on the agents' endpoint belongs here and is NOT yet written; it is +# the obvious follow-up once the change is applied and would catch drift the L0 check +# cannot see (a hand-edited DaemonSet, a Helm value overridden out of band). +# - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes +# it only on the node-local etcd metrics port, which is not reachable through the +# Kubernetes API this test is allowed to use. +# +# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it +# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test +# creates nothing outside its own namespace and mutates no platform object. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: ha-control-plane + labels: + conformance.home.k3s/R-PLAT-01: "true" + annotations: + conformance.home.k3s/requirements: "R-PLAT-01" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver + endpoints are probed, then every node is a k3s server that is both control-plane and a + registered embedded-etcd member, the count is odd and quorum-capable (exactly three + when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that + collides with no allocator, and the apiserver answers /readyz through that VIP AND on + every server's own address with a CA-valid cert. +spec: + timeouts: + apply: 60s + assert: 2m + exec: 3m + # -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE + # CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes + # from, so these are the same values the platform was rendered with. Test-level, so + # they are in scope for every step (step outputs/bindings are step-scoped). + bindings: + - name: clusterEnv + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV) + - name: gatewayVip + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP) + - name: lbPoolStart + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START) + - name: lbPoolStop + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP) + steps: + # ----------------------------------------------------------------------------------- + - name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane + description: >- + Read the Node objects: k3s on every node, every node both control-plane and a + registered embedded-etcd member, an odd quorum-capable count, exactly three in prod. + try: + - script: + env: + - name: CLUSTER_ENV + value: ($clusterEnv) + content: | + set -eu + fail() { echo "FAIL: $*" >&2; exit 1; } + + total="$(kubectl get nodes -o name | grep -c . || true)" + [ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all" + + # --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) ----- + # A non-k3s kubelet here would mean the platform is not running on k3s. + notk3s="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \ + | grep -v '+k3s' | grep -c . || true)" + [ "$notk3s" = "0" ] \ + || fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s" + + # --- every node is BOTH control-plane AND etcd (docs/02 converged topology) - + # Counted by label EXISTENCE, so it does not matter whether k3s writes "true" + # or "". A control-plane node without the etcd role would be a server with an + # external/absent datastore; an etcd node without control-plane would be an + # isolated datastore member. Both are excluded by requiring all three counts + # to be equal. + cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)" + etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)" + echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-}" + [ "$cp" = "$total" ] \ + || fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')" + [ "$etcd" = "$total" ] \ + || fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'" + + # --- every node is a REGISTERED embedded-etcd member ----------------------- + # k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the + # embedded-etcd cluster. The role label alone is a label; this annotation is + # the datastore's own record of membership. + members="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \ + | grep -c . || true)" + [ "$members" = "$total" ] \ + || fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server" + + # --- Ready --------------------------------------------------------------- + notready="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \ + | grep -v '=True$' | grep -c . || true)" + [ "$notready" = "0" ] \ + || fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test" + + # --- quorum arithmetic --------------------------------------------------- + # Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is + # N - quorum; 'production-grade HA' must survive at least one loss, which + # rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds + # a tie, so it is also rejected. + quorum=$(( total / 2 + 1 )) + tolerance=$(( total - quorum )) + echo "quorum=$quorum tolerance=$tolerance" + [ "$tolerance" -ge 1 ] \ + || fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server" + [ $(( total % 2 )) -eq 1 ] \ + || fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance" + + # --- the count R-PLAT-01 itself fixes, per environment -------------------- + # The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a + # flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST + # exercise the same multi-node topology'. + case "${CLUSTER_ENV:-}" in + prod) + [ "$total" = "3" ] \ + || fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;; + "") + fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;; + *) + [ "$total" -ge 2 ] \ + || fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;; + esac + echo "HA_TOPOLOGY_OK" + check: + (contains($stdout, 'HA_TOPOLOGY_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-a-control-plane-only-vip-is-floated-on-every-server + description: >- + Read the running kube-vip DaemonSet: Ready on every control-plane node, + control-plane-only, and floating an address that collides with no other allocator. + try: + - script: + env: + - name: GATEWAY_VIP + value: ($gatewayVip) + - name: LB_POOL_START + value: ($lbPoolStart) + - name: LB_POOL_STOP + value: ($lbPoolStop) + content: | + set -eu + fail() { echo "FAIL: $*" >&2; exit 1; } + # dotted quad -> integer, for range comparison without ipcalc/python. + n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; } + + # The DaemonSet is found by the label roles/kube_vip stamps on it, not by a + # hardcoded name in a namespace this test guessed. + ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)" + [ -n "$ds" ] \ + || fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it" + + desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')" + ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')" + cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)" + vip="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')" + cp_enable="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')" + svc_enable="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')" + echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable" + + [ -n "$vip" ] \ + || fail "the kube-vip DaemonSet announces no address — nothing is floated" + [ "$cp_enable" = "true" ] \ + || fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP" + [ "$svc_enable" = "false" ] \ + || fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)" + [ "$desired" = "$cp" ] \ + || fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane" + [ "$ready" = "$desired" ] \ + || fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over" + + # --- the VIP collides with nothing -------------------------------------- + node_ips="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \ + | grep -c "^${vip}$" || true)" + [ "$node_ips" = "0" ] \ + || fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node" + [ "$vip" != "${GATEWAY_VIP:-}" ] \ + || fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it" + if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then + v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")" + [ "$lo" -le "$hi" ] \ + || fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it" + if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then + fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service" + fi + else + fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)" + fi + echo "HA_VIP_CONTROL_PLANE_ONLY_OK" + check: + (contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-the-apiserver-answers-through-the-vip-and-on-every-server + description: >- + From inside the cluster, probe /readyz on the floated VIP and on every server's own + address, validating TLS against the cluster CA for the address dialled. + try: + - script: + # The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the + # Job cannot be a static fixture — writing the addresses down is the + # per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the + # same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped + # fixtures. The Job lives in chainsaw's per-test namespace and is torn down with + # it; nothing outside that namespace is created or changed. + content: | + set -eu + ns="$NAMESPACE" + fail() { echo "FAIL: $*" >&2; exit 1; } + + ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)" + vip="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')" + servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \ + -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')" + [ -n "$vip" ] || fail "no floated VIP to probe" + [ -n "$servers" ] || fail "no control-plane node addresses to probe" + echo "probing vip=$vip servers=$servers" + + # The probe pod authenticates with its own ServiceAccount token and validates + # TLS against the cluster CA, both from the projected SA volume. /readyz is + # authorized for any authenticated identity via the built-in + # system:public-info-viewer role — anonymous access is NOT relied on (verified: + # an unauthenticated /readyz on this platform answers 401). + kubectl -n "$ns" apply -f - <&1)" || rc=1 + echo "vip \$VIP -> [\$out]" + if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then + echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)" + exit 1 + fi + for s in \$SERVERS; do + rc=0 + out="\$(probe "\$s" 2>&1)" || rc=1 + echo "server \$s -> [\$out]" + if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then + echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node" + exit 1 + fi + done + echo "HA_API_ENDPOINT_OK" + EOF + + # Bounded wait; on failure surface the probe's own output before failing. + if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then + kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true + fail "the in-cluster API-endpoint probe did not complete — see its log above" + fi + logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)" + echo "$logs" + case "$logs" in + *HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;; + *) fail "the probe completed without reporting success" ;; + esac + check: + (contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true diff --git a/conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml b/conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml new file mode 100644 index 0000000..fd85f39 --- /dev/null +++ b/conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml @@ -0,0 +1,212 @@ +# ============================================================================= +# L2 BEHAVIORAL — every OIDC client's STORED secret equals the one git declares +# (R-IDP-02, R-IDP-04) [RUNNABLE] +# Spec: docs/15 identity-and-access; ADR-0014. +# Impl: platform/keycloak/client-secret-reconciler.yaml (the CronJob + its script). +# +# WHY THIS TEST EXISTS +# -------------------- +# The KeycloakRealmImport resolves each confidential client's secret from a `(env:NAME)` +# placeholder mapped to a key of the SOPS-encrypted keycloak-realm-clients Secret. When +# the Keycloak Operator cannot resolve a placeholder it stores the LITERAL placeholder +# text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; the CR then rests +# at Done=True / Started=False and never re-runs. Measured 2026-09-06: prod had all 12 +# clients holding the literal placeholder — every OIDC login on the cluster failed with +# "Invalid client or Invalid client credentials" — and dev had one (oauth2-proxy), whose +# Secret key was added after dev's import had already completed. +# +# Every SIGNAL the cluster offered was green: the import Job succeeded, the CR read +# Done=True, the Keycloak pods were healthy, and the existing keycloak-config-as-code +# test (which asserts Done=True) PASSED throughout. The only observable was a human +# failing to log in. This test is that observable, automated: it compares STORED against +# DECLARED for every client and fails on any difference, so the class cannot sit +# undetected for months again. +# +# GIVEN the `homelab` realm and the SOPS-encrypted keycloak-realm-clients Secret, +# WHEN the reconciler's script is run in --audit mode (READ-ONLY: it cannot write, and +# its endpoint allow-list makes users/groups/roles unreachable), +# -> THEN every declared client's stored secret matches git, and none holds an +# unresolved import placeholder. Emits KEYCLOAK_CLIENT_SECRETS_AUDIT_OK. +# +# WHY IT RUNS THE SHIPPED SCRIPT rather than re-implementing the comparison: a detector +# that duplicates the healer's logic drifts from it. This audits with the same code, in +# the same mode the L0 gate selftests, so a passing audit is evidence about the healer +# too. It runs as a Job IN the keycloak namespace (not in the ephemeral chainsaw +# namespace) because the desired state is the mounted keycloak-realm-clients Secret and +# the admin credential is keycloak-admin-credentials — both namespace-local, which is +# also why neither this Job nor the CronJob needs any Kubernetes API access. +# +# chainsaw v0.2.12: Job + status assert; the leftover-Job delete keeps a re-run from +# colliding with an immutable Job left by a previous failure. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: keycloak-client-secrets-match-git + labels: + conformance.home.k3s/R-IDP-02: "true" + conformance.home.k3s/R-IDP-04: "true" + annotations: + conformance.home.k3s/requirements: "R-IDP-02, R-IDP-04" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every OIDC client's stored secret equals the SOPS-declared one and none holds an + unresolved import placeholder — the observable that was missing when all 12 prod + clients were broken behind a green realm import. +spec: + timeouts: + apply: 60s + assert: 5m + exec: 5m + steps: + - name: given-the-reconciler-is-reconciled-onto-the-cluster + description: >- + The audit Job runs the CronJob's own script from its ConfigMap; if Flux has not + applied the reconciler, there is nothing to audit WITH and that is the failure. + try: + - assert: + resource: + apiVersion: v1 + kind: ConfigMap + metadata: + name: keycloak-client-secret-reconciler + namespace: keycloak + - assert: + resource: + apiVersion: batch/v1 + kind: CronJob + metadata: + name: keycloak-client-secret-reconciler + namespace: keycloak + spec: + concurrencyPolicy: Forbid + - name: then-every-stored-client-secret-matches-git + try: + - script: + content: | + kubectl -n keycloak delete job keycloak-client-secret-audit \ + --ignore-not-found --wait=true + - apply: + resource: + apiVersion: batch/v1 + kind: Job + metadata: + name: keycloak-client-secret-audit + namespace: keycloak + labels: + app.kubernetes.io/name: keycloak-client-secret-audit + app.kubernetes.io/part-of: conformance + spec: + backoffLimit: 0 + activeDeadlineSeconds: 240 + template: + metadata: + labels: + app.kubernetes.io/name: keycloak-client-secret-audit + spec: + restartPolicy: Never + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + seccompProfile: + type: RuntimeDefault + containers: + - name: audit + image: python:3.13.15-alpine3.23 + command: + - python3 + - /opt/reconciler/reconcile-client-secrets.py + # READ-ONLY. --audit never issues a write on any path; the + # script's --selftest proves that (L0 gate). + - --audit + env: + - name: KEYCLOAK_BASE_URL + value: http://keycloak-service.keycloak.svc.cluster.local:8080 + - name: KEYCLOAK_REALM + value: homelab + - name: PYTHONDONTWRITEBYTECODE + value: "1" + - name: PYTHONUNBUFFERED + value: "1" + - name: KEYCLOAK_ADMIN_USER + valueFrom: + secretKeyRef: + name: keycloak-admin-credentials + key: admin-user + - name: KEYCLOAK_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: keycloak-admin-credentials + key: admin-password + volumeMounts: + - name: script + mountPath: /opt/reconciler + readOnly: true + - name: client-secrets + mountPath: /etc/keycloak-realm-clients + readOnly: true + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 48Mi + limits: + memory: 128Mi + volumes: + - name: script + configMap: + name: keycloak-client-secret-reconciler + defaultMode: 0444 + - name: client-secrets + secret: + secretName: keycloak-realm-clients + defaultMode: 0400 + - assert: + resource: + apiVersion: batch/v1 + kind: Job + metadata: + name: keycloak-client-secret-audit + namespace: keycloak + status: + succeeded: 1 + # The Job's exit status already encodes the verdict, but the sentinel is what the + # run LEDGER records (conformance/runs/*/run.log): a JUnit alone cannot tell a real + # audit from a Job that succeeded without auditing anything. + - script: + content: | + set -u + out="$(kubectl -n keycloak logs job/keycloak-client-secret-audit \ + --tail=-1 2>&1 || true)" + echo "$out" + echo "$out" | grep -q 'KEYCLOAK_CLIENT_SECRETS_AUDIT_OK' \ + || { echo "FAIL: the audit did not report every client secret matching git"; exit 1; } + echo "$out" | grep -q 'UNRESOLVED IMPORT PLACEHOLDER' \ + && { echo "FAIL: a client still holds the literal import placeholder"; exit 1; } + echo KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK + check: + (contains($stdout, 'KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK')): true + catch: + - describe: + apiVersion: batch/v1 + kind: Job + namespace: keycloak + name: keycloak-client-secret-audit + - podLogs: + namespace: keycloak + selector: app.kubernetes.io/name=keycloak-client-secret-audit + tail: 60 + finally: + - script: + content: | + kubectl -n keycloak delete job keycloak-client-secret-audit \ + --ignore-not-found --wait=false diff --git a/conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml b/conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml index d9b6963..946b306 100644 --- a/conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml @@ -11,6 +11,18 @@ # # Goes GREEN once Keycloak is up + the realm client-secret hand-off (keycloak-realm-clients) is filled # so the placeholder resolves and the import completes. chainsaw v0.2.12: script + block contains check. +# +# NOT PROVEN HERE — READ THIS BEFORE TRUSTING A GREEN RUN (added 2026-09-06). +# Done=True means the import JOB EXITED 0. It does NOT mean the placeholders resolved: when the +# operator cannot resolve one it stores the LITERAL `(env:...)` text as the client secret and the +# import still reports success, then rests at Done=True / Started=False and never re-runs. THIS +# TEST PASSED THROUGHOUT the 2026-09-06 outage in which all 12 prod clients held the literal +# placeholder and every OIDC login on the cluster failed. Done=True is a necessary, badly +# insufficient condition. +# The sufficient check is tests/l2-behavioral/keycloak-client-secrets-match-git, which compares +# each client's STORED secret against the SOPS-declared one; the static half is +# tests/l0-static/keycloak-client-secret-reconciler. Do not "simplify" by folding those into this +# test: the whole lesson is that the import's own success signal is not evidence. # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test diff --git a/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml b/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml index 6fb93c1..4ef93fd 100644 --- a/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml @@ -3,10 +3,15 @@ # Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client # (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof. # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the +# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by +# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which +# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster. +# # GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm, # WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth) # -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on -# keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP +# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP # (R-IDP-01: central Identity Provider as the single user directory). # # Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up. @@ -30,6 +35,11 @@ spec: apply: 60s assert: 3m exec: 3m + # -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST + # (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from). + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: given-grafana-is-an-oidc-client try: @@ -59,6 +69,9 @@ spec: containers: - name: probe image: busybox:1.36.1 + env: + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c @@ -68,13 +81,14 @@ spec: # busybox wget has no --max-redirect; -S dumps every response header # (incl. the initial 302 Location to the IdP) before following, so the # grep below still matches the Keycloak authorize URL. (Grafana verified - # live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.) + # live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.) + idp="keycloak.${BASE_DOMAIN}/realms/homelab" out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)" echo "$out" - if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then - echo GRAFANA_OIDC_REDIRECT_OK + if echo "$out" | grep -qiF "$idp"; then + echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}" else - echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1 + echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1 fi - assert: resource: diff --git a/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml index 79d6ee2..24a72ad 100644 --- a/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml @@ -3,8 +3,13 @@ # Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access. # Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014. # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its +# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and +# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform +# substitutes; it is prose, not an input — nothing here needs parameterizing. +# # GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available -# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is +# after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is # the proxy->Keycloak flow working), # WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth) # -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal diff --git a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml index d6c7f71..babdbaf 100644 --- a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml @@ -3,8 +3,13 @@ # Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" # Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers) # +# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER +# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a +# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and +# prod's on prod, with nothing to pass at run time. +# # GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert, -# WHEN an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and +# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and # verifies the chain against the SYSTEM trust store, # THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and # chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers @@ -36,6 +41,11 @@ metadata: that chains to a publicly-trusted CA root (LE production); the wildcard SAN covers internal-only hostnames too. spec: + # -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars + # — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment. + bindings: + - name: baseDomain + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN) steps: - name: then-gateway-cert-chains-to-trusted-root try: @@ -54,13 +64,16 @@ spec: containers: - name: probe image: curlimages/curl:8.11.1 + env: + - name: BASE_DOMAIN + value: ($baseDomain) command: - sh - -c - | set -eu # Probe TWO wildcard hostnames — one "public", one "internal-only". Both - # match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA + # match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA # bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND # R-TLS-03 (the same trusted wildcard covers internal-only names). Names # resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no @@ -68,7 +81,7 @@ spec: # the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or # hostname fails. TLS terminates at the gateway before HTTP routing, so no # backend is needed. - for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do + for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \ -o /dev/null "https://${host}/" echo "TLS_TRUSTED ${host}" @@ -80,3 +93,20 @@ spec: kind: Job metadata: { name: tls-probe } status: { succeeded: 1 } + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo TLS_TRUSTED_PROVEN_OK + check: + (contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true diff --git a/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml index 815ccce..76b205d 100644 --- a/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml +++ b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml @@ -40,10 +40,17 @@ # alternative source for the data. # # KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them): -# * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are -# VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9), -# so this drill does not survive loss of that hypervisor. Off-site/off-host -# replication is an operator concern outside R-STATE-03's wording. +# * OFF-HOST is NOT proven, only OFF-CLUSTER — TRUE AS RUN, and being fixed. On the +# dev estate the k3s nodes are VMs on the same Proxmox host that runs the +# versitygw/MinIO S3 target (192.168.1.9), so this drill does not survive loss of +# that hypervisor: one machine takes the cluster AND its backups. +# That is exactly why the operator moved off-cluster DR to Cloudflare R2 on +# 2026-09-05 (commit 32d6766 — S3_BACKUP_ENDPOINT is now an R2 endpoint). Once that +# merges and this drill is RE-RUN against R2, off-host stops being a gap and becomes +# proven, because the target no longer shares a failure domain with the cluster. +# UNTIL THEN THIS TEST'S PASS IS AGAINST MinIO. R-STATE-03 is currently proven only +# for the same-host target; re-proving it against R2 is a prerequisite for claiming +# off-host DR, and the ledger row that proves R-STATE-03 predates the move. # * The test does NOT read the backup objects out of the bucket with an S3 client. # Doing so would require handing the conformance runner the backup credential, # widening a privileged test identity for weaker evidence than (1)-(4) above. diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 562ec54..e726ec4 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -31,10 +31,17 @@ k3s_secondary [hypervisor] localhost ansible_connection=local -# VPN egress-gateway node(s) — run the node-level split-tunnel OpenVPN + kill-switch -# (ADR-0015, playbooks/vpn.egress.gateway.yml). Initially just k3s-dev-3 (single gateway; -# add a second host here for Cilium Egress Gateway HA). Members are already defined above -# with their ansible_host under k3s_secondary; this group only tags the egress role target. +# VPN egress-gateway node — runs the node-level split-tunnel OpenVPN + kill-switch +# (ADR-0015, playbooks/vpn.egress.gateway.yml). EXACTLY ONE host belongs here. +# +# THIS GROUP IS SINGLETON BY DESIGN — a second host does NOT buy failover. Cilium OSS has no +# egress-gateway failover: the CRD's own `egressGateway` description says "In case multiple nodes +# are a match for the given set of labels, the first node in lexical ordering based on their name +# will be selected". Two labelled nodes therefore means one silently wins by NAME ORDER (a labelled +# `k3s-prod-1` beats a labelled `k3s-prod-3`) and the other carries no traffic. See +# platform/vpn-egress/README.md "Resilience — there is no gateway failover" for the full statement +# and the ADR-0016 escalation. Members are already defined above with their ansible_host under +# k3s_secondary; this group only tags the egress role target. [vpn_egress_gateway] k3s-dev-3 diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index 4fb3962..6726e29 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -43,6 +43,11 @@ resources: - ../platform/longhorn/volumesnapshotclass.yaml # monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack) - ../platform/monitoring/alerts.yaml + # Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule + # (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC + # was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the + # file header. (2026-09-06, from the CNPG WAL/archiving incident.) + - ../platform/monitoring/pvc-capacity-alerts.yaml # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the # operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform- # installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain @@ -73,9 +78,21 @@ resources: # keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak # operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor. # ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom. + # cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a + # CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass — + # identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG + # clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted. + - ../platform/cnpg/alerts.yaml - ../platform/keycloak/postgres-cluster.yaml - ../platform/keycloak/keycloak-instance.yaml - ../platform/keycloak/keycloak-realm-import.yaml + # The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is + # stored LITERALLY and the import still reports success, then rests at Done=True and + # never re-runs. This CronJob is the level-triggered loop that corrects that drift + # (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform, + # so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and + # alongside the import whose failure mode it covers. + - ../platform/keycloak/client-secret-reconciler.yaml - ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC) - ../platform/keycloak/httproute.yaml - ../platform/keycloak/servicemonitor.yaml diff --git a/platform/cnpg/alerts.yaml b/platform/cnpg/alerts.yaml new file mode 100644 index 0000000..9f2910c --- /dev/null +++ b/platform/cnpg/alerts.yaml @@ -0,0 +1,182 @@ +--- +# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals. +# +# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db, +# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with +# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP +# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert +# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06). +# +# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres +# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the +# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB +# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO +# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it. +# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early +# warning for a storage outage. The generic PVC-fill backstop is the separate +# platform/monitoring/pvc-capacity-alerts.yaml. +# +# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, +# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the +# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and +# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the +# operator's Prometheus select these rules (ruleSelector). It IS referenced from +# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't" +# failure this repo keeps hitting. +# +# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only, +# 2026-09-06). CNPG's exporter publishes TWO independent families: +# +# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status +# and the backup catalogue). Present for EVERY instance. +# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED. +# +# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db +# exporter cannot log in to its own database: +# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter] +# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all. +# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND +# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition +# itself so the blind spot cannot recur unnoticed. +# +# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not): +# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3 +# dev: 0, 0, 0 (min over 24h also 0) +# cnpg_collector_first_recoverability_point prod: 0, 0, 0 +# dev: 1788018752 / 1788018762 / 1788018770 +# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old +# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet) +# cnpg_last_error prod+dev: nextcloud-db = 1 +# +# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate. +# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already +# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in +# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually +# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml — +# which fires only when an outage is genuinely imminent. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: platform-health-cnpg + namespace: monitoring + labels: + release: kube-prometheus-stack + platform-health: "true" +spec: + groups: + - name: platform-health.cnpg + rules: + # --------------------------------------------------------------------------------------- + # THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within + # the hour instead of 38 hours later. + # + # pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet + # archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within + # seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck + # segment is 16 MiB that Postgres may not recycle. + # + # THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the + # condition to hold at EVERY evaluation for the full hour, so a write burst that transiently + # queues a segment resets the timer and never notifies; but a genuinely stuck archiver never + # returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over + # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3). + # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its + # backlog sits at 3 because it writes almost nothing. + - alert: CNPGWALArchiveBacklog + expr: | + max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 + for: 1h + labels: + severity: warning + annotations: + summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)" + description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" + # --------------------------------------------------------------------------------------- + # The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in + # time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint + # is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can + # be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires), + # and an old base backup can exist while archiving has just broken (only the other fires). + # + # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands + # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold + # start quiet while still catching a bootstrap that never produced a recoverability point. + - alert: CNPGNoRecoverabilityPoint + expr: | + max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0 + for: 6h + labels: + severity: warning + annotations: + summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}" + description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving" + # --------------------------------------------------------------------------------------- + # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in + # platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these + # databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a + # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by + # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be + # ~57 years and always true). 36h = one missed daily run plus slack. + - alert: CNPGNoRecentBaseBackup + expr: | + (time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600) + and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0 + for: 2h + labels: + severity: warning + annotations: + summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}" + description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed" + # --------------------------------------------------------------------------------------- + # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for, + # which means checkpoints are no longer reclaiming space — the mechanism that actually fills + # the PVC. Both series come from the same metric under different `value` labels, so each side + # is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching + # label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on + # both clusters), `count` = segments currently in pg_wal. + # + # This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to: + # CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough + # to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks — + # at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC + # runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on + # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native + # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is + # therefore NOT usable as a signal. + - alert: CNPGWALVolumeNotRecycling + expr: | + max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) + > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) + for: 2h + labels: + severity: warning + annotations: + summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)" + description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth" + # --------------------------------------------------------------------------------------- + # THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user + # queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while + # `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health. + # + # THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live + # 2026-09-06: `cnpg_errors_total` carries + # pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter + # database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) + # i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all. + # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod + # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the + # permanently-firing `critical` this repo has been burned by. + - alert: CNPGMetricsCollectorErroring + expr: | + max by (namespace, pod) (cnpg_last_error) == 1 + for: 6h + labels: + severity: warning + annotations: + summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}" + description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter" diff --git a/platform/keycloak/README.md b/platform/keycloak/README.md index 54aad4d..7fe518e 100644 --- a/platform/keycloak/README.md +++ b/platform/keycloak/README.md @@ -29,6 +29,7 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai | `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config | | `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config | | `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config | +| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config | | `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config | | `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config | @@ -47,10 +48,14 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO ## Operator hand-offs (fill before / during deploy) -1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. Two client - secrets (`grafana-client-secret`, `oauth2-proxy-client-secret`), `openssl rand -hex 32` each. The - realm import injects them into the Keycloak clients via `spec.placeholders`. Add to - `kustomization.yaml` once filled. The realm import stays PENDING until this exists. +1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One + `-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm + import injects them into the Keycloak clients via `spec.placeholders`. Add to + `kustomization.yaml` once filled. + **The key name MUST be `-client-secret`** — the reconciler below derives the clientId + from it, and CI rejects any other name. + *(This step used to read "the realm import stays PENDING until this exists". It does not — see + the next section. That false belief is what allowed the 2026-09-06 outage.)* 2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`, `client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`. 3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME @@ -60,6 +65,45 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the exact keys + steps. No plaintext secret is ever committed. +## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06) + +**The realm import is a one-shot that fails open.** Measured on both clusters: + +* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder + text** as the client secret — and the import Job still exits 0. There is no PENDING state, no + error condition, and `HasErrors` stays `False`. +* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key + added later is never picked up. + +Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; +every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client +on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster +signal was green throughout, including the conformance test that asserts `Done=True`. + +Three layers now cover it: + +| Layer | Where | What it does | +|---|---|---| +| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. | +| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). | +| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. | + +The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped +ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm), +no-secret-logging, the endpoint allow-list, and fail-closed behaviour. + +**Manual repair / one-off run** (the CronJob normally does this): + +```sh +kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s) +kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1 +``` + +**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed +redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on +its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit +landed. + ## Exceptions that keep native auth (do not force SSO) — ADR-0014 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native). diff --git a/platform/keycloak/client-secret-reconciler.yaml b/platform/keycloak/client-secret-reconciler.yaml new file mode 100644 index 0000000..5047110 --- /dev/null +++ b/platform/keycloak/client-secret-reconciler.yaml @@ -0,0 +1,720 @@ +--- +# ============================================================================= +# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04) +# +# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only +# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients +# Secret and the Keycloak Operator, and alongside the realm import itself. +# +# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06) +# ------------------------------------------------------------------- +# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are +# import placeholders resolved from the keycloak-realm-clients Secret via +# spec.placeholders. When the operator cannot resolve a placeholder it stores the +# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS +# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and +# NEVER RE-RUNS, so nothing ever corrects it: +# +# * prod: all 12 clients held the literal placeholder. Every OIDC login on the +# cluster failed with "Invalid client or Invalid client credentials". +# * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added +# AFTER dev's realm import had already completed, and a completed import is +# never re-run, so the new key was never picked up. +# +# Both were repaired live via the admin API on 2026-09-06 and verified. This +# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is +# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same +# posture Flux gives every other resource in this repo (R-GITOPS-03 drift +# correction, applied to the one piece of state Flux itself cannot reach -- +# Keycloak's own database). +# +# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph +# already guarantees the Secret is applied before the realm import (the Secret is +# rendered by ./platform; the import by ./platform-config, which dependsOn +# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of +# the Secret lands before revision N of the import). Ordering still cannot help, +# because the operator will not re-run a completed import when a LATER revision +# adds a key -- which is precisely how dev broke. Only a reconciler closes that. +# +# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state +# lives in Keycloak's Postgres database, not in a Kubernetes object, so no +# admission or mutation policy can see it. The admin REST API is the only +# interface to it. +# +# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token, +# client lookup, client-secret read, minimal client PUT). Users, groups, roles, +# authentication flows and realm settings are structurally unreachable from it -- +# prod's `homelab` realm carries a real user and the groups media-users / +# platform-admins / viewers, and they must survive every run. It never logs a +# secret, and it writes only where the stored value differs from git. +# +# PROOF: the script's own `--selftest` drives these real code paths against an +# in-process stub Keycloak and asserts compare-then-write, idempotence (a second +# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint +# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script +# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships: +# conformance/tests/l0-static/keycloak-client-secret-reconciler/. +# ============================================================================= +apiVersion: v1 +kind: ConfigMap +metadata: + name: keycloak-client-secret-reconciler + namespace: keycloak + labels: + app.kubernetes.io/name: keycloak-client-secret-reconciler + app.kubernetes.io/part-of: keycloak +data: + # SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy: + # the L0 gate extracts THIS block and executes its --selftest, so a second copy + # would be untested. Contains no dollar sign at all, on purpose -- this file is + # rendered by a Flux Kustomization with postBuild.substituteFrom, so an + # unescaped dollar-brace expansion here would be rewritten before any container + # ran (the trap documented at length in apps/wave-1/nextcloud.yaml). + reconcile-client-secrets.py: | + #!/usr/bin/env python3 + # ============================================================================= + # reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret + # equal the value git declares for it, idempotently, without touching anything + # else in the realm. + # + # WHY THIS EXISTS (the incident it prevents) + # ------------------------------------------ + # keycloak-realm-import.yaml declares each confidential client's secret as an + # `(env:_CLIENT_SECRET)` import placeholder and maps to a key of the + # SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the + # Keycloak Operator cannot resolve a placeholder at import time it stores the + # LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS + # SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and + # never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12 + # clients with secrets held the literal placeholder and every OIDC login failed + # with "Invalid client or Invalid client credentials"; on dev exactly one + # (oauth2-proxy) was broken, because that Secret key was added after dev's import + # had already completed. + # + # So the import is a ONE-SHOT that fails OPEN. This is the level-triggered + # reconciler that closes it. + # + # CONTRACT + # -------- + # Desired state files in SECRETS_DIR named `-client-secret`, each + # holding exactly the secret for that clientId (this is the + # `keycloak-realm-clients` Secret, mounted). + # Actual state GET /admin/realms//clients//client-secret + # Action only where they differ: PUT /admin/realms//clients/ + # with the MINIMAL representation {"clientId":..,"secret":..}. + # + # WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a + # two-field PUT rewrites the secret and nothing else. PUTting the full + # round-tripped representation back has a far larger blast radius and is in fact + # rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the + # JSON"} (measured during the live repair). The stub in --selftest models that + # rejection, so a regression to the fat PUT fails in CI, not on a live realm. + # + # SAFETY PROPERTIES, each enforced in code and proved by --selftest: + # 1. compare-then-write: a client already correct is SKIPPED, no PUT issued. + # 2. idempotent: a second run over a converged realm issues zero writes. + # 3. no secret is ever printed, in any mode, on any path. + # 4. the ONLY endpoints this program may touch are the master-realm token + # endpoint, the client LOOKUP, the client-secret READ and the minimal client + # PUT. Every other URL is refused by _http() before a socket is opened, so + # users, groups, roles, authentication flows and realm settings are + # structurally unreachable from here -- prod's `homelab` realm carries a + # real user and the groups media-users / platform-admins / viewers, and they + # MUST survive. + # 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped. + # + # MODES + # (default) reconcile: compare, and write only what differs. + # --audit READ-ONLY. Report drift and exit 1 if any client's stored secret + # differs from the declared one. Never writes. This is what the L2 + # conformance test runs. + # --selftest the offline behavioural proof, against an in-process stub + # Keycloak. No cluster; loopback only. This is what the L0 static + # gate runs, so the code the gate proves is the code that ships. + # + # NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor + # (hence re.fullmatch throughout), and not even inside these comments. It ships + # inside a ConfigMap that the `platform-config` Flux Kustomization renders with + # postBuild.substituteFrom, so Flux envsubst rewrites this text before any + # container runs: an unescaped dollar-brace expansion would be silently emptied at + # reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's + # OIDC guard dead code for months -- read the comment block there. Python needs no + # dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and + # the character is banned outright so no future edit has to reason about escaping. + # Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh. + # ============================================================================= + from __future__ import annotations + + import hmac + import json + import os + import re + import sys + import time + import urllib.error + import urllib.parse + import urllib.request + + SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients") + KEY_SUFFIX = "-client-secret" + TIMEOUT = 20 + READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24")) + READY_SLEEP = float(os.environ.get("READY_SLEEP", "5")) + + # Endpoint allow-list (safety property 4). fullmatch-ed against the path of every + # URL _http() is asked to open; anything else raises before a socket exists. + _ALLOWED = ( + r"/realms/master/protocol/openid-connect/token", + r"/admin/realms/[^/]+/clients", + r"/admin/realms/[^/]+/clients/[^/]+", + r"/admin/realms/[^/]+/clients/[^/]+/client-secret", + ) + + + class Refused(RuntimeError): + """A URL outside the allow-list was requested. Never expected in a normal run.""" + + + def _http(method, base, path, query=None, headers=None, form=None, body=None): + """Return (status, text). Refuses any path outside the allow-list.""" + if not any(re.fullmatch(p, path) for p in _ALLOWED): + raise Refused("endpoint not on the allow-list: %s %s" % (method, path)) + url = base.rstrip("/") + path + if query: + url = url + "?" + urllib.parse.urlencode(query) + hdrs = dict(headers or {}) + data = None + if form is not None: + data = urllib.parse.urlencode(form).encode() + hdrs["Content-Type"] = "application/x-www-form-urlencoded" + elif body is not None: + data = json.dumps(body).encode() + hdrs["Content-Type"] = "application/json" + req = urllib.request.Request(url, data=data, headers=hdrs, method=method) + try: + with urllib.request.urlopen(req, timeout=TIMEOUT) as resp: + return resp.status, resp.read().decode("utf-8", "replace") + except urllib.error.HTTPError as e: + return e.code, e.read().decode("utf-8", "replace") + + + def declared(secrets_dir): + """{clientId: secret} from the mounted Secret. + + Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal + entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A + key must be named `-client-secret`: that naming contract is what + lets this reconciler derive the client set from git without a second + declaration, and the L0 static check enforces it against + keycloak-realm-import.yaml so the two can never drift apart. + """ + out = {} + for name in sorted(os.listdir(secrets_dir)): + if name.startswith(".") or not name.endswith(KEY_SUFFIX): + continue + path = os.path.join(secrets_dir, name) + if not os.path.isfile(path): + continue + with open(path, "r", encoding="utf-8") as fh: + value = fh.read() + # A trailing newline in the SOPS source would otherwise become part of the + # secret and mismatch forever. Keycloak secrets never contain whitespace. + out[name[: -len(KEY_SUFFIX)]] = value.strip() + return out + + + def admin_token(base, user, password, log): + """Bearer token for admin-cli in the master realm, with a bounded readiness wait.""" + last = "" + for attempt in range(1, READY_ATTEMPTS + 1): + try: + st, text = _http( + "POST", base, "/realms/master/protocol/openid-connect/token", + form={"client_id": "admin-cli", "grant_type": "password", + "username": user, "password": password}) + except OSError as e: + st, text = 0, "connection error (%s)" % e.__class__.__name__ + if st == 200: + token = json.loads(text).get("access_token") + if token: + return token + last = "http 200 but no access_token in the response" + else: + # Deliberately do NOT echo the response body: it is harmless today, but + # the request that produced it carried the admin password, and keeping + # that class of mistake impossible by habit is cheaper than reviewing it. + last = "http %s" % st + log("waiting for the Keycloak admin API (attempt %d/%d): %s" + % (attempt, READY_ATTEMPTS, last)) + if attempt < READY_ATTEMPTS: + time.sleep(READY_SLEEP) + raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s" + % (READY_ATTEMPTS, last)) + + + def reconcile(base, realm, token, want, audit, log): + """Return (changed, problems). Writes only when audit is False.""" + hdrs = {"Authorization": "Bearer " + token} + changed = 0 + problems = [] + for cid in sorted(want): + st, text = _http("GET", base, "/admin/realms/%s/clients" % realm, + query={"clientId": cid}, headers=hdrs) + if st != 200: + problems.append("%s: client lookup failed (http %s)" % (cid, st)) + continue + found = json.loads(text) + if not isinstance(found, list) or not found: + problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it " + "but the realm has no such client (the operator runs the " + "realm import exactly once)" % (cid, realm)) + continue + client = found[0] + uuid = client.get("id") + if client.get("publicClient"): + log("%-14s skipped (public client, holds no secret)" % cid) + continue + st, text = _http("GET", base, + "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid), + headers=hdrs) + if st != 200: + problems.append("%s: could not read the stored secret (http %s)" % (cid, st)) + continue + stored = json.loads(text).get("value") or "" + if hmac.compare_digest(stored, want[cid]): + log("%-14s ok (already correct, no write)" % cid) + continue + # Describe the drift WITHOUT quoting either value. The literal-placeholder + # case is named because it is the failure this program exists for, and + # saying so reveals nothing secret. + kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER" + if "env:" in stored else "stored value differs from the declared one") + if audit: + problems.append("%s: DRIFT -- %s" % (cid, kind)) + continue + log("%-14s DRIFT (%s) -- repairing" % (cid, kind)) + # MINIMAL representation only -- see the header. Keycloak merges non-null + # fields, so protocol mappers, redirect URIs, flows, groups, roles and users + # are all untouched by this write. + st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid), + headers=hdrs, body={"clientId": client.get("clientId"), + "secret": want[cid]}) + if not 200 <= st < 300: + problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160])) + continue + st, text = _http("GET", base, + "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid), + headers=hdrs) + readback = (json.loads(text).get("value") or "") if st == 200 else "" + if not hmac.compare_digest(readback, want[cid]): + problems.append("%s: PUT was accepted but the readback still does not match" + % cid) + continue + changed += 1 + log("%-14s REPAIRED and verified by readback" % cid) + return changed, problems + + + def run(argv, log): + audit = "--audit" in argv + base = os.environ.get("KEYCLOAK_BASE_URL", "").strip() + realm = os.environ.get("KEYCLOAK_REALM", "").strip() + user = os.environ.get("KEYCLOAK_ADMIN_USER", "") + password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "") + for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm), + ("KEYCLOAK_ADMIN_USER", user), + ("KEYCLOAK_ADMIN_PASSWORD", password)): + if not value: + log("FATAL: %s is empty" % name) + return 1 + log("mode=%s realm=%s base=%s" + % ("audit (read-only)" if audit else "reconcile", realm, base)) + try: + want = declared(SECRETS_DIR) + except OSError as e: + log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__)) + return 1 + if not want: + log("FATAL: no `%s` keys found in %s -- the keycloak-realm-clients " + "Secret is missing or empty, and reporting success here would repeat the " + "very fail-open the realm import already has" + % (KEY_SUFFIX, SECRETS_DIR)) + return 1 + log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want)))) + try: + token = admin_token(base, user, password, log) + changed, problems = reconcile(base, realm, token, want, audit, log) + except Refused as e: + log("FATAL: %s" % e) + return 1 + except (RuntimeError, OSError, ValueError) as e: + log("FATAL: %s: %s" % (e.__class__.__name__, e)) + return 1 + if problems: + log("") + for p in problems: + log("PROBLEM %s" % p) + log("") + if audit: + log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems)) + else: + log("RECONCILE FAIL: %d client secret(s) could not be reconciled" + % len(problems)) + return 1 + if audit: + log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match " + "git" % len(want)) + else: + log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired" + % (len(want), changed)) + return 0 + + + # ============================================================================= + # --selftest -- the offline behavioural proof. It drives the REAL code paths above + # against an in-process stub Keycloak, so what the gate proves is what ships. + # ============================================================================= + def _refused(path): + """True iff _http() refuses `path` outright (allow-list, safety property 4).""" + try: + _http("PUT", "http://127.0.0.1:1", path, body={}) + except Refused: + return True + except OSError: + return False + return False + + + def _selftest(): + import contextlib + import http.server + import io + import tempfile + import threading + + global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP + + SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + SECRET_C = "cccccccccccccccccccccccccccccccc" + STALE = "0ld-r0tated-value" + ADMIN_PW = "stub-admin-password" + # The exact value the broken clusters stored. chr(36) is the dollar sign, + # spelled this way so the file stays free of the character envsubst reacts to. + LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)" + + state = {"clients": {}, "writes": [], "touched": []} + + class Stub(http.server.BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def _send(self, code, obj): + payload = json.dumps(obj).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + def _route(self, method): + parsed = urllib.parse.urlparse(self.path) + path, qs = parsed.path, urllib.parse.parse_qs(parsed.query) + state["touched"].append("%s %s" % (method, path)) + if path == "/realms/master/protocol/openid-connect/token": + return self._send(200, {"access_token": "stub-token"}) + if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path): + cid = (qs.get("clientId") or [""])[0] + c = state["clients"].get(cid) + return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)]) + m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret", + path) + if m and method == "GET": + c = state["clients"].get(m.group(1)) + if c is None: + return self._send(404, {"error": "not found"}) + return self._send(200, {"type": "secret", "value": c.get("secret", "")}) + m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path) + if m and method == "PUT": + raw = self.rfile.read(int(self.headers.get("Content-Length") or 0)) + rep = json.loads(raw or b"{}") + # The live server rejects a full round-tripped representation with + # 400 "Cannot parse the JSON"; modelled so a regression to the fat + # PUT fails here rather than against a real realm. + if set(rep) != {"clientId", "secret"}: + return self._send(400, { + "error": "unknown_error", + "error_description": "Cannot parse the JSON"}) + state["writes"].append(m.group(1)) + state["clients"][m.group(1)]["secret"] = rep["secret"] + return self._send(204, {}) + return self._send(500, {"error": "an endpoint the reconciler must not touch"}) + + def do_GET(self): + self._route("GET") + + def do_PUT(self): + self._route("PUT") + + def do_POST(self): + self._route("POST") + + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub) + threading.Thread(target=srv.serve_forever, daemon=True).start() + base = "http://127.0.0.1:%d" % srv.server_address[1] + + tmp = tempfile.mkdtemp() + for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B), + ("kubernetes", SECRET_C), ("gitea", SECRET_A)): + with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh: + fh.write(secret + "\n") # trailing newline: must be tolerated + # Kubernetes' own projection entries must be ignored, not read as clients. + os.mkdir(os.path.join(tmp, "..data")) + + SECRETS_DIR = tmp + READY_ATTEMPTS, READY_SLEEP = 2, 0.01 + os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab", + "KEYCLOAK_ADMIN_USER": "admin", + "KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW}) + + failures = [] + checks = [0] + + def expect(cond, what): + checks[0] += 1 + if not cond: + failures.append(what) + + def invoke(*argv): + state["writes"] = [] + state["touched"] = [] + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + rc = run(list(argv), print) + return rc, buf.getvalue() + + def reset(): + state["clients"] = { + # already correct + "grafana": {"clientId": "grafana", "publicClient": False, + "secret": SECRET_A}, + # THE INCIDENT: the literal, unresolved import placeholder + "pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL}, + # public client: no secret, must be skipped + "kubernetes": {"clientId": "kubernetes", "publicClient": True}, + # ordinary drift (rotated in git, stale in the realm) + "gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE}, + } + + def leaked(text): + return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW) + if s in text] + + # -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING + reset() + rc, out = invoke("--audit") + expect(rc == 1, "audit must exit 1 when a stored secret differs from git") + expect(state["writes"] == [], + "audit mode must never write (writes=%r)" % state["writes"]) + expect("UNRESOLVED IMPORT PLACEHOLDER" in out, + "audit must name the unresolved-placeholder case explicitly") + expect("pihole: DRIFT" in out and "gitea: DRIFT" in out, + "audit must report BOTH drifted clients, not stop at the first") + expect(re.search(r"grafana\s+ok \(already correct", out), + "an already-correct client must be reported as such") + expect(re.search(r"kubernetes\s+skipped \(public client", out), + "a public client must be skipped, not treated as drift") + expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out)) + + # -- 2. reconcile: repairs exactly the drifted clients, verified by readback + reset() + rc, out = invoke() + expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:]) + expect(sorted(state["writes"]) == ["gitea", "pihole"], + "reconcile must write exactly the drifted clients, got %r" % state["writes"]) + expect("grafana" not in state["writes"], + "compare-then-write: an already-correct client must not be written") + expect("kubernetes" not in state["writes"], "a public client must not be written") + expect("REPAIRED and verified by readback" in out, + "every repair must be readback-verified") + expect("2 repaired" in out, "the summary must count the repairs") + expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out)) + + # -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing + rc, out = invoke() + expect(rc == 0, "the second reconcile must exit 0") + expect(state["writes"] == [], + "IDEMPOTENCE: a converged realm must take ZERO writes, got %r" + % state["writes"]) + expect("0 repaired" in out, "the second run must report 0 repaired") + rc, out = invoke("--audit") + expect(rc == 0, "audit must pass once the realm has converged") + expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out, + "a clean audit must emit the sentinel the L2 conformance test greps for") + + # -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched + reset() + invoke() + for touched in state["touched"]: + expect(re.fullmatch( + r"POST /realms/master/protocol/openid-connect/token" + r"|GET /admin/realms/homelab/clients" + r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?", + touched), "the reconciler touched an off-list endpoint: %s" % touched) + expect(not [t for t in state["touched"] + if "/users" in t or "/groups" in t or "/roles" in t + or "/authentication" in t], + "the reconciler must never touch users/groups/roles/authentication") + expect(_refused("/admin/realms/homelab/users"), + "the allow-list must refuse a users endpoint before opening a socket") + expect(_refused("/admin/realms/homelab/groups"), + "the allow-list must refuse a groups endpoint before opening a socket") + expect(_refused("/admin/realms/homelab"), + "the allow-list must refuse a whole-realm update endpoint") + + # -- 5. a declared client the realm never got is LOUD, not a silent pass + reset() + del state["clients"]["gitea"] + rc, out = invoke("--audit") + expect(rc == 1, "a declared client missing from the realm must fail the audit") + expect("NOT FOUND" in out, "the missing client must be named in the output") + + # -- 6. an empty / unmounted Secret must fail closed, never report success + SECRETS_DIR = tempfile.mkdtemp() + rc, out = invoke() + expect(rc == 1, "an empty secrets dir must fail closed") + expect("keys found in" in out, "the empty-secret failure must say what is missing") + SECRETS_DIR = tmp + + srv.shutdown() + print("") + if failures: + print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0])) + for f in failures: + print(" FAIL %s" % f) + return 1 + print("SELFTEST OK: %d assertions (compare-then-write, idempotence, " + "no-secret-logging, endpoint allow-list, fail-closed)" % checks[0]) + return 0 + + + def main(): + argv = sys.argv[1:] + if "--selftest" in argv: + return _selftest() + return run(argv, lambda m: print(m, flush=True)) + + + if __name__ == "__main__": + sys.exit(main()) +--- +# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key +# added after the realm import already completed) converges within one interval +# instead of waiting for a human to notice a failed login. Runs are cheap: a +# converged realm performs 12 GETs and ZERO writes. +apiVersion: batch/v1 +kind: CronJob +metadata: + name: keycloak-client-secret-reconciler + namespace: keycloak + labels: + app.kubernetes.io/name: keycloak-client-secret-reconciler + app.kubernetes.io/part-of: keycloak +spec: + schedule: "*/15 * * * *" + # Forbid: a run that is still waiting for Keycloak must not be lapped by the next + # tick. Two concurrent reconcilers would be harmless (both converge to the same + # value) but would double the write attempts during an outage recovery. + concurrencyPolicy: Forbid + startingDeadlineSeconds: 300 + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 2 + activeDeadlineSeconds: 600 + ttlSecondsAfterFinished: 3600 + template: + metadata: + labels: + app.kubernetes.io/name: keycloak-client-secret-reconciler + app.kubernetes.io/part-of: keycloak + spec: + restartPolicy: Never + # The reconciler talks ONLY to the Keycloak admin API. It needs no + # Kubernetes API access at all — its two inputs are mounted Secrets — so + # it gets no ServiceAccount token (R-SEC-02, least privilege). + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + seccompProfile: + type: RuntimeDefault + containers: + - name: reconciler + # Stock upstream python; the reconciler is pure standard library, so + # nothing is installed at runtime. Pinned exactly, like busybox / + # curlimages elsewhere in the repo. Renovate does not manage plain + # `image:` tags (renovate.json disables image-tag updates so it never + # fights Flux image-automation), so this is bumped by hand. + image: python:3.13.15-alpine3.23 + imagePullPolicy: IfNotPresent + command: + - python3 + - /opt/reconciler/reconcile-client-secrets.py + env: + - name: KEYCLOAK_BASE_URL + value: http://keycloak-service.keycloak.svc.cluster.local:8080 + - name: KEYCLOAK_REALM + value: homelab + # readOnlyRootFilesystem is on; keep CPython from trying to write + # bytecode caches beside the read-only ConfigMap mount. + - name: PYTHONDONTWRITEBYTECODE + value: "1" + - name: PYTHONUNBUFFERED + value: "1" + # Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to + # bootstrap the admin; keys admin-user / admin-password. + - name: KEYCLOAK_ADMIN_USER + valueFrom: + secretKeyRef: + name: keycloak-admin-credentials + key: admin-user + - name: KEYCLOAK_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: keycloak-admin-credentials + key: admin-password + volumeMounts: + - name: script + mountPath: /opt/reconciler + readOnly: true + # The DESIRED state: one file per declared client, named + # `-client-secret`. Mounting the Secret (rather than + # reading it through the API) is what lets this pod run with no + # ServiceAccount token at all. + - name: client-secrets + mountPath: /etc/keycloak-realm-clients + readOnly: true + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 48Mi + limits: + memory: 128Mi + volumes: + - name: script + configMap: + name: keycloak-client-secret-reconciler + defaultMode: 0444 + - name: client-secrets + secret: + secretName: keycloak-realm-clients + defaultMode: 0400 diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index 63939f9..24624b3 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -247,8 +247,37 @@ spec: # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent() # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24. + # + # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO + # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired + # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel + # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed + # to have a tunnel, via two independent guards OR'd together: + # + # 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the + # CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the + # steering datapath has a target. Requires the kube-state-metrics node-label allowlist + # enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is + # what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no + # manual re-enable to forget. The Node object survives a NotReady node, so this stays true + # while the gateway node is DOWN: gateway-node-down is still alerted. + # 2. `tun0` was observed at all in the last hour — holds the alert armed through a + # kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently + # of the KSM rollout. + # + # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the + # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live + # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on + # dev (tunnel up), and clause 2 already true on dev. - alert: VPNTunnelDown - expr: absent(node_network_receive_bytes_total{device="tun0"}) + expr: | + absent(node_network_receive_bytes_total{device="tun0"}) + and on() + ( + (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) + or + (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) + ) for: 5m labels: severity: critical @@ -359,6 +388,56 @@ spec: summary: "Keycloak database unavailable" description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)." runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db" + # --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) --- + # WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import + # had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every + # signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True, + # KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts + # Done=True passed). The only observable was a human failing to log in. The CronJob is + # the healer; these two rules make its SILENCE observable, so the class cannot sit + # undetected again. They are the runtime counterpart of the L2 audit test. + # + # UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so + # these series had no samples to check against. `kube_job_status_failed` / + # `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm + # import Job exports them); the reconciler's own series appear with its first run. + # Confirm with: kube_job_status_succeeded{namespace="keycloak", + # job_name=~"keycloak-client-secret-reconciler-.*"} + - alert: KeycloakClientSecretReconcilerFailing + # A reconciler run failed. Either it found drift it could not repair, or it could not + # reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is + # deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the + # cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h). + expr: | + max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0 + for: 15m + labels: + severity: warning + annotations: + summary: "Keycloak client-secret reconciler failing" + description: >- + A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client + secret may differ from git (unresolved import placeholder, or an unrepairable + drift) and SSO logins for that client will fail. Read the Job log: it names the + client and never prints a secret. + runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets" + - alert: KeycloakClientSecretReconcilerStale + # No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the + # CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e. + # the healer being quietly absent, which is exactly the "looks applied and isn't" + # failure this whole change exists to end. + expr: | + absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1) + for: 2h + labels: + severity: warning + annotations: + summary: "Keycloak client-secret reconciler has not succeeded" + description: >- + No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m). + The CronJob may be suspended, unapplied, or unable to schedule — client-secret + drift would go uncorrected and unnoticed (docs/15). + runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets" # ===================================================================================== # The monitor's own capacity — don't let the monitor blind itself (docs/10) diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 67aff49..bd72b3b 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -221,6 +221,17 @@ spec: limits: memory: 256Mi kube-state-metrics: + # Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster + # have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this + # `kube_node_labels` does not exist and any join against it silently matches nothing. + # Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a + # cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled + # — which is how prod (no gateway today) stops emailing a permanently-firing critical alert + # without anyone having to remember to re-enable it later. Scoped to one label on one resource: + # 3 extra series, no cardinality concern. Label name is sanitized by KSM to + # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`). + metricLabelsAllowlist: + - "nodes=[homelab.keithjackson.dev/egress-gateway]" resources: requests: cpu: 20m diff --git a/platform/monitoring/pvc-capacity-alerts.yaml b/platform/monitoring/pvc-capacity-alerts.yaml new file mode 100644 index 0000000..3127e89 --- /dev/null +++ b/platform/monitoring/pvc-capacity-alerts.yaml @@ -0,0 +1,148 @@ +--- +# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04). +# +# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly +# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified +# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO +# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill +# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/ +# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on +# the SSO database with nothing watching it. +# +# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s +# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the +# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06: +# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is +# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and +# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and +# working on both clusters. +# +# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent +# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a +# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the +# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and +# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced +# from platform-config/kustomization.yaml. +# +# ------------------------------------------------------------------------------------------------ +# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck. +# +# Two facts, both established live rather than assumed: +# +# 1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry +# DIFFERENT labels for the same PVC: +# job="kubelet" -> namespace=, node= +# job="apiserver" -> namespace="default", exported_namespace= +# Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore +# only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every +# rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and +# `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job +# pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they +# say namespace="default") and this file WOULD double-fire on them — and would report the wrong +# namespace while doing it. +# +# 2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is +# covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep +# only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml, +# which this file does not touch — recorded here as an explicit hand-off, not a silent gap. +# +# ------------------------------------------------------------------------------------------------ +# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing. +# +# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet +# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity` +# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on +# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have +# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical +# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in +# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has +# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the +# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without +# it, unifi-rootdisk is the only match at every threshold. +# +# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on +# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of +# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1 +# at 88% free. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: platform-health-pvc-capacity + namespace: monitoring + labels: + release: kube-prometheus-stack + platform-health: "true" +spec: + groups: + - name: platform-health.pvc-capacity + rules: + # Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp + # (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace + # scope differs. + - alert: PersistentVolumeFillingUp + expr: | + ( + kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"} + / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"} + ) < 0.15 + and kubelet_volume_stats_inodes{job="kubelet"} > 0 + for: 15m + labels: + severity: warning + annotations: + summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}" + description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" + # Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a + # volume is genuinely about to run out, which for a Postgres data volume means an imminent + # disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a + # permanently-firing critical. There is no counterpart for this tier anywhere else in the + # repo, so it cannot duplicate one. + - alert: PersistentVolumeAlmostFull + expr: | + ( + kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"} + / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"} + ) < 0.05 + and kubelet_volume_stats_inodes{job="kubelet"} > 0 + for: 15m + labels: + severity: critical + annotations: + summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}" + description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" + # TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless + # for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking + # ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a + # surprise. This rule instead asks "is this volume on course to hit zero?". + # + # 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log + # rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d + # horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an + # hour, so a single burst of writes cannot notify on its own. + # + # The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is + # not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is + # dominated by noise. It costs some lead time on the WAL case — the CNPG rules in + # platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for + # every OTHER PVC in the cluster, which is what was missing entirely. + # + # NOTE the monitoring namespace is excluded here too (partition rule above). That also + # side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its + # retention high-water mark looks exactly like an unbounded leak until it plateaus. + - alert: PersistentVolumeFillingUpPredicted + expr: | + ( + kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"} + / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"} + ) < 0.50 + and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0 + and kubelet_volume_stats_inodes{job="kubelet"} > 0 + for: 1h + labels: + severity: warning + annotations: + summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}" + description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index 5857fb7..cbd3e58 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -28,6 +28,45 @@ Three layers, two owners: `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so `internet`-class and node traffic are unaffected. +## Resilience — there is NO gateway failover (spec correction, escalated) + +**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) +provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature +does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**. + +Verified against the CRD actually deployed on this cluster +(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`, +Cilium **v1.20.1**) — the field's own description: + +> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic. +> **In case multiple nodes are a match for the given set of labels, the first node in lexical +> ordering based on their name will be selected.**" + +Consequences, stated plainly: + +- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s + `[vpn_egress_gateway]` group is singleton by design. +- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic + and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down + (fail-closed, per R-NET-06 — no leak, but no service). +- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while + `k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1` + sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert. +- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints + across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node + blackholes that entry's share and re-shards the remainder (breaking live connections), and each + shard leaves from a *different* public IP. Upstream confirmation: + [cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't + support failover between nodes") and + [cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not + tolerate single node failure"). + +**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents +(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited +here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in +`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file +and `egress-steering.yaml` are the accurate statement of what the implementation can actually do. + ## Secrets — host-side now PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not** a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one. diff --git a/platform/vpn-egress/egress-steering.yaml b/platform/vpn-egress/egress-steering.yaml index cf356e7..1cabfa8 100644 --- a/platform/vpn-egress/egress-steering.yaml +++ b/platform/vpn-egress/egress-steering.yaml @@ -69,6 +69,26 @@ spec: # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED, # internet fails OPEN). + # + # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby. + # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD + # deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 — + # `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`): + # + # "EgressGateway is the gateway node responsible for SNATing traffic. + # In case multiple nodes are a match for the given set of labels, the first node + # in lexical ordering based on their name will be selected." + # + # So labelling a second node does not create an active/standby pair — it just hands the role to + # whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a + # labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert. + # LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix: + # it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes + # that shard and re-shards the rest (breaking live connections), and each shard exits from a + # DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230. + # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)" + # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see + # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06. egressGateway: nodeSelector: matchLabels: