From cb1916f092bd5a02082668bb2e189b2eb26f47f6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 6 Sep 2026 19:35:18 -0500 Subject: [PATCH] =?UTF-8?q?feat(conformance):=20cover=20R-PLAT-01,=20R-NET?= =?UTF-8?q?-04,=20R-SEC-03=20=E2=80=94=2088.2%=20->=2094.1%,=20and=20fix?= =?UTF-8?q?=20an=20unrecordable-proof=20defect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 --- .../chainsaw-test.yaml | 106 ++++ .../auditability-proofs-producible/check.sh | 431 +++++++++++++ .../chainsaw-test.yaml | 94 +++ .../egress-paths-selected-by-rules/check.sh | 513 ++++++++++++++++ .../chainsaw-test.yaml | 83 +++ .../ha-control-plane-as-code/check.sh | 571 ++++++++++++++++++ .../default-deny/chainsaw-test.yaml | 17 + .../egress-default-path/chainsaw-test.yaml | 199 ++++++ .../fixtures/three-egress-pods.yaml | 67 ++ .../chainsaw-test.yaml | 17 + .../ha-control-plane/chainsaw-test.yaml | 373 ++++++++++++ .../trusted-tls/chainsaw-test.yaml | 17 + 12 files changed, 2488 insertions(+) create mode 100644 conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/auditability-proofs-producible/check.sh create mode 100644 conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh create mode 100644 conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/ha-control-plane-as-code/check.sh create mode 100644 conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml create mode 100644 conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml create mode 100644 conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml diff --git a/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml new file mode 100644 index 0000000..85117d3 --- /dev/null +++ b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml @@ -0,0 +1,106 @@ +# ============================================================================= +# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an +# operator can prove isolation, egress path, cert validity, and backup integrity)"), +# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator +# must be able to prove": Isolation (denied-probe test), Egress path (egress IP +# check), Kill-switch (fault-injection test), TLS (cert status + external check), +# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not +# one-time checks", with the acceptance gate "The five auditability proofs above can +# be produced on demand." +# +# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement) +# ----------------------------------------------------------------------------------- +# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that +# compiles and would keep passing while every proof rotted into a scaffold. Four distinct +# failure modes are asserted instead, each by DRIVING the real machinery: +# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer +# the spec's own words require (fault injection / a DR drill => L3), taken from +# lib/status.py's own verdict so this check cannot disagree with the status report; +# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements ` +# must resolve to real test dirs AND pass the merge gate's coverage assertion; +# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can +# attribute — a proof that passes but records nothing leaves the operator with a green +# terminal and no evidence; +# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the +# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP +# token, and a ledger row claiming `proven` at the repository's ROOT commit — and +# requires record_run.py to report not-proven and status.py to report `stale-proof`. It +# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in +# ci/ or run.sh ran before this test existed. +# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo +# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly, +# so a sixth proof added to the spec fails here instead of being silently unverified. +# +# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster, +# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE +# anyone needs it. +# +# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted), +# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate, +# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN, +# and two synthetic forgeries are presented to them, +# THEN each of docs/07's five proofs maps to an executable test at the required layer, the +# documented one-command invocation selects and asserts them, each is capturable as a +# ledger row, and neither a self-skip nor a stale row can be recorded as proof. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's +# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run +# it directly, and this Chainsaw Test wraps the SAME script. +# +# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE +# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny, +# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass +# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs +# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The +# assertion here was NOT relaxed to accommodate them (R-TEST-06). +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * That the five proofs PASS. This asserts they are present, strong, runnable by one +# command, capturable, and that the chain refuses forgeries — not that isolation holds +# today (R-NET-01's test) or that the restore works (R-STATE-03's). +# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this +# suite queries Hubble. Only the denied-probe half is automated — a real, named gap. +# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that +# would need signed history. +# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every +# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately +# not re-asserted here. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: auditability-proofs-producible + labels: + conformance.home.k3s/R-SEC-03: "true" + annotations: + conformance.home.k3s/requirements: "R-SEC-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree and the harness's own tooling, when status.py, the requirement + resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence + tools' self-tests are driven and two synthetic forgeries are presented to them, then each + of docs/07's five auditability proofs has an executable test at the layer the spec + requires, is selectable and asserted by the documented one-command invocation, is + capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as + evidence. +spec: + steps: + - name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh +# --------------------------------------------------------------------------------------------- +# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder +# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is +# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof +# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would +# report itself as unexecutable. Observed and worked around here; the detector's crudeness is +# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is +# the wrong trade (R-TEST-06). diff --git a/conformance/tests/l0-static/auditability-proofs-producible/check.sh b/conformance/tests/l0-static/auditability-proofs-producible/check.sh new file mode 100755 index 0000000..bb36f30 --- /dev/null +++ b/conformance/tests/l0-static/auditability-proofs-producible/check.sh @@ -0,0 +1,431 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — auditability-proofs-producible +# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can +# prove isolation, egress path, cert validity, and backup integrity)." +# — spec docs/00-vision-and-scope.md). +# +# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md +# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove": +# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a +# denied-probe test. +# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check. +# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test. +# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert +# status + external check. +# - **Backups**: "irreplaceable data is restorable" — via a DR drill. +# "These proofs are part of the platform acceptance suite, not one-time checks." +# and its acceptance gate: "The five auditability proofs above can be produced on demand." +# +# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY +# ------------------------------------------------------------ +# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01, +# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and +# would keep passing while every proof rotted into a scaffold. Four separate failure modes +# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted +# by DRIVING the real machinery rather than by reading files: +# +# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a +# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and, +# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an +# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this +# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the +# repository's own status report. +# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh +# --requirements ` must resolve to a non-empty test set and must ASSERT +# their coverage (that flag is the merge gate's own path: it passes --require to +# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the +# gate wiring break, R-SEC-03 fails. +# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator +# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must +# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with +# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail +# instead of silently making the requirement unprovable. +# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause +# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the +# real tools, in a throwaway directory: +# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log +# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3 +# test. record_run.py must refuse to call it `proven`. +# * a ledger row claiming `proven` at a commit that PREDATES the requirement's +# implementing code — status.py must degrade it to `stale-proof`, never green. +# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing +# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof +# were themselves unexercised in the merge gate until this check. +# +# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec +# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's +# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather +# than quietly ignoring it. +# +# WHY L0: every clause above is a property of the repository and its own tooling, provable +# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact +# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true +# with one. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one +# documented command, capturable as evidence, and that the evidence chain refuses +# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR +# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the +# proof, which is what is asserted. +# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this +# suite queries Hubble, and this check does not pretend otherwise: it asserts the +# denied-probe half, which is the half that is automated. The Hubble half remains an +# operator-interactive capability — a real, named gap in R-SEC-03's coverage. +# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status +# report's prose, the ledger's provenance fields) is a usability property, not a testable +# one. +# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review; +# nothing here can prove a row was never removed from it (that would need signed history). +# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than +# the five bullets docs/07 enumerates. The five are what the spec made concrete and are +# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim, +# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +: "${SPEC_REPO:=${ROOT}/../home-network-k3s}" +export SPEC_REPO + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(sys.argv[1]) +CONF = ROOT / "conformance" +LIB = CONF / "lib" +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +def die(): + print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} " + "assertions failed", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +def run(cmd, **kw) -> subprocess.CompletedProcess: + return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw) + + +# ================================================= THE FIVE PROOFS docs/07 ENUMERATES +# key = the bullet label docs/07 uses (verified against the spec below when it is mounted) +# reqs = the requirement(s) whose verifying test IS that proof +# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection +# test" / "a DR drill", an L2 happy-path test does not discharge the proof. +PROOFS = { + "Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"}, + "why": '"service X cannot reach service Y" — via Hubble flow logs + a ' + "denied-probe test"}, + "Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"}, + "why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'}, + "Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"}, + "why": '"with VPN down, X leaks nothing" — via a fault-injection test'}, + "TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"}, + "why": '"every endpoint serves a valid, non-self-signed, unexpired cert" ' + "— via cert status + external check"}, + "Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"}, + "why": '"irreplaceable data is restorable" — via a DR drill'}, +} +PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]}) + +# ---- SELF-POLICING: the proof list must still be the SPEC's proof list --------------- +spec = Path(os.environ.get("SPEC_REPO", "")) +sec_doc = spec / "docs" / "07-security-and-secrets.md" +if sec_doc.is_file(): + text = sec_doc.read_text() + m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text, + re.M | re.S) + check(m is not None, + "docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — " + "the proof list this check is coupled to has moved or been removed, so the mapping " + "below can no longer be trusted (R-TEST-06: follow the spec, never the other way)") + if m: + labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M) + check(set(labels) == set(PROOFS), + f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check " + f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying " + "test, and a proof this check does not know about would be silently unverified") + check(len(labels) == 5, + f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance " + "gate says 'The five auditability proofs above can be produced on demand' — the " + "prose and the list have drifted apart") +else: + note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be " + "re-derived from docs/07, so the mapping below was used as written. Every other " + "assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)") + +# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH +# Driven through lib/status.py so this check and the repository's own status report cannot +# disagree about what is executable and what is a scaffold. +tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-")) +try: + out = tmp / "status" + st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT), + "--registry", str(CONF / "requirements" / "registry.yaml"), + "--tests", str(CONF / "tests"), "--out", str(out)]) + check(st.returncode == 0, + f"lib/status.py exited {st.returncode} — the report an operator would use to find " + f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}") + status = {} + sj = out / "status.json" + if sj.is_file(): + status = json.loads(sj.read_text()) + check(bool(status.get("by_requirement")), + "lib/status.py produced no per-requirement report — the audit posture of each " + "requirement is not derivable, so R-SEC-03 cannot be assessed at all") + by_req = status.get("by_requirement", {}) + + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + rec = by_req.get(rid) + check(rec is not None, + f"proof '{label}' maps to {rid}, which the status report does not know — " + "either the registry lost the requirement or this mapping is stale") + if not rec: + continue + tests = (rec.get("verification") or {}).get("tests") or [] + executable = [t for t in tests if t.get("state", "").startswith("executable")] + check(bool(executable), + f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are " + f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: " + f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the " + "operator cannot prove this property") + at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]] + check(bool(at_layer), + f"proof '{label}' ({rid}) has no executable test at layer(s) " + f"{sorted(spec_proof['layers'])} — it has " + f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. " + f"docs/07 requires this proof to be produced {spec_proof['why']}, and a " + "weaker layer does not discharge it (a rendering check is not a drill)") + + # ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND + # The resolver: --requirements must translate the five R-IDs into real test dirs. + resolver = run(["bash", "-c", + f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; ' + f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"']) + dirs = [l for l in resolver.stdout.splitlines() if l.strip()] + check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS), + f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) " + f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's " + "acceptance gate depends on ('produced on demand') cannot select the proofs") + + # The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a + # zero exit here is the repository's own statement that all five proofs are covered. + gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)]) + check(gate.returncode == 0, + f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited " + f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit " + f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}") + + # ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE + # Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass + # token can pass forever and never be recorded as proof of anything. + sys.path.insert(0, str(LIB)) + os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1") + sys.dont_write_bytecode = True + import record_run as rr # noqa: E402 + + smap = rr.sentinel_map(CONF / "tests") + check(len(smap) >= 10, + f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its " + "grip on the test tree, so the assertions below would assert nothing") + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + rec = by_req.get(rid) or {} + tests = (rec.get("verification") or {}).get("tests") or [] + names = [t.get("name") for t in tests + if t.get("state", "").startswith("executable") + and t.get("layer") in spec_proof["layers"]] + for name in names: + ent = smap.get(name) + check(ent is not None, + f"proof '{label}': record_run.py does not know the test {name!r} — its " + "result could never be attributed to a ledger row") + if not ent: + continue + if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS: + continue # offline layers are truthful by JUnit alone + check(bool(ent.get("pass")), + f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, " + "so lib/record_run.py can never record it as proven. It would go green " + "in the terminal and leave the operator with no evidence — exactly the " + "gap the ledger exists to close (docs/07: the proofs are 'part of the " + "platform acceptance suite, not one-time checks')") + check(bool(ent.get("terminal_pass")), + f"proof '{label}': the test {name!r} has no TERMINAL pass token, so " + "record_run.py cannot decide its verdict") + + # =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE + # Both tools' own guards, which NOTHING in ci/ or run.sh drives today. + for tool in ("record_run.py", "status.py"): + r = run([sys.executable, str(LIB / tool), "--selftest"]) + check(r.returncode == 0, + f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a " + "self-skipped or stale run from being recorded as proof are broken, so nothing " + f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}") + + # ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token. + # This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof. + forge_target, forge_skip = None, None + for label, spec_proof in sorted(PROOFS.items()): + for rid in spec_proof["reqs"]: + for name, ent in smap.items(): + if rid in (ent.get("requirements") or []) and ent.get("skip") \ + and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS: + forge_target, forge_skip = name, ent["skip"][0] + break + if forge_target: + break + if forge_target: + break + check(forge_target is not None, + "no Chainsaw-layer proof test declares a skip token, so the forgery attempt below " + "could not be mounted. That is not a pass: it means this check could not verify " + "that a self-skip is refused as proof, which is the whole point of the ledger") + if forge_target: + run_dir = tmp / "20260101T000000Z-forged" + run_dir.mkdir(parents=True) + (run_dir / "report.junit.xml").write_text( + '\n' + '' + f'' + "\n") + (run_dir / "run.log").write_text( + f"=== RUN chainsaw/{forge_target}\n" + f" === STDOUT\n {forge_skip}\n") + forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir), + "--dry-run", "--ledger", str(tmp / "never-written.jsonl"), + "--tests", str(CONF / "tests"), "--repo", str(ROOT)]) + verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b", + forged.stdout, re.M) + check(verdicts and all(v != "proven" for v in verdicts), + f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: " + "a JUnit saying `passed` with nothing in the log but that test's own SKIP token " + f"({forge_skip}). A self-skip must never become proof — with this hole every " + "audit proof in docs/07 could be produced by a cluster that skipped it") + check(not (tmp / "never-written.jsonl").exists(), + "record_run.py --dry-run WROTE to the ledger — a tool that records while " + "claiming not to cannot be trusted to record only real runs") + + # ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code. + # status.py must degrade it to stale-proof (not green), or an old green run would keep + # vouching for code it never executed. + root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"]) + old_sha = (root_sha.stdout.strip().splitlines() or [""])[0] + check(len(old_sha) == 40, + "the repository's root commit could not be resolved, so the stale-proof guard could " + "not be exercised (that is a gap in this check, not a pass)") + if len(old_sha) == 40 and forge_target: + stale_ledger = tmp / "stale.jsonl" + rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0] + stale_ledger.write_text(json.dumps({ + "ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev", + "chainsaw_version": "v0.0.0", "kube_version": "unknown", + "layer": smap[forge_target].get("layer", "L3"), "test": forge_target, + "requirements": [rid], "junit_result": "passed", + "sentinel": (smap[forge_target].get("terminal_pass") or "X"), + "proof": "proven", "run_id": "20260101T000000Z-forged", + "note": "synthetic row written by the R-SEC-03 conformance check", + }) + "\n") + out2 = tmp / "status-stale" + st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT), + "--registry", str(CONF / "requirements" / "registry.yaml"), + "--tests", str(CONF / "tests"), "--out", str(out2), + "--ledger", str(stale_ledger)]) + check(st2.returncode == 0, + f"lib/status.py failed against a synthetic ledger: " + f"{(st2.stderr or st2.stdout).strip()[:300]}") + s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {} + rec2 = (s2.get("by_requirement") or {}).get(rid) or {} + per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])} + check(per.get(forge_target) == "stale-proof", + f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT " + f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A " + "proof from a run that predates the code it vouches for is not evidence, and " + "honouring it would let one old green run certify every later change") + check((rec2.get("proof") or {}).get("state") != "proven", + f"{rid} is reported PROVEN on the strength of a row from the root commit — the " + "freshness guard is not holding") + + # ---- the ledger itself is machine-readable, and no row proves a test that is gone ---- + ledger = CONF / "runs" / "ledger.jsonl" + if ledger.is_file(): + rows, bad = [], [] + for i, line in enumerate(ledger.read_text().splitlines(), 1): + if not line.strip(): + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + bad.append(i) + check(not bad, + f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record " + "is not machine-readable, so no proof can be derived from it") + for r in rows: + for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof", + "run_id"): + check(key in r, + f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without " + "provenance cannot be audited") + if r.get("proof") == "proven": + check(bool(r.get("sentinel")), + f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — " + "the sentinel is the only thing separating a real pass from a self-skip") + check(r.get("test") in smap, + f"a ledger row claims `proven` for {r.get('test')!r}, which is not a " + "test in the suite any more — a proof of something that no longer exists") + check("-dirty" not in str(r.get("git_sha", "")), + f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree " + f"({r.get('git_sha')}) — what ran is not what is committed") + else: + note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions " + "were skipped; every other assertion ran, and the forgery probes above use " + "synthetic ledgers, not this file") +finally: + shutil.rmtree(tmp, ignore_errors=True) + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 30, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +for n in NOTES: + print(f" note: {n}") +print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 " + f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the " + f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves " + "and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can " + "attribute, both evidence tools' self-tests pass, and two independent forgeries — a " + "self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. " + "R-SEC-03's producibility half only: whether each proof PASSES is its own " + "requirement's test, and the Hubble half of the isolation proof is automated nowhere " + "(see the header's gap list).") +PY diff --git a/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml new file mode 100644 index 0000000..7870a99 --- /dev/null +++ b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml @@ -0,0 +1,94 @@ +# ============================================================================= +# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by +# routing rules ... realized as two paths — the local `internet` path and the +# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress +# routing: two paths, one physical WAN": "the `internet` path is simply the default +# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and +# the nodes themselves — egress the local/private WAN by default and **never** the +# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the +# kill-switch blackholes **only** that class." +# +# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here) +# ------------------------------------------------------------------------ +# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class +# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and +# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES +# that select among them, which nothing else asserts: +# * exactly TWO paths, the same closed set in the schema, the admission policy and the +# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope); +# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one +# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one +# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because +# absence of a matching rule IS the default path; +# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a +# vpn-class pod's cluster/LAN traffic still takes the local path; +# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing +# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored; +# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every +# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN. +# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.) +# +# WHY L0: every clause above is a statement about ALL rules — including rules that must be +# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default +# route in the main table). A cluster can show which path two pods took; it cannot show that +# no third rule exists or that the vocabulary is closed. +# +# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are +# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod +# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked +# against its own values. +# +# GIVEN the repository as git tracks it, +# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy +# routing / OpenVPN / nftables templates, and every environment's overlay addresses are +# parsed and cross-checked, +# THEN there are exactly two paths with `internet` as the default; exactly one steering rule +# selects exactly the vpn class by label; the gateway is selected by node label with a +# named egress interface; all RFC1918 space and every address any environment assigns +# itself is excluded from the tunnel; the tunnel's default route and its fail-closed +# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN +# cannot take the main table; and no nftables DROP is unscoped. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this +# Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by +# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled +# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test. +# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates +# that generate them; a hand-added rule on the box is invisible here. +# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no +# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default". +# * IPv6 egress. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-paths-selected-by-rules + labels: + conformance.home.k3s/R-NET-04: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-04" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the egress vocabulary, every steering rule and the gateway + node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers + exactly two egress paths with the WAN as the default, exactly one rule selects exactly + the vpn class, local destinations stay on the local path in every environment, the + tunnel never becomes the default route, and nothing unscoped can blackhole the default + path. +spec: + steps: + - name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh b/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh new file mode 100755 index 0000000..87986c1 --- /dev/null +++ b/conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh @@ -0,0 +1,513 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — egress-paths-selected-by-rules +# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing +# rules. Per OQ-1 the environment has **one physical commercial WAN and no +# multi-ISP plans**, so this is realized as two paths — the local `internet` path and +# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP +# uplinks) is **out of scope** unless a future ADR reopens it." +# — spec docs/00-vision-and-scope.md). +# +# The elaborating spec text this check quotes and asserts against — docs/03 +# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)": +# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge +# router -> internet | +# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN +# egress gateway (ADR-0013) | +# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing +# to policy-route across; the `internet` path is simply the default route out." +# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes +# themselves — egress the local/private WAN by default and **never** the VPN. Only +# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch +# blackholes **only** that class." +# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default +# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN — +# only what Cilium points at `tun0` uses the VPN." +# +# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST +# --------------------------------------------------------------------------------- +# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it, +# and the class can be flipped via GitOps. Those are proven by +# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class +# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod). +# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label. +# +# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its +# distinctive content — none of which any existing test asserts — is: +# +# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is +# one closed set of two values agreed by the schema, the admission policy and the +# renderer. A third value appearing in one of them but not the others would be a path +# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third +# class would contradict OQ-1's resolution. +# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn` +# path is entered only via rules that select the vpn class — a Cilium +# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped +# to the tunnel source address. Nothing selects the `internet` class, because +# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS +# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN. +# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn +# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class +# pod's cluster/LAN traffic still takes the local path at the same time as its +# internet traffic takes the tunnel. +# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route +# lives in a DEDICATED routing table reached only through the scoped rule, the +# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN +# from touching the main table. Otherwise there would be one path (the VPN), not two. +# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the +# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified +# by the vpn class's own source set / fwmark. The `internet` path therefore fails +# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails +# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check +# asserts the OTHER direction, which that test does not.) +# +# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration: +# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting +# exactly one class"), and scoping ("no unqualified drop, no default route in the main +# table") are statements about ALL rules, including the ones that would have to be absent. +# A running cluster can show you which path two pods took; it cannot show you that no third +# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural +# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in +# tests/l2-behavioral/egress-default-path (same requirement label). +# +# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must +# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay +# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different +# LAN is checked against its own addresses, and the check keeps working when they change. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled +# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled +# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default +# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3. +# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that +# generate them, driven by no node. A hand-added `ip rule` on the gateway node is +# invisible here (and to every other test); drift on the box is an Ansible check-mode +# exercise for the operator. +# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no +# honest repository signal that would distinguish "one WAN, correctly" from "one WAN, +# because nobody tried" — and inventing one would be a hollow assertion. +# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform. +# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and +# this check would NOT catch its absence. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 +LBL = "homelab.keithjackson.dev" +CLASS_LABEL = f"{LBL}/egress-class" + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def die(): + print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} " + "assertions failed", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +def docs(path: Path): + """Every YAML document in a file, skipping empties.""" + try: + return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)] + except yaml.YAMLError: + return [] + + +TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml") + if ".git" not in p.parts and "_artifacts" not in p.parts) +SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json" +RENDER = ROOT / "catalog" / "render.py" +VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway" +CLUSTERS = ROOT / "clusters" + +for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS): + check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be " + "read at all (R-NET-04)") +if FAILURES: + die() + + +# ======================================================= (1) EXACTLY TWO PATHS, AGREED +import json # noqa: E402 — only needed for the schema + +schema = json.loads(SCHEMA.read_text()) +egress_prop = ((schema.get("properties") or {}).get("egress") or {}) +enum = egress_prop.get("enum") +check(isinstance(enum, list) and len(enum) == 2, + f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is " + "realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path " + "nothing routes; one value is not 'multiple egress paths'") +PATHS = set(enum or []) +check(PATHS == {"internet", "vpn"}, + f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as " + "`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)") +check(egress_prop.get("default") == "internet", + f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: " + "'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply " + "the default route out'. A service that says nothing must get the WAN") + +# The admission policy must constrain the pod label to the SAME closed set, or a workload +# could carry a class the schema never allowed (and that no rule routes). +pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml")) +check(len(pol_files) >= 1, + "no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod " + "carrying an egress class outside the declared path set") +pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files) +for cls in sorted(PATHS): + check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None, + f"the egress admission policy never names the {cls!r} path — a class the policy " + "does not know cannot be validated against the path set") +# No tracked object may carry a class outside the closed set — with ONE self-policing +# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the +# admission test can assert the policy REJECTS it; that pod is exempt only while a +# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the +# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist. +def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool: + tests_root = fixture.parent + while tests_root != ROOT and tests_root.name != "tests": + tests_root = tests_root.parent + if tests_root == ROOT: + return False + for tf in sorted(tests_root.rglob("kyverno-test.yaml")): + for d in docs(tf): + for case in (d.get("results") or []): + if case.get("result") == "fail" and resource_name in (case.get("resources") or []): + return True + return False + + +stray: list[str] = [] +_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)") +for f in TRACKED_YAML: + for d in docs(f): + labels = ((d.get("metadata") or {}).get("labels") or {}) + cls = labels.get(CLASS_LABEL) + # Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs). + tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {} + for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)): + if value is None or value in PATHS or value == "CLASS": + continue + name = ((owner.get("metadata") or {}).get("name") or "?") + if rejected_by_a_policy_test(name, f): + continue + stray.append(f"{f.relative_to(ROOT)}:{name}={value}") + # Values written as raw text (templates, docs-as-code) are caught too. + for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))): + if value in PATHS or value == "CLASS": + continue + if any(value in s for s in stray): + continue + # Raw-text hit with no parsed owner: exempt only if EVERY object in the file that + # carries this value is a policy-test fixture proven to be rejected. + owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f) + if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value] + if owners and all(rejected_by_a_policy_test(o, f) for o in owners): + continue + stray.append(f"{f.relative_to(ROOT)}:={value}") +check(not stray, + f"tracked YAML assigns egress-class value(s) outside the declared path set " + f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission " + "policy allows is a workload on no path at all (the only exemption is a Kyverno " + "policy-test fixture that a kyverno-test.yaml still expects to FAIL)") + +# The renderer must project the DECLARED class onto the pod label, unmodified: that label is +# what the routing rule selects on, so a renderer that rewrote it would break the coupling +# between the declaration and the path. +rtext = RENDER.read_text() +check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None + or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None, + "catalog/render.py does not project the ServiceDefinition's own `egress` value onto the " + f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be " + "two independent facts that could disagree") + + +# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT +policies: list[tuple[Path, dict]] = [] +for p in TRACKED_YAML: + for d in docs(p): + if d.get("kind") == "CiliumEgressGatewayPolicy": + policies.append((p, d)) +check(len(policies) == 1, + f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored " + f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE " + "rule. Two rules can overlap, and which one wins for an overlapping pod is not " + "something the repository states") + +for path, pol in policies: + rel = str(path.relative_to(ROOT)) + spec = pol.get("spec") or {} + selectors = spec.get("selectors") or [] + check(len(selectors) == 1, + f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional " + "selector widens the set of pods steered off the default path, and R-NET-04's " + "selection must be exactly 'the vpn class'") + for sel in selectors: + pod_sel = sel.get("podSelector") + check(isinstance(pod_sel, dict) and pod_sel, + f"{rel}: a selector has no podSelector — an empty selector in Cilium matches " + "EVERY pod, which would put the whole cluster on the VPN path and leave no " + "default path at all") + ml = (pod_sel or {}).get("matchLabels") or {} + me = (pod_sel or {}).get("matchExpressions") or [] + check(not me, + f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a " + "single label equality; expressions (In/NotIn/Exists) can select pods that " + "never declared a class, so the rule set would no longer be disjoint from the " + "default path") + check(set(ml) == {CLASS_LABEL}, + f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly " + f"{CLASS_LABEL}. Any other key makes path selection depend on something other " + "than the declared class (and a namespace/app key would silently exempt pods)") + check(ml.get(CLASS_LABEL) == "vpn", + f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only " + "the `vpn` class may be steered: docs/03 — 'Only services labeled " + "egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would " + "route the default path through the tunnel, i.e. leave one path, not two") + + # The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a + # rule, so the gateway can move (or gain a standby) with no rule rewrite. + gw = spec.get("egressGateway") or {} + node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {} + check(bool(node_sel), + f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a " + "named host means the vpn path cannot fail over (docs/03 requires two gateway " + "nodes + Egress Gateway HA)") + check("nodeName" not in gw, + f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by " + "label so the path survives losing that node") + check(isinstance(gw.get("interface"), str) and gw.get("interface"), + f"{rel}: the egressGateway names no egress interface — without it the steered class " + "is not actually sent down a different path from the default") + + # ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION ------------- + dests = spec.get("destinationCIDRs") or [] + excl = spec.get("excludedCIDRs") or [] + check([str(c) for c in dests] == ["0.0.0.0/0"], + f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations " + "('Routed through the in-cluster VPN tunnel'), expressed as the whole internet " + "minus the exclusions below — narrowing it here would silently leave public " + "destinations on the WAN for a vpn-class service") + try: + excluded = [ipaddress.ip_network(str(c)) for c in excl] + except ValueError as e: + excluded = [] + check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})") + check(len(excluded) >= 3, + f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, " + "pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918") + + def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool: + return any(net.subnet_of(e) for e in excluded if e.version == net.version) + + # RFC1918 in full — computed, not string-matched, so a typo'd mask is caught. + for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"): + net = ipaddress.ip_network(rfc) + check(covered(net), + f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class " + "pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not " + "coexist for the same pod and east-west traffic would break") + + # Every address each ENVIRONMENT declares for itself must fall in the excluded set: that + # is what makes 'local stays local' true per environment rather than by coincidence. + def overlay_addresses() -> list[tuple[str, str]]: + found: list[tuple[str, str]] = [] + base = CLUSTERS / "base" / "cluster-vars.yaml" + data = {} + if base.is_file(): + for d in docs(base): + if d.get("kind") == "ConfigMap": + data = {k: str(v) for k, v in (d.get("data") or {}).items()} + for f in sorted(CLUSTERS.glob("*/kustomization.yaml")): + env = {**data} + for m in re.finditer( + r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P[A-Z0-9_]+)\s*\n" + r"\s*value:\s*(?P\S.*?)\s*$", f.read_text(), re.M): + env[m.group("k")] = m.group("v").strip().strip('"').strip("'") + label = f.parent.name + for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START", + "CILIUM_LB_POOL_STOP", "K3S_API_HOST"): + v = env.get(key, "") + if v and not v.startswith("TBD"): + found.append((f"{label}/{key}", v)) + return found + + addrs = overlay_addresses() + check(len(addrs) >= 5, + f"only {len(addrs)} environment address(es) could be read from the Flux overlays — " + "the 'local traffic stays local' assertion would be checking almost nothing") + for label, value in addrs: + try: + net = ipaddress.ip_network(value, strict=False) + except ValueError: + continue + check(covered(net), + f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is " + "NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment " + "would try to reach its own cluster/LAN through the VPN") + + +# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ======== +TPL = VPN_ROLE / "templates" +up = (TPL / "vpn-egress-up.sh.j2") +down = (TPL / "vpn-egress-down.sh.j2") +conf = (TPL / "vpn-egress.conf.j2") +nft = (TPL / "vpn-egress-killswitch.nft.j2") +for f in (up, down, conf, nft): + check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing " + "rules are not authored") +if FAILURES: + die() + +defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {} + + +def render(text: str) -> str: + """Substitute `{{ var }}` with the role's own default value, for every default. + + The templates reference their knobs by NAME, so an analysis that looked for the VALUE + (e.g. the nft set name) would silently match nothing and pass vacuously — which is + exactly what an earlier draft of this check did. Rendering first means the assertions + below examine the text the node actually gets. Only simple `{{ name }}` expressions are + substituted; anything else is left as-is (and the assertions are written so an + unsubstituted expression fails rather than passes). + """ + out = text + for k, v in defaults.items(): + if isinstance(v, (str, int)): + out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out) + return out + + +up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text()) +table_id = str(defaults.get("vpn_egress_gateway_route_table_id", "")) +check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"), + f"the tunnel's routing table id is {table_id!r} — the second path must live in a " + "DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and " + "putting the tunnel's default route in any of them makes the VPN the node's default " + "route instead of a selectable second path") +check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(), + "no ip-rule priority is declared for the tunnel table — 'selected by routing rules' " + "needs a rule, at a defined precedence") + +# Every `ip route ... default` the up-script installs must name the dedicated table. +route_lines = [l.strip() for l in up_t.splitlines() + if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)] +check(len(route_lines) >= 2, + f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own " + "default route via the tunnel AND the fail-closed blackhole behind it") +for l in route_lines: + check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None, + f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route " + "landing in the main table changes the path every other pod and the node itself " + "uses — the VPN would stop being opt-in") + check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None, + f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the " + "kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never " + "be blackholed") + +# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup `) sends +# everything into the tunnel table, which would leave one path instead of two. +rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)] +check(len(rule_lines) >= 1, + "the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second " + "routing table would never be consulted (R-NET-04's 'selected by routing rules')") +for l in rule_lines: + check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None, + f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector " + "EVERY packet is looked up in the tunnel table, so the WAN path disappears and the " + "VPN is no longer opt-in") + check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None, + f"an ip rule looks up a table other than the tunnel's: {l!r}") +check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None, + "the steering rule is not scoped to ${SRC}, the tunnel's own local address — that " + "address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that " + "distinguishes the steered class from every other flow on the gateway node") +check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None, + "the up-script does not refuse to run when the tunnel address is unknown — installing " + "a partial rule set is how a path ends up half-selected") + +# The provider must not be able to make the tunnel the default route either. +check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None, + "the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed " + "routes itself, in the MAIN table, and the VPN would become the node's default route " + "(docs/03: 'does **not** set the node's default route (`route-noexec` + policy " + "routing)')") +check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None, + "the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN " + "provider could redirect the node's whole default route into the tunnel, collapsing " + "the two paths into one") +check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None, + "the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the " + "default route") + + +# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN) +src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", "")) +fwmark = str(defaults.get("vpn_egress_gateway_fwmark", "")) +check(bool(src_set), + "the role declares no nftables source set — the kill-switch would have no way to name " + "'the steered class' and could only match broadly") +policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)] +check(len(policies_lines) >= 1, + "the kill-switch nft template declares no base chain — nothing would be enforced") +for l in policies_lines: + check(re.search(r"policy\s+accept\s*;", l) is not None, + f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here " + "blackholes the `internet` path too, and docs/03 requires the two classes to fail " + "in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'") +drop_lines = [l.strip() for l in nft_t.splitlines() + if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")] +check(len(drop_lines) >= 1, + "the kill-switch nft template contains no drop rule at all — there is no kill-switch, " + "and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope") +for l in drop_lines: + scoped = (src_set and src_set in l) or (fwmark and "mark" in l) + check(scoped, + f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel " + f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole " + "the default `internet` path, which must never be blackholed") + + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 35, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; " + f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), " + f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route " + f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP " + f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR " + "is L2 (egress-class for the two labelled classes, egress-default-path for the " + "unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).") +PY diff --git a/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml new file mode 100644 index 0000000..97f8005 --- /dev/null +++ b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml @@ -0,0 +1,83 @@ +# ============================================================================= +# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with +# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST +# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA, +# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)", +# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs"). +# +# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the +# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments +# from one code path, with exactly one --cluster-init founder, no external datastore, and a +# floating API endpoint that is SAN'd into the serving cert and cannot collide with the +# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live +# cluster cannot tell you what the NEXT environment brought up from this repo would look +# like. The observed half is asserted separately at L2 by +# tests/l2-behavioral/ha-control-plane, which carries the same requirement label. +# +# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down. +# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group / +# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay +# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name. +# +# GIVEN the repository as git tracks it, +# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each +# environment's Flux overlay are parsed and cross-checked against each other, +# THEN k3s is the pinned distro and nothing is installed as an agent; every environment +# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY +# ONE --cluster-init founder and every other server joining it with --server; +# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd); +# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip +# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is +# in the apiserver serving cert's SANs together with every server's own address, and +# it is neither a node address nor the ingress Gateway VIP nor inside that +# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME +# roles and plays retargeted by variables only. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no +# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this +# Chainsaw Test wraps the SAME script. +# +# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them): +# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating, +# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane. +# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves) +# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss +# drains a node for R-STATE-02's data claim and asserts nothing about the control plane. +# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of +# infra/PROD-PROVISIONING.md). +# * kube-vip is NOT imported by playbooks/site.yml, so the floating endpoint is a separate +# operator step. Observed and reported, NOT asserted — R-PLAT-01 does not fix the +# bring-up order (R-TEST-06). +# * `K3S_API_HOST` (Cilium's k8sServiceHost) is the PRIMARY NODE's address in both +# environments rather than the floating VIP, contradicting the prod overlay's own comment. +# A real single-node dependency, ESCALATED to the PM rather than asserted. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: ha-control-plane-as-code + labels: + conformance.home.k3s/R-PLAT-01: "true" + annotations: + conformance.home.k3s/requirements: "R-PLAT-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each + environment's Flux overlay are parsed and cross-checked, then every environment is + authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one + embedded-etcd founder, no external datastore, and a leader-elected control-plane-only + VIP that is in the serving cert's SANs and collides with no other allocator — built by + one code path for both environments. +spec: + steps: + - name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane + description: >- + Run the offline assertion script over the tracked tree (same script ci/l0.sh and + conformance/run.sh --layer l0 execute). + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/ha-control-plane-as-code/check.sh b/conformance/tests/l0-static/ha-control-plane-as-code/check.sh new file mode 100755 index 0000000..8880736 --- /dev/null +++ b/conformance/tests/l0-static/ha-control-plane-as-code/check.sh @@ -0,0 +1,571 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — ha-control-plane-as-code +# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade +# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the +# same multi-node topology." — spec docs/00-vision-and-scope.md). +# +# The elaborating spec text this check quotes and asserts against: +# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)" +# docs/02-environments-and-provisioning.md, table: +# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd +# (k3s server on all 3)" (the SAME cell for dev/test and prod) +# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes" +# docs/02 §"k3s topology decisions": +# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)." +# "Nodes are both control-plane and workload (3-node converged cluster); with only 3 +# nodes we do not isolate control-plane nodes." +# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs" +# +# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives) +# --------------------------------------------------------------------- +# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers: +# +# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control +# plane in BOTH environments, from ONE code path": provable with no cluster, and +# ONLY provable here. A live cluster cannot tell you whether the *next* environment +# brought up from this repo will also be HA, whether a second `--cluster-init` +# founder could exist, or whether an external datastore is configured anywhere. That +# is this check. +# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all +# etcd members, with the API answering through the floating VIP": that is cluster +# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label). +# +# Neither half is sufficient alone, and this file claims only the first. +# +# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod" +# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the +# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every +# group_vars//vars.yml that overrides them — and identifies WHICH environment is +# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by +# its group name. Add a fourth prod node, rename a group, or add a third environment and +# the assertions follow; nothing here restates an address, a hostname or a node count. +# +# GIVEN the repository as git tracks it, +# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's +# Flux overlay are parsed and cross-checked against each other, +# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is +# installed as an `agent` (an agent is not a control-plane member); +# SERVERS: every environment's server group resolves to a quorum-capable, ODD +# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to +# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while +# every other server joins it with `--server` (so a second founder — the +# split-brain shape — cannot be authored); +# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no +# `--disable-etcd` anywhere in the provisioning code, so every server is +# an etcd member (docs/02's "no external datastore", "on all 3"); +# ENDPOINT: the API has a floating endpoint that survives losing the founder — the +# kube-vip VIP is control-plane-only (never Service LB, or it would fight +# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and +# baked into the apiserver serving cert's SAN list ALONG WITH every +# server's own address (so TLS validates whichever node answers); +# ADDRESSES: each environment's control-plane VIP is not one of its node addresses, +# is not that environment's ingress Gateway VIP, and lies OUTSIDE that +# environment's Cilium LoadBalancer pool — the two allocators can never be +# handed the same address; +# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays, +# retargeted by variables only (docs/02 "the *same* Ansible bootstrap +# runs"), so "exercise the same multi-node topology" is a property of the +# code rather than a coincidence of two hand-written inventories. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them): +# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with +# a healthy quorum, that kube-vip is actually floating the VIP, and that the API +# answers through it. All of that is asserted at L2 by +# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test. +# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds, +# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing +# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's +# data-availability claim and does NOT assert control-plane survival. A dedicated +# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and +# is NOT claimed here or at L2. +# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged +# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator +# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This +# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict. +# - THAT kube-vip IS PART OF THE ONE-COMMAND BRING-UP. playbooks/site.yml chains +# node-baseline -> k3s -> cilium -> verify -> flux-bootstrap and does NOT import +# kube-vip.yml, so the floating endpoint is a separate operator step. That is recorded +# here as an observation for the PM, NOT asserted: R-PLAT-01 requires an HA control +# plane, and does not name kube-vip or fix the bring-up order (R-TEST-06 — do not +# invent a requirement's meaning). +# - CILIUM'S OWN API ENDPOINT. `K3S_API_HOST` (clusters/*/kustomization.yaml), which +# becomes Cilium's `k8sServiceHost` for kube-proxy replacement, is the PRIMARY NODE's +# address in both environments — not the floating VIP — even though the prod overlay's +# own comment says it "is the prod CONTROL-PLANE VIP (kube-vip), NOT ... the single node +# address .10". That is a real single-node dependency in the CNI's control-plane path +# and it is ESCALATED to the PM rather than asserted: making this check fail would turn +# the merge gate red over a design question this workstream may not decide (R-TEST-06). +# - IPv6 / dual-stack, and any claim about the edge network. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +NOTES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def note(msg): + NOTES.append(msg) + + +def die(): + print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + + +ANS = ROOT / "infra" / "ansible" +INV = ANS / "inventory" / "hosts.ini" +GV = ANS / "inventory" / "group_vars" +K3S_ROLE = ANS / "roles" / "k3s_server" +VIP_ROLE = ANS / "roles" / "kube_vip" +PLAYBOOKS = ANS / "playbooks" +CLUSTERS = ROOT / "clusters" + +for p, why in ((INV, "the inventory that defines the node set"), + (K3S_ROLE, "the role that installs the k3s control plane"), + (VIP_ROLE, "the role that floats the control-plane VIP"), + (CLUSTERS, "the Flux overlays that identify each environment")): + check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA " + "control plane is authored at all (R-PLAT-01)") +if FAILURES: + die() + +K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml"))) +VIP_TEXT = "\n".join(p.read_text(errors="replace") + for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2")))) + + +# ===================================================================== inventory parse +# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a +# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]` +# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'. +ini_hosts: dict[str, list[str]] = {} +ini_children: dict[str, list[str]] = {} +host_vars: dict[str, dict[str, str]] = {} +cur = None +for line in INV.read_text().splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + cur = m.group(1) + continue + if cur is None or cur.endswith(":vars"): + continue + host = s.split()[0] + if cur.endswith(":children"): + ini_children.setdefault(cur.split(":")[0], []).append(host) + else: + ini_hosts.setdefault(cur, []).append(host) + hv = host_vars.setdefault(host, {}) + for kv in s.split()[1:]: + if "=" in kv: + k, v = kv.split("=", 1) + hv[k] = v + + +def hosts_in(group: str, seen=None) -> set[str]: + seen = seen or set() + if group in seen: + return set() + seen.add(group) + out = set(ini_hosts.get(group, [])) + for child in ini_children.get(group, []): + out |= hosts_in(child, seen) + return out + + +def load_yaml(p: Path) -> dict: + try: + d = yaml.safe_load(p.read_text()) or {} + except yaml.YAMLError: + return {} + return d if isinstance(d, dict) else {} + + +ALL_VARS = load_yaml(GV / "all.yml") + + +# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM +# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds +# the default pair; every group_vars//vars.yml that overrides `k3s_primary_group` is +# another environment. Nothing below knows the words "dev" or "prod" as group names. +class Env: + def __init__(self, label, vars_, path): + self.label = label + self.vars = vars_ + self.path = path + + def var(self, name, default=None): + # group-scoped value wins, else the cluster-wide default from all.yml. + if name in self.vars: + return self.vars[name] + return ALL_VARS.get(name, default) + + +envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")] +for vf in sorted(GV.glob("*/vars.yml")): + v = load_yaml(vf) + if "k3s_primary_group" in v or "k3s_servers_group" in v: + envs.append(Env(str(vf.relative_to(ROOT)), v, vf)) + +check(len(envs) >= 2, + "only ONE environment is derivable from the inventory's R-ENV-02 seam " + "(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires " + "dev/test to exercise the same multi-node topology, so two must be authored") + +# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay. +# roles/flux_bootstrap's own default supplies the value for an environment that does not +# override it, so this is read from code, never guessed from a group name. +fb_default = None +fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml" +if fb_defaults_file.is_file(): + fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path") +check(isinstance(fb_default, str) and fb_default.startswith("clusters/"), + "roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from " + "an inventory environment to its Flux overlay is not expressed in code, so this check " + "cannot tell which environment is prod") + +prod_envs = [] +for e in envs: + overlay = e.var("flux_bootstrap_env_path", fb_default) + e.overlay = overlay if isinstance(overlay, str) else "" + if e.overlay.rstrip("/").endswith("/prod"): + prod_envs.append(e) +check(len(prod_envs) == 1, + f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay " + f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in " + "R-PLAT-01 has no unambiguous referent") + + +# ============================================== overlay knob values (per environment) +# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map; +# each clusters//kustomization.yaml overrides values with JSON6902 patches written as +# - op: replace +# path: /data/ +# value: "" +# (verified shape across both overlays). A key not patched keeps the base value. +BASE_VARS: dict[str, str] = {} +base_cm = CLUSTERS / "base" / "cluster-vars.yaml" +if base_cm.is_file(): + d = load_yaml(base_cm) + BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()} +check(len(BASE_VARS) > 5, + "clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface " + "cannot be read, so the VIP/pool separation cannot be checked") + +_PATCH_RE = re.compile( + r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P[A-Z0-9_]+)\s*\n\s*value:\s*(?P\S.*?)\s*$", + re.M) + + +def overlay_vars(overlay: str) -> dict[str, str]: + out = dict(BASE_VARS) + f = ROOT / overlay / "kustomization.yaml" + if not f.is_file(): + return out + for m in _PATCH_RE.finditer(f.read_text()): + out[m.group("k")] = m.group("v").strip().strip('"').strip("'") + return out + + +# ============================================================ PER-ENVIRONMENT TOPOLOGY +def as_ip(s): + try: + return ipaddress.ip_address(str(s).strip()) + except ValueError: + return None + + +for e in envs: + L = e.label + pg = e.var("k3s_primary_group") + sg = e.var("k3s_servers_group") + check(isinstance(pg, str) and isinstance(sg, str) and pg and sg, + f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot " + "resolve 'my environment's primary/servers' and would only ever build dev") + if not (isinstance(pg, str) and isinstance(sg, str)): + continue + primaries = hosts_in(pg) + servers = hosts_in(sg) + + # --- SERVERS: quorum-capable, one founder, prod exactly three --------------------- + check(len(primaries) == 1, + f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run " + "`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no " + "cluster at all") + check(primaries and primaries <= servers, + f"{L}: the primary {sorted(primaries)} is not a member of the servers group " + f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or " + "it is not part of the HA control plane it founds") + n = len(servers) + check(n >= 3, + f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes " + "'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd " + "cluster of fewer than 3 members tolerates ZERO losses — it is not HA") + check(n % 2 == 1, + f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd " + f"member count adds a member without adding fault tolerance and introduces a tie: " + f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}") + if n: + tolerance = n - (n // 2 + 1) + check(tolerance >= 1, + f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate " + f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one") + if e in prod_envs: + check(n == 3, + f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: " + "'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'") + else: + check(n >= 2, + f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer " + "nodes than prod but requires dev/test to 'exercise the same multi-node " + "topology' — one node exercises no topology") + + # --- ADDRESSES: the VIP can collide with nothing ---------------------------------- + vip_raw = e.var("kube_vip_address") + vip = as_ip(vip_raw) + check(vip is not None, + f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has " + "no floating endpoint, so losing the founder takes the API endpoint with it") + node_ips = {} + for h in sorted(servers): + ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", "")) + check(ip is not None, + f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join " + "target and the serving-cert SAN list all derive from it") + if ip is not None: + node_ips[h] = ip + if vip is not None: + check(vip not in set(node_ips.values()), + f"{L}: kube_vip_address {vip} is ALSO a node address " + f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a " + "node's own address cannot float off that node") + ov = overlay_vars(getattr(e, "overlay", "")) + gw = as_ip(ov.get("GATEWAY_VIP", "")) + lo = as_ip(ov.get("CILIUM_LB_POOL_START", "")) + hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", "")) + check(gw is not None and lo is not None and hi is not None, + f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + " + "CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and " + "the Service-LB allocator cannot be checked") + if gw is not None: + check(vip != gw, + f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME " + f"address ({vip}) — kube-vip and Cilium would both ARP for it") + if lo is not None and hi is not None: + check(not (lo <= vip <= hi), + f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium " + f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a " + "Service, and the two allocators would fight over ARP for it") + check(lo <= hi, + f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be " + "used to prove the VIP is outside it") + +# ==================================================================== DISTRO: k3s only +check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M) + is not None, + "roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane " + "version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)") +check("get.k3s.io" in K3S_TEXT, + "roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the " + "distro (ADR-0001) and no other distro may stand in for it") +check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None, + "roles/k3s_server never composes the `server` subcommand — without it k3s installs an " + "AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')") +for bad, why in ( + (r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 " + "nodes (no external datastore)')"), + (r"datastore-endpoint\s*:", "an external datastore via config.yaml"), + (r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not " + "an etcd member (docs/02: etcd 'across all 3 nodes')"), + (r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"), +): + hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*")) + if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini") + and re.search(bad, p.read_text(errors="replace"))] + check(not hits, + f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA " + f"mode, and {bad!r} contradicts it") + +# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH +check(re.search(r"--cluster-init", K3S_TEXT) is not None, + "roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never " + "founded, so there is no HA datastore to join") +init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'", + K3S_TEXT) +check(init_when is not None, + "the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an " + "ungated founder flag would re-found the cluster on every server (split brain)") +join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT) +check(join is not None, + "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` " + "— without a join target the servers are separate single-node clusters") +join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'", + K3S_TEXT, re.S) +check(join_when is not None, + "the join install task is not gated on k3s_server_mode == 'secondary' — the founder " + "would also try to join itself") +k3s_play = PLAYBOOKS / "k3s.yml" +check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint") +if k3s_play.is_file(): + play_txt = k3s_play.read_text() + modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt) + check(sorted(modes) == ["primary", "secondary"], + f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and " + "one secondary play must exist, or the founder/joiner split is not authored") + check(re.search(r"serial:\s*1", play_txt) is not None, + "playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — " + "converging two of three servers at once drops embedded etcd below quorum, which " + "is exactly the outage an HA control plane exists to avoid") + hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M) + check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns), + f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal " + "group name means this play can only ever build ONE environment, so prod cannot " + "be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')") + +# =========================================== SERVING-CERT SANs: VIP + every server +sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", "")) +check("kube_vip_address" in sans, + "roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's " + "serving cert would carry no SAN for the floating VIP, so every client reaching the " + "API through it fails TLS validation and the HA endpoint is unusable") +check("k3s_servers_group" in sans and "k3s_node_ip" in sans, + "k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the " + "cert must be valid for WHICHEVER server answers behind the VIP, and per-environment " + "resolution of that set is what keeps one role serving both environments (R-ENV-02)") +check(re.search(r"tls-san:", K3S_TEXT) is not None, + "the SAN list is passed only as an install flag and never persisted to " + "/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a " + "no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert") + +# ============================================ FLOATING ENDPOINT: kube-vip on every server +# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose. +# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with +# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the +# thing that ships.) Parser contract: the template writes each env var as an adjacent +# `- name: X` / `value: "Y"` pair; comment lines are dropped first. +VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2")) +check(len(VIP_TEMPLATES) >= 1, + "roles/kube_vip ships no manifest template — nothing deploys the API VIP") +vip_manifest = "\n".join( + "\n".join(l for l in p.read_text(errors="replace").splitlines() + if not l.lstrip().startswith("#")) + for p in VIP_TEMPLATES) +vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?", + vip_manifest)) +check(len(vip_env) >= 6, + f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the " + "parser lost its grip on the file, so the assertions below would assert nothing") +check(vip_env.get("cp_enable") == "true", + f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane " + "mode must be ON or it floats no API VIP at all") +check(vip_env.get("svc_enable") == "false", + f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be " + "false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM " + "for the same addresses (ADR-0002 gives Service LB to Cilium)") +check(vip_env.get("vip_leaderelection") == "true", + f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} " + "— without leader election the VIP has no defined owner to fail over FROM, which is " + "the whole HA mechanism") +check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}", + f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the " + "environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on " + "prod (R-ENV-02) and would not match the serving-cert SAN list") +check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None, + "the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its " + "node, which is the opposite of a floating endpoint") +vip_play = PLAYBOOKS / "kube-vip.yml" +check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP") +if vip_play.is_file(): + vp = vip_play.read_text() + vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M) + check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0], + f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers " + "group through the same environment-scoped variable the other plays use, so the " + "manifest exists on EVERY server and the VIP survives losing any one of them") +check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None + and re.search(r"k3s_data_dir", VIP_TEXT) is not None, + "roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies " + "/server/manifests, so an environment that moves --data-dir would write the " + "manifest somewhere k3s never reads and come up with no VIP at all") + +# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS +# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation +# step differs". A prod-only role or playbook would be a fork of the control-plane path. +role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()} +play_names = {p.name for p in PLAYBOOKS.glob("*.yml")} +env_words = re.compile(r"prod|dev", re.I) +forked = sorted(n for n in (role_names | play_names) + if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml")) +check(not forked, + f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by " + "the SAME code retargeted by variables (docs/02), not by a per-environment fork") +for e in envs: + if e.path.name != "vars.yml": + continue + # Every key a non-default environment sets must be a variable something already reads, + # or it is an invented knob nothing consumes (a silent no-op that looks like config). + consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*")) + if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh")) + for k in sorted(e.vars): + if k.startswith("ansible_"): + continue + check(consumers.count(k) > 1, + f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an " + "environment override that nothing reads is not a scoped override, it is dead " + "configuration (and would make prod silently inherit a dev default)") + +# --------------------------------------------------------------------------- verdict +# NO-OP GUARD: a check that quietly asserts nothing is worse than no check. +check(CHECKS >= 40, + f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass " + "would be meaningless") + +if FAILURES: + die() +for n in NOTES: + print(f" note: {n}") +summary = "; ".join( + f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]=" + f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers" + for e in envs) +print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. " + "R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, " + "embedded etcd on every server with no external datastore, a leader-elected " + "control-plane-only VIP on every server that is SAN'd into the serving cert and " + "collides with no other allocator, from one code path for both environments. The LIVE " + "cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane " + "SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).") +PY diff --git a/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml index 62a892b..36e9be7 100644 --- a/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml @@ -153,3 +153,20 @@ spec: finally: - script: content: 'echo "cleanup handled by chainsaw namespace teardown"' + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo DEFAULT_DENY_PROVEN_OK + check: + (contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true diff --git a/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml b/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml new file mode 100644 index 0000000..d2f6801 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml @@ -0,0 +1,199 @@ +# ============================================================================= +# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE] +# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster") +# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by +# routing rules ... realized as two paths — the local `internet` path and the +# in-cluster `vpn` path"). +# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is +# simply the default route out". +# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.** +# Every pod — and the nodes themselves — egress the local/private WAN by default and +# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the +# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0` +# uses the VPN." +# +# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT +# -------------------------------------------------------------- +# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their +# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing +# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds +# that pod, and asserts the two claims that make a rule-selected path set complete: +# +# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the +# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is +# the local one. If the steering rule ever widened (an empty selector, a +# matchExpression, a second rule), this pod's IP would move to the tunnel and +# this assertion is what would notice. +# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one +# pass: three pods, two distinct public IPs. "Multiple egress paths" is a +# simultaneity claim, not a claim that one switch has two positions. +# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918 +# destination over the LOCAL path while its public traffic is tunnelled. The +# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod +# uses BOTH paths at once. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a +# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike +# and survives the WAN or the VPN provider rotating addresses. +# +# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three +# times, differing only in the class they declare (or do not), +# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod +# also resolves a name through the in-cluster resolver, +# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out +# is the WAN, and the VPN is strictly opt-in); +# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path); +# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not +# swallowed by the tunnel), so the same pod is using both paths at once. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one +# steering rule is authored, that it selects only the vpn class, that no `ip rule` is +# unscoped and no nftables DROP unqualified is the L0 half, in +# tests/l0-static/egress-paths-selected-by-rules (same requirement label). +# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo +# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need +# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in +# this suite does. +# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn` +# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically +# by the L0 half, not here. +# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That +# is a host-level fact this test cannot see from a pod. +# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster, +# so a WAN outage or the echo being down fails this test. That is the same dependency +# tests/l2-behavioral/egress-class already carries; the alternative — trusting the +# cluster's own account of which path it used — would not be a behavioural test at all. +# +# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside +# that namespace is created or changed — no label on a platform object, no policy edit. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-default-path + labels: + conformance.home.k3s/R-NET-04: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-04" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public + IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the + default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two + paths live at once — and the vpn-class pod still reaches in-cluster destinations over the + local path, so one pod uses both paths simultaneously. +spec: + timeouts: + apply: 60s + assert: 2m + exec: 3m + steps: + # ----------------------------------------------------------------------------------- + - name: given-three-pods-differing-only-in-their-declared-egress-class + try: + - apply: + file: fixtures/three-egress-pods.yaml + # Bare pods (restartPolicy: Never) — Ready == Running here. + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-unlabelled } + status: { phase: Running } + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-internet } + status: { phase: Running } + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: egress-vpn } + status: { phase: Running } + # ----------------------------------------------------------------------------------- + - name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too + try: + - script: + content: | + set -eu + ns="$NAMESPACE" + get_ip() { + # Echo the pod's observed public IP, or empty on failure. + kubectl -n "$ns" exec "$1" -c curl -- \ + curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true + } + + # Poll until all three answer AND the vpn class has diverged from the WAN: + # Cilium may still be programming the egress datapath just after the pods go + # Ready. Bounded by the loop, never a bare sleep-and-hope. + none="" inet="" vpn="" + for i in $(seq 1 12); do + [ -n "$none" ] || none="$(get_ip egress-unlabelled)" + [ -n "$inet" ] || inet="$(get_ip egress-internet)" + [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)" + if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then + [ "$vpn" != "$inet" ] && break + # Steering may not have converged; re-read the vpn pod only. + vpn="" + fi + sleep 5 + done + echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn" + + [ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; } + [ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; } + [ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; } + + # TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN — + # the same WAN the explicitly-internet pod uses. + if [ "$none" != "$inet" ]; then + echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2 + exit 1 + fi + # COEXISTENCE: and the vpn path is a genuinely different path, live right now. + if [ "$vpn" = "$inet" ]; then + echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2 + exit 1 + fi + echo "EGRESS_DEFAULT_PATH_IS_WAN_OK" + check: + (contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-the-vpn-class-pod-still-reaches-private-destinations-locally + try: + - script: + # PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all + # RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the + # local datapath. The cluster resolver is an in-cluster (RFC1918) address reached + # from the pod's own /etc/resolv.conf — no literal is written here — so a + # successful lookup from the vpn-class pod proves the tunnel did not swallow its + # local traffic, i.e. ONE pod is using BOTH paths at the same time. + content: | + set -eu + ns="$NAMESPACE" + resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ + sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)" + [ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; } + echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)" + + # A cluster-internal name, resolvable only by the in-cluster resolver: reaching + # it proves the private-destination path works from a vpn-class pod. + out="" + for i in $(seq 1 6); do + out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ + nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)" + case "$out" in *"Address"*) break ;; esac + sleep 5 + done + case "$out" in + *Address*) : ;; + *) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;; + esac + echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK" + check: + (contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true diff --git a/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml b/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml new file mode 100644 index 0000000..0815781 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml @@ -0,0 +1,67 @@ +# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing +# ONLY in how (or whether) they declare an egress class: +# +# egress-unlabelled NO egress-class label at all -> selected by NO routing rule +# egress-internet egress-class: internet -> the declared default path +# egress-vpn egress-class: vpn -> the only class any rule steers +# +# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress +# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no +# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must +# come out the same WAN as the explicitly-`internet` pod. That is the totality half of +# R-NET-04's "selected by routing rules" — absence of a rule is itself a path. +# +# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an +# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in +# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee +# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe +# the datapath's own default at all. +apiVersion: v1 +kind: Pod +metadata: + name: egress-unlabelled + labels: + app: egress-default-path-test +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never +--- +apiVersion: v1 +kind: Pod +metadata: + name: egress-internet + labels: + app: egress-default-path-test + homelab.keithjackson.dev/egress-class: internet +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never +--- +apiVersion: v1 +kind: Pod +metadata: + name: egress-vpn + labels: + app: egress-default-path-test + homelab.keithjackson.dev/egress-class: vpn +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never diff --git a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml index 32301b4..ccf8e95 100644 --- a/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml @@ -313,3 +313,20 @@ spec: apiVersion: v1 kind: Namespace name: conformance-gw-probe + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK + check: + (contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true diff --git a/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml b/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml new file mode 100644 index 0000000..43db761 --- /dev/null +++ b/conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml @@ -0,0 +1,373 @@ +# ============================================================================= +# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane (R-PLAT-01) [RUNNABLE] +# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster") +# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with +# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST +# exercise the same multi-node topology.") +# docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02" +# docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA +# embedded etcd (k3s server on all 3)" +# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes" +# docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across +# all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload +# (3-node converged cluster)". +# +# WHY L2 — AND WHAT L0 ALREADY DID +# -------------------------------- +# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED +# half offline: the repo builds a 3-server embedded-etcd control plane in both environments, +# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving +# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory +# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test: +# every assertion below is read from the cluster the suite is pointed at. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written +# here. The expected node count comes from the cluster's OWN declared identity +# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same +# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip +# DaemonSet the cluster is actually running; the server addresses come from the Node +# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars +# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY +# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the +# environment from the cluster, never from a flag or a literal, which is precisely the +# mechanism the tests/README.md convention prescribes. +# +# GIVEN the cluster this suite is pointed at, +# WHEN its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected, +# THEN DISTRO: every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s; +# SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node +# topology docs/02 fixes — no agent-only node, no control-plane-only node), +# the count is ODD and quorum-capable (tolerates >= 1 loss), and it is +# EXACTLY three when the cluster declares itself prod; +# ETCD: every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps +# `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so +# the datastore is embedded and replicated across all of them; +# VIP: kube-vip is Ready on EVERY control-plane node, is control-plane-only +# (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and +# the address it floats is neither a node address, nor the ingress Gateway +# VIP, nor inside the Cilium LoadBalancer pool; +# ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that +# validates against the cluster CA for the VIP itself — i.e. the VIP really +# fronts the API and the serving cert really carries it; +# EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own +# address, again with a CA-valid cert for that address. This is the assertion +# that separates a real HA control plane from one server plus two joiners: +# if only the founder ran an apiserver, the other addresses would refuse. +# +# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node +# addresses from the RUNNER. That works from a host on the node LAN (verified against prod) +# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a +# forwarded port — i.e. it would have made the test a property of where it was launched +# from. A Job on the cluster is on the node network in both environments, so the probe is +# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than +# as a fixture file because its address list is DISCOVERED from the cluster at run time — +# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): +# - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP +# migrates, and that the API stays reachable when a server dies is an L3 fault-injection +# claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss +# drains a node for R-STATE-02's data-availability claim and asserts nothing about the +# control plane). This is the largest remaining R-PLAT-01 gap. +# - THE NEXT environment. That the repo would build an HA control plane again is the L0 +# half (tests/l0-static/ha-control-plane-as-code). +# - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of +# infra/PROD-PROVISIONING.md). +# - Cilium's own `k8sServiceHost` (cluster-vars `K3S_API_HOST`) is the PRIMARY NODE's +# address, not this VIP, in both environments. That single-node dependency in the CNI's +# control-plane path is escalated to the PM, not asserted here (R-TEST-06). +# - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes +# it only on the node-local etcd metrics port, which is not reachable through the +# Kubernetes API this test is allowed to use. +# +# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it +# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test +# creates nothing outside its own namespace and mutates no platform object. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: ha-control-plane + labels: + conformance.home.k3s/R-PLAT-01: "true" + annotations: + conformance.home.k3s/requirements: "R-PLAT-01" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver + endpoints are probed, then every node is a k3s server that is both control-plane and a + registered embedded-etcd member, the count is odd and quorum-capable (exactly three + when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that + collides with no allocator, and the apiserver answers /readyz through that VIP AND on + every server's own address with a CA-valid cert. +spec: + timeouts: + apply: 60s + assert: 2m + exec: 3m + # -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE + # CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes + # from, so these are the same values the platform was rendered with. Test-level, so + # they are in scope for every step (step outputs/bindings are step-scoped). + bindings: + - name: clusterEnv + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV) + - name: gatewayVip + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP) + - name: lbPoolStart + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START) + - name: lbPoolStop + value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP) + steps: + # ----------------------------------------------------------------------------------- + - name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane + description: >- + Read the Node objects: k3s on every node, every node both control-plane and a + registered embedded-etcd member, an odd quorum-capable count, exactly three in prod. + try: + - script: + env: + - name: CLUSTER_ENV + value: ($clusterEnv) + content: | + set -eu + fail() { echo "FAIL: $*" >&2; exit 1; } + + total="$(kubectl get nodes -o name | grep -c . || true)" + [ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all" + + # --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) ----- + # A non-k3s kubelet here would mean the platform is not running on k3s. + notk3s="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \ + | grep -v '+k3s' | grep -c . || true)" + [ "$notk3s" = "0" ] \ + || fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s" + + # --- every node is BOTH control-plane AND etcd (docs/02 converged topology) - + # Counted by label EXISTENCE, so it does not matter whether k3s writes "true" + # or "". A control-plane node without the etcd role would be a server with an + # external/absent datastore; an etcd node without control-plane would be an + # isolated datastore member. Both are excluded by requiring all three counts + # to be equal. + cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)" + etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)" + echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-}" + [ "$cp" = "$total" ] \ + || fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')" + [ "$etcd" = "$total" ] \ + || fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'" + + # --- every node is a REGISTERED embedded-etcd member ----------------------- + # k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the + # embedded-etcd cluster. The role label alone is a label; this annotation is + # the datastore's own record of membership. + members="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \ + | grep -c . || true)" + [ "$members" = "$total" ] \ + || fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server" + + # --- Ready --------------------------------------------------------------- + notready="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \ + | grep -v '=True$' | grep -c . || true)" + [ "$notready" = "0" ] \ + || fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test" + + # --- quorum arithmetic --------------------------------------------------- + # Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is + # N - quorum; 'production-grade HA' must survive at least one loss, which + # rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds + # a tie, so it is also rejected. + quorum=$(( total / 2 + 1 )) + tolerance=$(( total - quorum )) + echo "quorum=$quorum tolerance=$tolerance" + [ "$tolerance" -ge 1 ] \ + || fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server" + [ $(( total % 2 )) -eq 1 ] \ + || fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance" + + # --- the count R-PLAT-01 itself fixes, per environment -------------------- + # The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a + # flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST + # exercise the same multi-node topology'. + case "${CLUSTER_ENV:-}" in + prod) + [ "$total" = "3" ] \ + || fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;; + "") + fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;; + *) + [ "$total" -ge 2 ] \ + || fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;; + esac + echo "HA_TOPOLOGY_OK" + check: + (contains($stdout, 'HA_TOPOLOGY_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-a-control-plane-only-vip-is-floated-on-every-server + description: >- + Read the running kube-vip DaemonSet: Ready on every control-plane node, + control-plane-only, and floating an address that collides with no other allocator. + try: + - script: + env: + - name: GATEWAY_VIP + value: ($gatewayVip) + - name: LB_POOL_START + value: ($lbPoolStart) + - name: LB_POOL_STOP + value: ($lbPoolStop) + content: | + set -eu + fail() { echo "FAIL: $*" >&2; exit 1; } + # dotted quad -> integer, for range comparison without ipcalc/python. + n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; } + + # The DaemonSet is found by the label roles/kube_vip stamps on it, not by a + # hardcoded name in a namespace this test guessed. + ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)" + [ -n "$ds" ] \ + || fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it" + + desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')" + ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')" + cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)" + vip="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')" + cp_enable="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')" + svc_enable="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')" + echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable" + + [ -n "$vip" ] \ + || fail "the kube-vip DaemonSet announces no address — nothing is floated" + [ "$cp_enable" = "true" ] \ + || fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP" + [ "$svc_enable" = "false" ] \ + || fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)" + [ "$desired" = "$cp" ] \ + || fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane" + [ "$ready" = "$desired" ] \ + || fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over" + + # --- the VIP collides with nothing -------------------------------------- + node_ips="$(kubectl get nodes \ + -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \ + | grep -c "^${vip}$" || true)" + [ "$node_ips" = "0" ] \ + || fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node" + [ "$vip" != "${GATEWAY_VIP:-}" ] \ + || fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it" + if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then + v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")" + [ "$lo" -le "$hi" ] \ + || fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it" + if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then + fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service" + fi + else + fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)" + fi + echo "HA_VIP_CONTROL_PLANE_ONLY_OK" + check: + (contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true + # ----------------------------------------------------------------------------------- + - name: then-the-apiserver-answers-through-the-vip-and-on-every-server + description: >- + From inside the cluster, probe /readyz on the floated VIP and on every server's own + address, validating TLS against the cluster CA for the address dialled. + try: + - script: + # The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the + # Job cannot be a static fixture — writing the addresses down is the + # per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the + # same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped + # fixtures. The Job lives in chainsaw's per-test namespace and is torn down with + # it; nothing outside that namespace is created or changed. + content: | + set -eu + ns="$NAMESPACE" + fail() { echo "FAIL: $*" >&2; exit 1; } + + ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)" + vip="$(kubectl -n kube-system get "$ds" \ + -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')" + servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \ + -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')" + [ -n "$vip" ] || fail "no floated VIP to probe" + [ -n "$servers" ] || fail "no control-plane node addresses to probe" + echo "probing vip=$vip servers=$servers" + + # The probe pod authenticates with its own ServiceAccount token and validates + # TLS against the cluster CA, both from the projected SA volume. /readyz is + # authorized for any authenticated identity via the built-in + # system:public-info-viewer role — anonymous access is NOT relied on (verified: + # an unauthenticated /readyz on this platform answers 401). + kubectl -n "$ns" apply -f - <&1)" || rc=1 + echo "vip \$VIP -> [\$out]" + if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then + echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)" + exit 1 + fi + for s in \$SERVERS; do + rc=0 + out="\$(probe "\$s" 2>&1)" || rc=1 + echo "server \$s -> [\$out]" + if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then + echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node" + exit 1 + fi + done + echo "HA_API_ENDPOINT_OK" + EOF + + # Bounded wait; on failure surface the probe's own output before failing. + if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then + kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true + fail "the in-cluster API-endpoint probe did not complete — see its log above" + fi + logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)" + echo "$logs" + case "$logs" in + *HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;; + *) fail "the probe completed without reporting success" ;; + esac + check: + (contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true diff --git a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml index 2cc6485..babdbaf 100644 --- a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml @@ -93,3 +93,20 @@ spec: kind: Job metadata: { name: tls-probe } status: { succeeded: 1 } + # ----------------------------------------------------------------------------------- + # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so + # chainsaw already fails the test before reaching this step if any of them fails — but a + # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip + # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records + # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass + # forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list + # — which is precisely what docs/07 asks for ("These proofs are part of the platform + # acceptance suite, not one-time checks"). This step exists only to emit that token, and it + # is reached only when everything above has passed. Asserted by + # tests/l0-static/auditability-proofs-producible. + - name: then-emit-the-proof-of-pass-sentinel + try: + - script: + content: echo TLS_TRUSTED_PROVEN_OK + check: + (contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true