feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
5ec1460532
commit
cb1916f092
12 files changed
+2488
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
||||
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
||||
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
||||
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
||||
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
||||
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
||||
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
||||
# be produced on demand."
|
||||
#
|
||||
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
||||
# -----------------------------------------------------------------------------------
|
||||
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
||||
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
||||
# failure modes are asserted instead, each by DRIVING the real machinery:
|
||||
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
||||
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
||||
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
||||
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
||||
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
||||
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
||||
# attribute — a proof that passes but records nothing leaves the operator with a green
|
||||
# terminal and no evidence;
|
||||
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
||||
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
||||
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
||||
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
||||
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
||||
# ci/ or run.sh ran before this test existed.
|
||||
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
||||
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
||||
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
||||
#
|
||||
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
||||
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
||||
# anyone needs it.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
||||
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
||||
# and two synthetic forgeries are presented to them,
|
||||
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
||||
# documented one-command invocation selects and asserts them, each is capturable as a
|
||||
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
||||
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
||||
# it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
||||
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
||||
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
||||
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
||||
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
||||
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
||||
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
||||
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
||||
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
||||
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
||||
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
||||
# would need signed history.
|
||||
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
||||
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
||||
# not re-asserted here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: auditability-proofs-producible
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
||||
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
||||
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
||||
of docs/07's five auditability proofs has an executable test at the layer the spec
|
||||
requires, is selectable and asserted by the documented one-command invocation, is
|
||||
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
||||
evidence.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
||||
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
||||
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
||||
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
||||
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
||||
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
||||
# the wrong trade (R-TEST-06).
|
||||
@@ -0,0 +1,431 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — auditability-proofs-producible
|
||||
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
|
||||
# prove isolation, egress path, cert validity, and backup integrity)."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
|
||||
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
|
||||
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
|
||||
# denied-probe test.
|
||||
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
||||
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
||||
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
|
||||
# status + external check.
|
||||
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
||||
# "These proofs are part of the platform acceptance suite, not one-time checks."
|
||||
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
|
||||
#
|
||||
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
|
||||
# ------------------------------------------------------------
|
||||
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
|
||||
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
|
||||
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
|
||||
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
|
||||
# by DRIVING the real machinery rather than by reading files:
|
||||
#
|
||||
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
|
||||
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
|
||||
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
|
||||
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
|
||||
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
|
||||
# repository's own status report.
|
||||
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
|
||||
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
|
||||
# their coverage (that flag is the merge gate's own path: it passes --require to
|
||||
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
|
||||
# gate wiring break, R-SEC-03 fails.
|
||||
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
|
||||
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
|
||||
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
|
||||
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
|
||||
# instead of silently making the requirement unprovable.
|
||||
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
|
||||
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
|
||||
# real tools, in a throwaway directory:
|
||||
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
|
||||
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
|
||||
# test. record_run.py must refuse to call it `proven`.
|
||||
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
|
||||
# implementing code — status.py must degrade it to `stale-proof`, never green.
|
||||
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
|
||||
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
|
||||
# were themselves unexercised in the merge gate until this check.
|
||||
#
|
||||
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
|
||||
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
|
||||
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
|
||||
# than quietly ignoring it.
|
||||
#
|
||||
# WHY L0: every clause above is a property of the repository and its own tooling, provable
|
||||
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
|
||||
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
|
||||
# with one.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
|
||||
# documented command, capturable as evidence, and that the evidence chain refuses
|
||||
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
|
||||
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
|
||||
# proof, which is what is asserted.
|
||||
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
|
||||
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
|
||||
# denied-probe half, which is the half that is automated. The Hubble half remains an
|
||||
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
|
||||
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
|
||||
# report's prose, the ledger's provenance fields) is a usability property, not a testable
|
||||
# one.
|
||||
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
|
||||
# nothing here can prove a row was never removed from it (that would need signed history).
|
||||
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
|
||||
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
|
||||
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
|
||||
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||||
export SPEC_REPO
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
CONF = ROOT / "conformance"
|
||||
LIB = CONF / "lib"
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run(cmd, **kw) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
|
||||
|
||||
|
||||
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
|
||||
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
|
||||
# reqs = the requirement(s) whose verifying test IS that proof
|
||||
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
|
||||
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
|
||||
PROOFS = {
|
||||
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
|
||||
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
|
||||
"denied-probe test"},
|
||||
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
|
||||
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
|
||||
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
|
||||
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
|
||||
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
|
||||
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
|
||||
"— via cert status + external check"},
|
||||
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
|
||||
"why": '"irreplaceable data is restorable" — via a DR drill'},
|
||||
}
|
||||
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
|
||||
|
||||
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
|
||||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||||
sec_doc = spec / "docs" / "07-security-and-secrets.md"
|
||||
if sec_doc.is_file():
|
||||
text = sec_doc.read_text()
|
||||
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
|
||||
re.M | re.S)
|
||||
check(m is not None,
|
||||
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
|
||||
"the proof list this check is coupled to has moved or been removed, so the mapping "
|
||||
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
|
||||
if m:
|
||||
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
|
||||
check(set(labels) == set(PROOFS),
|
||||
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
|
||||
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
|
||||
"test, and a proof this check does not know about would be silently unverified")
|
||||
check(len(labels) == 5,
|
||||
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
|
||||
"gate says 'The five auditability proofs above can be produced on demand' — the "
|
||||
"prose and the list have drifted apart")
|
||||
else:
|
||||
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
|
||||
"re-derived from docs/07, so the mapping below was used as written. Every other "
|
||||
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
|
||||
|
||||
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
|
||||
# Driven through lib/status.py so this check and the repository's own status report cannot
|
||||
# disagree about what is executable and what is a scaffold.
|
||||
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
|
||||
try:
|
||||
out = tmp / "status"
|
||||
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out)])
|
||||
check(st.returncode == 0,
|
||||
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
|
||||
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
|
||||
status = {}
|
||||
sj = out / "status.json"
|
||||
if sj.is_file():
|
||||
status = json.loads(sj.read_text())
|
||||
check(bool(status.get("by_requirement")),
|
||||
"lib/status.py produced no per-requirement report — the audit posture of each "
|
||||
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
|
||||
by_req = status.get("by_requirement", {})
|
||||
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid)
|
||||
check(rec is not None,
|
||||
f"proof '{label}' maps to {rid}, which the status report does not know — "
|
||||
"either the registry lost the requirement or this mapping is stale")
|
||||
if not rec:
|
||||
continue
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
executable = [t for t in tests if t.get("state", "").startswith("executable")]
|
||||
check(bool(executable),
|
||||
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
|
||||
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
|
||||
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
|
||||
"operator cannot prove this property")
|
||||
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
|
||||
check(bool(at_layer),
|
||||
f"proof '{label}' ({rid}) has no executable test at layer(s) "
|
||||
f"{sorted(spec_proof['layers'])} — it has "
|
||||
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
|
||||
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
|
||||
"weaker layer does not discharge it (a rendering check is not a drill)")
|
||||
|
||||
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
|
||||
# The resolver: --requirements must translate the five R-IDs into real test dirs.
|
||||
resolver = run(["bash", "-c",
|
||||
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
|
||||
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
|
||||
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
|
||||
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
|
||||
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
|
||||
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
|
||||
"acceptance gate depends on ('produced on demand') cannot select the proofs")
|
||||
|
||||
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
|
||||
# zero exit here is the repository's own statement that all five proofs are covered.
|
||||
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
|
||||
check(gate.returncode == 0,
|
||||
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
|
||||
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
|
||||
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
|
||||
|
||||
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
|
||||
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
|
||||
# token can pass forever and never be recorded as proof of anything.
|
||||
sys.path.insert(0, str(LIB))
|
||||
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
|
||||
sys.dont_write_bytecode = True
|
||||
import record_run as rr # noqa: E402
|
||||
|
||||
smap = rr.sentinel_map(CONF / "tests")
|
||||
check(len(smap) >= 10,
|
||||
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
|
||||
"grip on the test tree, so the assertions below would assert nothing")
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid) or {}
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
names = [t.get("name") for t in tests
|
||||
if t.get("state", "").startswith("executable")
|
||||
and t.get("layer") in spec_proof["layers"]]
|
||||
for name in names:
|
||||
ent = smap.get(name)
|
||||
check(ent is not None,
|
||||
f"proof '{label}': record_run.py does not know the test {name!r} — its "
|
||||
"result could never be attributed to a ledger row")
|
||||
if not ent:
|
||||
continue
|
||||
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
continue # offline layers are truthful by JUnit alone
|
||||
check(bool(ent.get("pass")),
|
||||
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
|
||||
"so lib/record_run.py can never record it as proven. It would go green "
|
||||
"in the terminal and leave the operator with no evidence — exactly the "
|
||||
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
|
||||
"platform acceptance suite, not one-time checks')")
|
||||
check(bool(ent.get("terminal_pass")),
|
||||
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
|
||||
"record_run.py cannot decide its verdict")
|
||||
|
||||
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
|
||||
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
|
||||
for tool in ("record_run.py", "status.py"):
|
||||
r = run([sys.executable, str(LIB / tool), "--selftest"])
|
||||
check(r.returncode == 0,
|
||||
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
|
||||
"self-skipped or stale run from being recorded as proof are broken, so nothing "
|
||||
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
|
||||
|
||||
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
|
||||
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
|
||||
forge_target, forge_skip = None, None
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
for name, ent in smap.items():
|
||||
if rid in (ent.get("requirements") or []) and ent.get("skip") \
|
||||
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
forge_target, forge_skip = name, ent["skip"][0]
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
check(forge_target is not None,
|
||||
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
|
||||
"could not be mounted. That is not a pass: it means this check could not verify "
|
||||
"that a self-skip is refused as proof, which is the whole point of the ledger")
|
||||
if forge_target:
|
||||
run_dir = tmp / "20260101T000000Z-forged"
|
||||
run_dir.mkdir(parents=True)
|
||||
(run_dir / "report.junit.xml").write_text(
|
||||
'<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
|
||||
f'<testcase name="{forge_target}" classname="forged"/>'
|
||||
"</testsuite></testsuites>\n")
|
||||
(run_dir / "run.log").write_text(
|
||||
f"=== RUN chainsaw/{forge_target}\n"
|
||||
f" === STDOUT\n {forge_skip}\n")
|
||||
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
|
||||
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
|
||||
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
|
||||
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
|
||||
forged.stdout, re.M)
|
||||
check(verdicts and all(v != "proven" for v in verdicts),
|
||||
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
|
||||
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
|
||||
f"({forge_skip}). A self-skip must never become proof — with this hole every "
|
||||
"audit proof in docs/07 could be produced by a cluster that skipped it")
|
||||
check(not (tmp / "never-written.jsonl").exists(),
|
||||
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
|
||||
"claiming not to cannot be trusted to record only real runs")
|
||||
|
||||
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
|
||||
# status.py must degrade it to stale-proof (not green), or an old green run would keep
|
||||
# vouching for code it never executed.
|
||||
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
|
||||
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
|
||||
check(len(old_sha) == 40,
|
||||
"the repository's root commit could not be resolved, so the stale-proof guard could "
|
||||
"not be exercised (that is a gap in this check, not a pass)")
|
||||
if len(old_sha) == 40 and forge_target:
|
||||
stale_ledger = tmp / "stale.jsonl"
|
||||
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
|
||||
stale_ledger.write_text(json.dumps({
|
||||
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
|
||||
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
|
||||
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
|
||||
"requirements": [rid], "junit_result": "passed",
|
||||
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
|
||||
"proof": "proven", "run_id": "20260101T000000Z-forged",
|
||||
"note": "synthetic row written by the R-SEC-03 conformance check",
|
||||
}) + "\n")
|
||||
out2 = tmp / "status-stale"
|
||||
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out2),
|
||||
"--ledger", str(stale_ledger)])
|
||||
check(st2.returncode == 0,
|
||||
f"lib/status.py failed against a synthetic ledger: "
|
||||
f"{(st2.stderr or st2.stdout).strip()[:300]}")
|
||||
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
|
||||
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
|
||||
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
|
||||
check(per.get(forge_target) == "stale-proof",
|
||||
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
|
||||
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
|
||||
"proof from a run that predates the code it vouches for is not evidence, and "
|
||||
"honouring it would let one old green run certify every later change")
|
||||
check((rec2.get("proof") or {}).get("state") != "proven",
|
||||
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
|
||||
"freshness guard is not holding")
|
||||
|
||||
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
|
||||
ledger = CONF / "runs" / "ledger.jsonl"
|
||||
if ledger.is_file():
|
||||
rows, bad = [], []
|
||||
for i, line in enumerate(ledger.read_text().splitlines(), 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
bad.append(i)
|
||||
check(not bad,
|
||||
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
|
||||
"is not machine-readable, so no proof can be derived from it")
|
||||
for r in rows:
|
||||
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
|
||||
"run_id"):
|
||||
check(key in r,
|
||||
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
|
||||
"provenance cannot be audited")
|
||||
if r.get("proof") == "proven":
|
||||
check(bool(r.get("sentinel")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
|
||||
"the sentinel is the only thing separating a real pass from a self-skip")
|
||||
check(r.get("test") in smap,
|
||||
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
|
||||
"test in the suite any more — a proof of something that no longer exists")
|
||||
check("-dirty" not in str(r.get("git_sha", "")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
|
||||
f"({r.get('git_sha')}) — what ran is not what is committed")
|
||||
else:
|
||||
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
|
||||
"were skipped; every other assertion ran, and the forgery probes above use "
|
||||
"synthetic ledgers, not this file")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 30,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
|
||||
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
|
||||
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
|
||||
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
|
||||
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
|
||||
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
|
||||
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
|
||||
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
|
||||
"(see the header's gap list).")
|
||||
PY
|
||||
@@ -0,0 +1,94 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
||||
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
||||
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
||||
# the nodes themselves — egress the local/private WAN by default and **never** the
|
||||
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
||||
# kill-switch blackholes **only** that class."
|
||||
#
|
||||
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
||||
# ------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
||||
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
||||
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
||||
# that select among them, which nothing else asserts:
|
||||
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
||||
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
||||
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
||||
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
||||
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
||||
# absence of a matching rule IS the default path;
|
||||
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
||||
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
||||
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
||||
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
||||
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
||||
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
||||
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
||||
#
|
||||
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
||||
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
||||
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
||||
# no third rule exists or that the vocabulary is closed.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
||||
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
||||
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
||||
# against its own values.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
||||
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
||||
# parsed and cross-checked,
|
||||
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
||||
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
||||
# named egress interface; all RFC1918 space and every address any environment assigns
|
||||
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
||||
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
||||
# cannot take the main table; and no nftables DROP is unscoped.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
||||
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
||||
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
||||
# that generate them; a hand-added rule on the box is invisible here.
|
||||
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
||||
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
||||
# * IPv6 egress.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-paths-selected-by-rules
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
||||
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
||||
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
||||
the vpn class, local destinations stay on the local path in every environment, the
|
||||
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
||||
path.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,513 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
|
||||
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
|
||||
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
|
||||
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
|
||||
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
|
||||
# uplinks) is **out of scope** unless a future ADR reopens it."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against — docs/03
|
||||
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
|
||||
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
|
||||
# router -> internet |
|
||||
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
|
||||
# egress gateway (ADR-0013) |
|
||||
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
|
||||
# to policy-route across; the `internet` path is simply the default route out."
|
||||
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
|
||||
# themselves — egress the local/private WAN by default and **never** the VPN. Only
|
||||
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
|
||||
# blackholes **only** that class."
|
||||
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
|
||||
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
|
||||
# only what Cilium points at `tun0` uses the VPN."
|
||||
#
|
||||
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
|
||||
# ---------------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
|
||||
# and the class can be flipped via GitOps. Those are proven by
|
||||
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
|
||||
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
|
||||
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
|
||||
#
|
||||
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
|
||||
# distinctive content — none of which any existing test asserts — is:
|
||||
#
|
||||
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
|
||||
# one closed set of two values agreed by the schema, the admission policy and the
|
||||
# renderer. A third value appearing in one of them but not the others would be a path
|
||||
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
|
||||
# class would contradict OQ-1's resolution.
|
||||
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
|
||||
# path is entered only via rules that select the vpn class — a Cilium
|
||||
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
|
||||
# to the tunnel source address. Nothing selects the `internet` class, because
|
||||
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
|
||||
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
|
||||
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
|
||||
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
|
||||
# pod's cluster/LAN traffic still takes the local path at the same time as its
|
||||
# internet traffic takes the tunnel.
|
||||
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
|
||||
# lives in a DEDICATED routing table reached only through the scoped rule, the
|
||||
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
|
||||
# from touching the main table. Otherwise there would be one path (the VPN), not two.
|
||||
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
|
||||
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
|
||||
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
|
||||
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
|
||||
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
|
||||
# asserts the OTHER direction, which that test does not.)
|
||||
#
|
||||
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
|
||||
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
|
||||
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
|
||||
# table") are statements about ALL rules, including the ones that would have to be absent.
|
||||
# A running cluster can show you which path two pods took; it cannot show you that no third
|
||||
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
|
||||
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label).
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
|
||||
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
|
||||
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
|
||||
# LAN is checked against its own addresses, and the check keeps working when they change.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
|
||||
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
|
||||
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
|
||||
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
|
||||
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
|
||||
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
|
||||
# invisible here (and to every other test); drift on the box is an Ansible check-mode
|
||||
# exercise for the operator.
|
||||
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
|
||||
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
|
||||
# because nobody tried" — and inventing one would be a hollow assertion.
|
||||
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
|
||||
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
|
||||
# this check would NOT catch its absence.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
LBL = "homelab.keithjackson.dev"
|
||||
CLASS_LABEL = f"{LBL}/egress-class"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def docs(path: Path):
|
||||
"""Every YAML document in a file, skipping empties."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "_artifacts" not in p.parts)
|
||||
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
|
||||
RENDER = ROOT / "catalog" / "render.py"
|
||||
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
|
||||
"read at all (R-NET-04)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
|
||||
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
|
||||
import json # noqa: E402 — only needed for the schema
|
||||
|
||||
schema = json.loads(SCHEMA.read_text())
|
||||
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
|
||||
enum = egress_prop.get("enum")
|
||||
check(isinstance(enum, list) and len(enum) == 2,
|
||||
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
|
||||
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
|
||||
"nothing routes; one value is not 'multiple egress paths'")
|
||||
PATHS = set(enum or [])
|
||||
check(PATHS == {"internet", "vpn"},
|
||||
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
|
||||
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
|
||||
check(egress_prop.get("default") == "internet",
|
||||
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
|
||||
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
|
||||
"the default route out'. A service that says nothing must get the WAN")
|
||||
|
||||
# The admission policy must constrain the pod label to the SAME closed set, or a workload
|
||||
# could carry a class the schema never allowed (and that no rule routes).
|
||||
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
|
||||
check(len(pol_files) >= 1,
|
||||
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
|
||||
"carrying an egress class outside the declared path set")
|
||||
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
|
||||
for cls in sorted(PATHS):
|
||||
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
|
||||
f"the egress admission policy never names the {cls!r} path — a class the policy "
|
||||
"does not know cannot be validated against the path set")
|
||||
# No tracked object may carry a class outside the closed set — with ONE self-policing
|
||||
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
|
||||
# admission test can assert the policy REJECTS it; that pod is exempt only while a
|
||||
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
|
||||
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
|
||||
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
|
||||
tests_root = fixture.parent
|
||||
while tests_root != ROOT and tests_root.name != "tests":
|
||||
tests_root = tests_root.parent
|
||||
if tests_root == ROOT:
|
||||
return False
|
||||
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
|
||||
for d in docs(tf):
|
||||
for case in (d.get("results") or []):
|
||||
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
stray: list[str] = []
|
||||
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
|
||||
for f in TRACKED_YAML:
|
||||
for d in docs(f):
|
||||
labels = ((d.get("metadata") or {}).get("labels") or {})
|
||||
cls = labels.get(CLASS_LABEL)
|
||||
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
|
||||
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
|
||||
if value is None or value in PATHS or value == "CLASS":
|
||||
continue
|
||||
name = ((owner.get("metadata") or {}).get("name") or "?")
|
||||
if rejected_by_a_policy_test(name, f):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
|
||||
# Values written as raw text (templates, docs-as-code) are caught too.
|
||||
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
|
||||
if value in PATHS or value == "CLASS":
|
||||
continue
|
||||
if any(value in s for s in stray):
|
||||
continue
|
||||
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
|
||||
# carries this value is a policy-test fixture proven to be rejected.
|
||||
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
|
||||
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
|
||||
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
|
||||
check(not stray,
|
||||
f"tracked YAML assigns egress-class value(s) outside the declared path set "
|
||||
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
|
||||
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
|
||||
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
|
||||
|
||||
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
|
||||
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
|
||||
# between the declaration and the path.
|
||||
rtext = RENDER.read_text()
|
||||
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
|
||||
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
|
||||
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
|
||||
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
|
||||
"two independent facts that could disagree")
|
||||
|
||||
|
||||
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
|
||||
policies: list[tuple[Path, dict]] = []
|
||||
for p in TRACKED_YAML:
|
||||
for d in docs(p):
|
||||
if d.get("kind") == "CiliumEgressGatewayPolicy":
|
||||
policies.append((p, d))
|
||||
check(len(policies) == 1,
|
||||
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
|
||||
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
|
||||
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
|
||||
"something the repository states")
|
||||
|
||||
for path, pol in policies:
|
||||
rel = str(path.relative_to(ROOT))
|
||||
spec = pol.get("spec") or {}
|
||||
selectors = spec.get("selectors") or []
|
||||
check(len(selectors) == 1,
|
||||
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
|
||||
"selector widens the set of pods steered off the default path, and R-NET-04's "
|
||||
"selection must be exactly 'the vpn class'")
|
||||
for sel in selectors:
|
||||
pod_sel = sel.get("podSelector")
|
||||
check(isinstance(pod_sel, dict) and pod_sel,
|
||||
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
|
||||
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
|
||||
"default path at all")
|
||||
ml = (pod_sel or {}).get("matchLabels") or {}
|
||||
me = (pod_sel or {}).get("matchExpressions") or []
|
||||
check(not me,
|
||||
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
|
||||
"single label equality; expressions (In/NotIn/Exists) can select pods that "
|
||||
"never declared a class, so the rule set would no longer be disjoint from the "
|
||||
"default path")
|
||||
check(set(ml) == {CLASS_LABEL},
|
||||
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
|
||||
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
|
||||
"than the declared class (and a namespace/app key would silently exempt pods)")
|
||||
check(ml.get(CLASS_LABEL) == "vpn",
|
||||
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
|
||||
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
|
||||
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
|
||||
"route the default path through the tunnel, i.e. leave one path, not two")
|
||||
|
||||
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
|
||||
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
|
||||
gw = spec.get("egressGateway") or {}
|
||||
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
|
||||
check(bool(node_sel),
|
||||
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
|
||||
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
|
||||
"nodes + Egress Gateway HA)")
|
||||
check("nodeName" not in gw,
|
||||
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
|
||||
"label so the path survives losing that node")
|
||||
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
|
||||
f"{rel}: the egressGateway names no egress interface — without it the steered class "
|
||||
"is not actually sent down a different path from the default")
|
||||
|
||||
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
|
||||
dests = spec.get("destinationCIDRs") or []
|
||||
excl = spec.get("excludedCIDRs") or []
|
||||
check([str(c) for c in dests] == ["0.0.0.0/0"],
|
||||
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
|
||||
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
|
||||
"minus the exclusions below — narrowing it here would silently leave public "
|
||||
"destinations on the WAN for a vpn-class service")
|
||||
try:
|
||||
excluded = [ipaddress.ip_network(str(c)) for c in excl]
|
||||
except ValueError as e:
|
||||
excluded = []
|
||||
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
|
||||
check(len(excluded) >= 3,
|
||||
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
|
||||
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
|
||||
|
||||
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
|
||||
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
|
||||
|
||||
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
|
||||
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
|
||||
net = ipaddress.ip_network(rfc)
|
||||
check(covered(net),
|
||||
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
|
||||
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
|
||||
"coexist for the same pod and east-west traffic would break")
|
||||
|
||||
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
|
||||
# is what makes 'local stays local' true per environment rather than by coincidence.
|
||||
def overlay_addresses() -> list[tuple[str, str]]:
|
||||
found: list[tuple[str, str]] = []
|
||||
base = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
data = {}
|
||||
if base.is_file():
|
||||
for d in docs(base):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
|
||||
env = {**data}
|
||||
for m in re.finditer(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
|
||||
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
|
||||
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
label = f.parent.name
|
||||
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
|
||||
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
|
||||
v = env.get(key, "")
|
||||
if v and not v.startswith("TBD"):
|
||||
found.append((f"{label}/{key}", v))
|
||||
return found
|
||||
|
||||
addrs = overlay_addresses()
|
||||
check(len(addrs) >= 5,
|
||||
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
|
||||
"the 'local traffic stays local' assertion would be checking almost nothing")
|
||||
for label, value in addrs:
|
||||
try:
|
||||
net = ipaddress.ip_network(value, strict=False)
|
||||
except ValueError:
|
||||
continue
|
||||
check(covered(net),
|
||||
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
|
||||
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
|
||||
"would try to reach its own cluster/LAN through the VPN")
|
||||
|
||||
|
||||
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
|
||||
TPL = VPN_ROLE / "templates"
|
||||
up = (TPL / "vpn-egress-up.sh.j2")
|
||||
down = (TPL / "vpn-egress-down.sh.j2")
|
||||
conf = (TPL / "vpn-egress.conf.j2")
|
||||
nft = (TPL / "vpn-egress-killswitch.nft.j2")
|
||||
for f in (up, down, conf, nft):
|
||||
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
|
||||
"rules are not authored")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
|
||||
|
||||
|
||||
def render(text: str) -> str:
|
||||
"""Substitute `{{ var }}` with the role's own default value, for every default.
|
||||
|
||||
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
|
||||
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
|
||||
exactly what an earlier draft of this check did. Rendering first means the assertions
|
||||
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
|
||||
substituted; anything else is left as-is (and the assertions are written so an
|
||||
unsubstituted expression fails rather than passes).
|
||||
"""
|
||||
out = text
|
||||
for k, v in defaults.items():
|
||||
if isinstance(v, (str, int)):
|
||||
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
|
||||
return out
|
||||
|
||||
|
||||
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
|
||||
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
|
||||
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
|
||||
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
|
||||
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
|
||||
"putting the tunnel's default route in any of them makes the VPN the node's default "
|
||||
"route instead of a selectable second path")
|
||||
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
|
||||
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
|
||||
"needs a rule, at a defined precedence")
|
||||
|
||||
# Every `ip route ... default` the up-script installs must name the dedicated table.
|
||||
route_lines = [l.strip() for l in up_t.splitlines()
|
||||
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
|
||||
check(len(route_lines) >= 2,
|
||||
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
|
||||
"default route via the tunnel AND the fail-closed blackhole behind it")
|
||||
for l in route_lines:
|
||||
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
|
||||
"landing in the main table changes the path every other pod and the node itself "
|
||||
"uses — the VPN would stop being opt-in")
|
||||
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
|
||||
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
|
||||
"be blackholed")
|
||||
|
||||
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
|
||||
# everything into the tunnel table, which would leave one path instead of two.
|
||||
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
|
||||
check(len(rule_lines) >= 1,
|
||||
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
|
||||
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
|
||||
for l in rule_lines:
|
||||
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
|
||||
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
|
||||
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
|
||||
"VPN is no longer opt-in")
|
||||
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"an ip rule looks up a table other than the tunnel's: {l!r}")
|
||||
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
|
||||
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
|
||||
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
|
||||
"distinguishes the steered class from every other flow on the gateway node")
|
||||
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
|
||||
"the up-script does not refuse to run when the tunnel address is unknown — installing "
|
||||
"a partial rule set is how a path ends up half-selected")
|
||||
|
||||
# The provider must not be able to make the tunnel the default route either.
|
||||
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
|
||||
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
|
||||
"routes itself, in the MAIN table, and the VPN would become the node's default route "
|
||||
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
|
||||
"routing)')")
|
||||
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
|
||||
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
|
||||
"provider could redirect the node's whole default route into the tunnel, collapsing "
|
||||
"the two paths into one")
|
||||
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
|
||||
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
|
||||
"default route")
|
||||
|
||||
|
||||
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
|
||||
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
|
||||
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
|
||||
check(bool(src_set),
|
||||
"the role declares no nftables source set — the kill-switch would have no way to name "
|
||||
"'the steered class' and could only match broadly")
|
||||
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
|
||||
check(len(policies_lines) >= 1,
|
||||
"the kill-switch nft template declares no base chain — nothing would be enforced")
|
||||
for l in policies_lines:
|
||||
check(re.search(r"policy\s+accept\s*;", l) is not None,
|
||||
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
|
||||
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
|
||||
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
|
||||
drop_lines = [l.strip() for l in nft_t.splitlines()
|
||||
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
|
||||
check(len(drop_lines) >= 1,
|
||||
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
|
||||
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
|
||||
for l in drop_lines:
|
||||
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
|
||||
check(scoped,
|
||||
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
|
||||
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
|
||||
"the default `internet` path, which must never be blackholed")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 35,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
|
||||
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
|
||||
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
|
||||
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
|
||||
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
|
||||
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
|
||||
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
|
||||
PY
|
||||
@@ -0,0 +1,83 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
||||
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
||||
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
||||
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
||||
#
|
||||
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
||||
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
||||
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
||||
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
||||
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
||||
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
||||
# like. The observed half is asserted separately at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
||||
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
||||
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
||||
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
||||
# environment's Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
||||
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
||||
# ONE --cluster-init founder and every other server joining it with --server;
|
||||
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
||||
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
||||
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
||||
# in the apiserver serving cert's SANs together with every server's own address, and
|
||||
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
||||
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
||||
# roles and plays retargeted by variables only.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
||||
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
||||
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
||||
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
||||
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
||||
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
||||
# infra/PROD-PROVISIONING.md).
|
||||
# * kube-vip is NOT imported by playbooks/site.yml, so the floating endpoint is a separate
|
||||
# operator step. Observed and reported, NOT asserted — R-PLAT-01 does not fix the
|
||||
# bring-up order (R-TEST-06).
|
||||
# * `K3S_API_HOST` (Cilium's k8sServiceHost) is the PRIMARY NODE's address in both
|
||||
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
||||
# A real single-node dependency, ESCALATED to the PM rather than asserted.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: ha-control-plane-as-code
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
||||
environment's Flux overlay are parsed and cross-checked, then every environment is
|
||||
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
||||
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
||||
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
||||
one code path for both environments.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,571 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — ha-control-plane-as-code
|
||||
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
|
||||
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
|
||||
# same multi-node topology." — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against:
|
||||
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
|
||||
# docs/02-environments-and-provisioning.md, table:
|
||||
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
|
||||
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
|
||||
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
||||
# docs/02 §"k3s topology decisions":
|
||||
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
|
||||
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
|
||||
# nodes we do not isolate control-plane nodes."
|
||||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs"
|
||||
#
|
||||
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
|
||||
# ---------------------------------------------------------------------
|
||||
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
|
||||
#
|
||||
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
|
||||
# plane in BOTH environments, from ONE code path": provable with no cluster, and
|
||||
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
|
||||
# brought up from this repo will also be HA, whether a second `--cluster-init`
|
||||
# founder could exist, or whether an external datastore is configured anywhere. That
|
||||
# is this check.
|
||||
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
|
||||
# etcd members, with the API answering through the floating VIP": that is cluster
|
||||
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
|
||||
#
|
||||
# Neither half is sufficient alone, and this file claims only the first.
|
||||
#
|
||||
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
|
||||
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
|
||||
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
|
||||
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
|
||||
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
|
||||
# its group name. Add a fourth prod node, rename a group, or add a third environment and
|
||||
# the assertions follow; nothing here restates an address, a hostname or a node count.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
|
||||
# Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
|
||||
# installed as an `agent` (an agent is not a control-plane member);
|
||||
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
|
||||
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
|
||||
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
|
||||
# every other server joins it with `--server` (so a second founder — the
|
||||
# split-brain shape — cannot be authored);
|
||||
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
|
||||
# `--disable-etcd` anywhere in the provisioning code, so every server is
|
||||
# an etcd member (docs/02's "no external datastore", "on all 3");
|
||||
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
|
||||
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
|
||||
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
|
||||
# baked into the apiserver serving cert's SAN list ALONG WITH every
|
||||
# server's own address (so TLS validates whichever node answers);
|
||||
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
|
||||
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
|
||||
# environment's Cilium LoadBalancer pool — the two allocators can never be
|
||||
# handed the same address;
|
||||
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
|
||||
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
|
||||
# runs"), so "exercise the same multi-node topology" is a property of the
|
||||
# code rather than a coincidence of two hand-written inventories.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
|
||||
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
|
||||
# answers through it. All of that is asserted at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
|
||||
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
|
||||
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
|
||||
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
|
||||
# data-availability claim and does NOT assert control-plane survival. A dedicated
|
||||
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
|
||||
# is NOT claimed here or at L2.
|
||||
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
|
||||
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
|
||||
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
|
||||
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
|
||||
# - THAT kube-vip IS PART OF THE ONE-COMMAND BRING-UP. playbooks/site.yml chains
|
||||
# node-baseline -> k3s -> cilium -> verify -> flux-bootstrap and does NOT import
|
||||
# kube-vip.yml, so the floating endpoint is a separate operator step. That is recorded
|
||||
# here as an observation for the PM, NOT asserted: R-PLAT-01 requires an HA control
|
||||
# plane, and does not name kube-vip or fix the bring-up order (R-TEST-06 — do not
|
||||
# invent a requirement's meaning).
|
||||
# - CILIUM'S OWN API ENDPOINT. `K3S_API_HOST` (clusters/*/kustomization.yaml), which
|
||||
# becomes Cilium's `k8sServiceHost` for kube-proxy replacement, is the PRIMARY NODE's
|
||||
# address in both environments — not the floating VIP — even though the prod overlay's
|
||||
# own comment says it "is the prod CONTROL-PLANE VIP (kube-vip), NOT ... the single node
|
||||
# address .10". That is a real single-node dependency in the CNI's control-plane path
|
||||
# and it is ESCALATED to the PM rather than asserted: making this check fail would turn
|
||||
# the merge gate red over a design question this workstream may not decide (R-TEST-06).
|
||||
# - IPv6 / dual-stack, and any claim about the edge network.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
GV = ANS / "inventory" / "group_vars"
|
||||
K3S_ROLE = ANS / "roles" / "k3s_server"
|
||||
VIP_ROLE = ANS / "roles" / "kube_vip"
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p, why in ((INV, "the inventory that defines the node set"),
|
||||
(K3S_ROLE, "the role that installs the k3s control plane"),
|
||||
(VIP_ROLE, "the role that floats the control-plane VIP"),
|
||||
(CLUSTERS, "the Flux overlays that identify each environment")):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
|
||||
"control plane is authored at all (R-PLAT-01)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
|
||||
VIP_TEXT = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
|
||||
|
||||
|
||||
# ===================================================================== inventory parse
|
||||
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
|
||||
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
|
||||
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
|
||||
ini_hosts: dict[str, list[str]] = {}
|
||||
ini_children: dict[str, list[str]] = {}
|
||||
host_vars: dict[str, dict[str, str]] = {}
|
||||
cur = None
|
||||
for line in INV.read_text().splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
host = s.split()[0]
|
||||
if cur.endswith(":children"):
|
||||
ini_children.setdefault(cur.split(":")[0], []).append(host)
|
||||
else:
|
||||
ini_hosts.setdefault(cur, []).append(host)
|
||||
hv = host_vars.setdefault(host, {})
|
||||
for kv in s.split()[1:]:
|
||||
if "=" in kv:
|
||||
k, v = kv.split("=", 1)
|
||||
hv[k] = v
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> set[str]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return set()
|
||||
seen.add(group)
|
||||
out = set(ini_hosts.get(group, []))
|
||||
for child in ini_children.get(group, []):
|
||||
out |= hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
def load_yaml(p: Path) -> dict:
|
||||
try:
|
||||
d = yaml.safe_load(p.read_text()) or {}
|
||||
except yaml.YAMLError:
|
||||
return {}
|
||||
return d if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
ALL_VARS = load_yaml(GV / "all.yml")
|
||||
|
||||
|
||||
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
|
||||
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
|
||||
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
|
||||
# another environment. Nothing below knows the words "dev" or "prod" as group names.
|
||||
class Env:
|
||||
def __init__(self, label, vars_, path):
|
||||
self.label = label
|
||||
self.vars = vars_
|
||||
self.path = path
|
||||
|
||||
def var(self, name, default=None):
|
||||
# group-scoped value wins, else the cluster-wide default from all.yml.
|
||||
if name in self.vars:
|
||||
return self.vars[name]
|
||||
return ALL_VARS.get(name, default)
|
||||
|
||||
|
||||
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
|
||||
for vf in sorted(GV.glob("*/vars.yml")):
|
||||
v = load_yaml(vf)
|
||||
if "k3s_primary_group" in v or "k3s_servers_group" in v:
|
||||
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
|
||||
|
||||
check(len(envs) >= 2,
|
||||
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
|
||||
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
|
||||
"dev/test to exercise the same multi-node topology, so two must be authored")
|
||||
|
||||
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
|
||||
# roles/flux_bootstrap's own default supplies the value for an environment that does not
|
||||
# override it, so this is read from code, never guessed from a group name.
|
||||
fb_default = None
|
||||
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
|
||||
if fb_defaults_file.is_file():
|
||||
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
|
||||
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
|
||||
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
|
||||
"an inventory environment to its Flux overlay is not expressed in code, so this check "
|
||||
"cannot tell which environment is prod")
|
||||
|
||||
prod_envs = []
|
||||
for e in envs:
|
||||
overlay = e.var("flux_bootstrap_env_path", fb_default)
|
||||
e.overlay = overlay if isinstance(overlay, str) else ""
|
||||
if e.overlay.rstrip("/").endswith("/prod"):
|
||||
prod_envs.append(e)
|
||||
check(len(prod_envs) == 1,
|
||||
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
|
||||
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
|
||||
"R-PLAT-01 has no unambiguous referent")
|
||||
|
||||
|
||||
# ============================================== overlay knob values (per environment)
|
||||
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
|
||||
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
|
||||
# - op: replace
|
||||
# path: /data/<KEY>
|
||||
# value: "<v>"
|
||||
# (verified shape across both overlays). A key not patched keeps the base value.
|
||||
BASE_VARS: dict[str, str] = {}
|
||||
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
if base_cm.is_file():
|
||||
d = load_yaml(base_cm)
|
||||
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
check(len(BASE_VARS) > 5,
|
||||
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
|
||||
"cannot be read, so the VIP/pool separation cannot be checked")
|
||||
|
||||
_PATCH_RE = re.compile(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
|
||||
re.M)
|
||||
|
||||
|
||||
def overlay_vars(overlay: str) -> dict[str, str]:
|
||||
out = dict(BASE_VARS)
|
||||
f = ROOT / overlay / "kustomization.yaml"
|
||||
if not f.is_file():
|
||||
return out
|
||||
for m in _PATCH_RE.finditer(f.read_text()):
|
||||
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
return out
|
||||
|
||||
|
||||
# ============================================================ PER-ENVIRONMENT TOPOLOGY
|
||||
def as_ip(s):
|
||||
try:
|
||||
return ipaddress.ip_address(str(s).strip())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
for e in envs:
|
||||
L = e.label
|
||||
pg = e.var("k3s_primary_group")
|
||||
sg = e.var("k3s_servers_group")
|
||||
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
|
||||
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
|
||||
"resolve 'my environment's primary/servers' and would only ever build dev")
|
||||
if not (isinstance(pg, str) and isinstance(sg, str)):
|
||||
continue
|
||||
primaries = hosts_in(pg)
|
||||
servers = hosts_in(sg)
|
||||
|
||||
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
|
||||
check(len(primaries) == 1,
|
||||
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
|
||||
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
|
||||
"cluster at all")
|
||||
check(primaries and primaries <= servers,
|
||||
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
|
||||
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
|
||||
"it is not part of the HA control plane it founds")
|
||||
n = len(servers)
|
||||
check(n >= 3,
|
||||
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
|
||||
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
|
||||
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
|
||||
check(n % 2 == 1,
|
||||
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
|
||||
f"member count adds a member without adding fault tolerance and introduces a tie: "
|
||||
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
|
||||
if n:
|
||||
tolerance = n - (n // 2 + 1)
|
||||
check(tolerance >= 1,
|
||||
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
|
||||
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
|
||||
if e in prod_envs:
|
||||
check(n == 3,
|
||||
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
|
||||
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
|
||||
else:
|
||||
check(n >= 2,
|
||||
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
|
||||
"nodes than prod but requires dev/test to 'exercise the same multi-node "
|
||||
"topology' — one node exercises no topology")
|
||||
|
||||
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
|
||||
vip_raw = e.var("kube_vip_address")
|
||||
vip = as_ip(vip_raw)
|
||||
check(vip is not None,
|
||||
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
|
||||
"no floating endpoint, so losing the founder takes the API endpoint with it")
|
||||
node_ips = {}
|
||||
for h in sorted(servers):
|
||||
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
||||
check(ip is not None,
|
||||
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
|
||||
"target and the serving-cert SAN list all derive from it")
|
||||
if ip is not None:
|
||||
node_ips[h] = ip
|
||||
if vip is not None:
|
||||
check(vip not in set(node_ips.values()),
|
||||
f"{L}: kube_vip_address {vip} is ALSO a node address "
|
||||
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
|
||||
"node's own address cannot float off that node")
|
||||
ov = overlay_vars(getattr(e, "overlay", ""))
|
||||
gw = as_ip(ov.get("GATEWAY_VIP", ""))
|
||||
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
|
||||
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
|
||||
check(gw is not None and lo is not None and hi is not None,
|
||||
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
|
||||
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
|
||||
"the Service-LB allocator cannot be checked")
|
||||
if gw is not None:
|
||||
check(vip != gw,
|
||||
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
|
||||
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
|
||||
if lo is not None and hi is not None:
|
||||
check(not (lo <= vip <= hi),
|
||||
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
|
||||
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
|
||||
"Service, and the two allocators would fight over ARP for it")
|
||||
check(lo <= hi,
|
||||
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
|
||||
"used to prove the VIP is outside it")
|
||||
|
||||
# ==================================================================== DISTRO: k3s only
|
||||
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
|
||||
is not None,
|
||||
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
|
||||
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
|
||||
check("get.k3s.io" in K3S_TEXT,
|
||||
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
|
||||
"distro (ADR-0001) and no other distro may stand in for it")
|
||||
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
|
||||
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
|
||||
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
|
||||
for bad, why in (
|
||||
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
|
||||
"nodes (no external datastore)')"),
|
||||
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
|
||||
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
|
||||
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
|
||||
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
|
||||
):
|
||||
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
|
||||
and re.search(bad, p.read_text(errors="replace"))]
|
||||
check(not hits,
|
||||
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
|
||||
f"mode, and {bad!r} contradicts it")
|
||||
|
||||
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
|
||||
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
|
||||
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
|
||||
"founded, so there is no HA datastore to join")
|
||||
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
|
||||
K3S_TEXT)
|
||||
check(init_when is not None,
|
||||
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
|
||||
"ungated founder flag would re-found the cluster on every server (split brain)")
|
||||
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
|
||||
check(join is not None,
|
||||
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
|
||||
"— without a join target the servers are separate single-node clusters")
|
||||
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
||||
K3S_TEXT, re.S)
|
||||
check(join_when is not None,
|
||||
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
|
||||
"would also try to join itself")
|
||||
k3s_play = PLAYBOOKS / "k3s.yml"
|
||||
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
|
||||
if k3s_play.is_file():
|
||||
play_txt = k3s_play.read_text()
|
||||
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
|
||||
check(sorted(modes) == ["primary", "secondary"],
|
||||
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
|
||||
"one secondary play must exist, or the founder/joiner split is not authored")
|
||||
check(re.search(r"serial:\s*1", play_txt) is not None,
|
||||
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
|
||||
"converging two of three servers at once drops embedded etcd below quorum, which "
|
||||
"is exactly the outage an HA control plane exists to avoid")
|
||||
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
|
||||
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
|
||||
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
|
||||
"group name means this play can only ever build ONE environment, so prod cannot "
|
||||
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
|
||||
|
||||
# =========================================== SERVING-CERT SANs: VIP + every server
|
||||
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
|
||||
check("kube_vip_address" in sans,
|
||||
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
|
||||
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
|
||||
"API through it fails TLS validation and the HA endpoint is unusable")
|
||||
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
|
||||
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
|
||||
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
|
||||
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
|
||||
check(re.search(r"tls-san:", K3S_TEXT) is not None,
|
||||
"the SAN list is passed only as an install flag and never persisted to "
|
||||
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
|
||||
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
|
||||
|
||||
# ============================================ FLOATING ENDPOINT: kube-vip on every server
|
||||
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
|
||||
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
|
||||
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
|
||||
# thing that ships.) Parser contract: the template writes each env var as an adjacent
|
||||
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
|
||||
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
|
||||
check(len(VIP_TEMPLATES) >= 1,
|
||||
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
|
||||
vip_manifest = "\n".join(
|
||||
"\n".join(l for l in p.read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#"))
|
||||
for p in VIP_TEMPLATES)
|
||||
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
|
||||
vip_manifest))
|
||||
check(len(vip_env) >= 6,
|
||||
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
|
||||
"parser lost its grip on the file, so the assertions below would assert nothing")
|
||||
check(vip_env.get("cp_enable") == "true",
|
||||
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
|
||||
"mode must be ON or it floats no API VIP at all")
|
||||
check(vip_env.get("svc_enable") == "false",
|
||||
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
|
||||
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
|
||||
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
|
||||
check(vip_env.get("vip_leaderelection") == "true",
|
||||
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
|
||||
"— without leader election the VIP has no defined owner to fail over FROM, which is "
|
||||
"the whole HA mechanism")
|
||||
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
|
||||
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
|
||||
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
|
||||
"prod (R-ENV-02) and would not match the serving-cert SAN list")
|
||||
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
|
||||
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
|
||||
"node, which is the opposite of a floating endpoint")
|
||||
vip_play = PLAYBOOKS / "kube-vip.yml"
|
||||
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
|
||||
if vip_play.is_file():
|
||||
vp = vip_play.read_text()
|
||||
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
|
||||
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
|
||||
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
|
||||
"group through the same environment-scoped variable the other plays use, so the "
|
||||
"manifest exists on EVERY server and the VIP survives losing any one of them")
|
||||
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
|
||||
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
|
||||
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
|
||||
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
|
||||
"manifest somewhere k3s never reads and come up with no VIP at all")
|
||||
|
||||
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
|
||||
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
|
||||
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
|
||||
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
|
||||
env_words = re.compile(r"prod|dev", re.I)
|
||||
forked = sorted(n for n in (role_names | play_names)
|
||||
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
|
||||
check(not forked,
|
||||
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
|
||||
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
|
||||
for e in envs:
|
||||
if e.path.name != "vars.yml":
|
||||
continue
|
||||
# Every key a non-default environment sets must be a variable something already reads,
|
||||
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
|
||||
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
|
||||
for k in sorted(e.vars):
|
||||
if k.startswith("ansible_"):
|
||||
continue
|
||||
check(consumers.count(k) > 1,
|
||||
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
|
||||
"environment override that nothing reads is not a scoped override, it is dead "
|
||||
"configuration (and would make prod silently inherit a dev default)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 40,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
summary = "; ".join(
|
||||
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
|
||||
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
|
||||
for e in envs)
|
||||
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
|
||||
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
|
||||
"embedded etcd on every server with no external datastore, a leader-elected "
|
||||
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
|
||||
"collides with no other allocator, from one code path for both environments. The LIVE "
|
||||
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
|
||||
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
|
||||
PY
|
||||
@@ -153,3 +153,20 @@ spec:
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "cleanup handled by chainsaw namespace teardown"'
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo DEFAULT_DENY_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true
|
||||
@@ -0,0 +1,199 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE]
|
||||
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path").
|
||||
# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is
|
||||
# simply the default route out".
|
||||
# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.**
|
||||
# Every pod — and the nodes themselves — egress the local/private WAN by default and
|
||||
# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the
|
||||
# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0`
|
||||
# uses the VPN."
|
||||
#
|
||||
# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT
|
||||
# --------------------------------------------------------------
|
||||
# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their
|
||||
# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing
|
||||
# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds
|
||||
# that pod, and asserts the two claims that make a rule-selected path set complete:
|
||||
#
|
||||
# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the
|
||||
# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is
|
||||
# the local one. If the steering rule ever widened (an empty selector, a
|
||||
# matchExpression, a second rule), this pod's IP would move to the tunnel and
|
||||
# this assertion is what would notice.
|
||||
# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one
|
||||
# pass: three pods, two distinct public IPs. "Multiple egress paths" is a
|
||||
# simultaneity claim, not a claim that one switch has two positions.
|
||||
# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918
|
||||
# destination over the LOCAL path while its public traffic is tunnelled. The
|
||||
# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod
|
||||
# uses BOTH paths at once.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a
|
||||
# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike
|
||||
# and survives the WAN or the VPN provider rotating addresses.
|
||||
#
|
||||
# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three
|
||||
# times, differing only in the class they declare (or do not),
|
||||
# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod
|
||||
# also resolves a name through the in-cluster resolver,
|
||||
# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out
|
||||
# is the WAN, and the VPN is strictly opt-in);
|
||||
# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path);
|
||||
# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not
|
||||
# swallowed by the tunnel), so the same pod is using both paths at once.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one
|
||||
# steering rule is authored, that it selects only the vpn class, that no `ip rule` is
|
||||
# unscoped and no nftables DROP unqualified is the L0 half, in
|
||||
# tests/l0-static/egress-paths-selected-by-rules (same requirement label).
|
||||
# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo
|
||||
# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need
|
||||
# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in
|
||||
# this suite does.
|
||||
# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn`
|
||||
# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically
|
||||
# by the L0 half, not here.
|
||||
# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That
|
||||
# is a host-level fact this test cannot see from a pod.
|
||||
# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster,
|
||||
# so a WAN outage or the echo being down fails this test. That is the same dependency
|
||||
# tests/l2-behavioral/egress-class already carries; the alternative — trusting the
|
||||
# cluster's own account of which path it used — would not be a behavioural test at all.
|
||||
#
|
||||
# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside
|
||||
# that namespace is created or changed — no label on a platform object, no policy edit.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-default-path
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public
|
||||
IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the
|
||||
default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two
|
||||
paths live at once — and the vpn-class pod still reaches in-cluster destinations over the
|
||||
local path, so one pod uses both paths simultaneously.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 3m
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: given-three-pods-differing-only-in-their-declared-egress-class
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/three-egress-pods.yaml
|
||||
# Bare pods (restartPolicy: Never) — Ready == Running here.
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-unlabelled }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-internet }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-vpn }
|
||||
status: { phase: Running }
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
get_ip() {
|
||||
# Echo the pod's observed public IP, or empty on failure.
|
||||
kubectl -n "$ns" exec "$1" -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Poll until all three answer AND the vpn class has diverged from the WAN:
|
||||
# Cilium may still be programming the egress datapath just after the pods go
|
||||
# Ready. Bounded by the loop, never a bare sleep-and-hope.
|
||||
none="" inet="" vpn=""
|
||||
for i in $(seq 1 12); do
|
||||
[ -n "$none" ] || none="$(get_ip egress-unlabelled)"
|
||||
[ -n "$inet" ] || inet="$(get_ip egress-internet)"
|
||||
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
||||
if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then
|
||||
[ "$vpn" != "$inet" ] && break
|
||||
# Steering may not have converged; re-read the vpn pod only.
|
||||
vpn=""
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn"
|
||||
|
||||
[ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; }
|
||||
[ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; }
|
||||
[ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; }
|
||||
|
||||
# TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN —
|
||||
# the same WAN the explicitly-internet pod uses.
|
||||
if [ "$none" != "$inet" ]; then
|
||||
echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2
|
||||
exit 1
|
||||
fi
|
||||
# COEXISTENCE: and the vpn path is a genuinely different path, live right now.
|
||||
if [ "$vpn" = "$inet" ]; then
|
||||
echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "EGRESS_DEFAULT_PATH_IS_WAN_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-vpn-class-pod-still-reaches-private-destinations-locally
|
||||
try:
|
||||
- script:
|
||||
# PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all
|
||||
# RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the
|
||||
# local datapath. The cluster resolver is an in-cluster (RFC1918) address reached
|
||||
# from the pod's own /etc/resolv.conf — no literal is written here — so a
|
||||
# successful lookup from the vpn-class pod proves the tunnel did not swallow its
|
||||
# local traffic, i.e. ONE pod is using BOTH paths at the same time.
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)"
|
||||
[ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; }
|
||||
echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)"
|
||||
|
||||
# A cluster-internal name, resolvable only by the in-cluster resolver: reaching
|
||||
# it proves the private-destination path works from a vpn-class pod.
|
||||
out=""
|
||||
for i in $(seq 1 6); do
|
||||
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)"
|
||||
case "$out" in *"Address"*) break ;; esac
|
||||
sleep 5
|
||||
done
|
||||
case "$out" in
|
||||
*Address*) : ;;
|
||||
*) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;;
|
||||
esac
|
||||
echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true
|
||||
@@ -0,0 +1,67 @@
|
||||
# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing
|
||||
# ONLY in how (or whether) they declare an egress class:
|
||||
#
|
||||
# egress-unlabelled NO egress-class label at all -> selected by NO routing rule
|
||||
# egress-internet egress-class: internet -> the declared default path
|
||||
# egress-vpn egress-class: vpn -> the only class any rule steers
|
||||
#
|
||||
# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress
|
||||
# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no
|
||||
# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must
|
||||
# come out the same WAN as the explicitly-`internet` pod. That is the totality half of
|
||||
# R-NET-04's "selected by routing rules" — absence of a rule is itself a path.
|
||||
#
|
||||
# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an
|
||||
# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in
|
||||
# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee
|
||||
# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe
|
||||
# the datapath's own default at all.
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-unlabelled
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-internet
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-vpn
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: vpn
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
@@ -313,3 +313,20 @@ spec:
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
name: conformance-gw-probe
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true
|
||||
@@ -0,0 +1,373 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane (R-PLAT-01) [RUNNABLE]
|
||||
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
|
||||
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
||||
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology.")
|
||||
# docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02"
|
||||
# docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA
|
||||
# embedded etcd (k3s server on all 3)"
|
||||
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
||||
# docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across
|
||||
# all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload
|
||||
# (3-node converged cluster)".
|
||||
#
|
||||
# WHY L2 — AND WHAT L0 ALREADY DID
|
||||
# --------------------------------
|
||||
# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED
|
||||
# half offline: the repo builds a 3-server embedded-etcd control plane in both environments,
|
||||
# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving
|
||||
# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory
|
||||
# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test:
|
||||
# every assertion below is read from the cluster the suite is pointed at.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written
|
||||
# here. The expected node count comes from the cluster's OWN declared identity
|
||||
# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same
|
||||
# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip
|
||||
# DaemonSet the cluster is actually running; the server addresses come from the Node
|
||||
# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars
|
||||
# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY
|
||||
# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the
|
||||
# environment from the cluster, never from a flag or a literal, which is precisely the
|
||||
# mechanism the tests/README.md convention prescribes.
|
||||
#
|
||||
# GIVEN the cluster this suite is pointed at,
|
||||
# WHEN its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected,
|
||||
# THEN DISTRO: every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s;
|
||||
# SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node
|
||||
# topology docs/02 fixes — no agent-only node, no control-plane-only node),
|
||||
# the count is ODD and quorum-capable (tolerates >= 1 loss), and it is
|
||||
# EXACTLY three when the cluster declares itself prod;
|
||||
# ETCD: every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps
|
||||
# `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so
|
||||
# the datastore is embedded and replicated across all of them;
|
||||
# VIP: kube-vip is Ready on EVERY control-plane node, is control-plane-only
|
||||
# (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and
|
||||
# the address it floats is neither a node address, nor the ingress Gateway
|
||||
# VIP, nor inside the Cilium LoadBalancer pool;
|
||||
# ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that
|
||||
# validates against the cluster CA for the VIP itself — i.e. the VIP really
|
||||
# fronts the API and the serving cert really carries it;
|
||||
# EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own
|
||||
# address, again with a CA-valid cert for that address. This is the assertion
|
||||
# that separates a real HA control plane from one server plus two joiners:
|
||||
# if only the founder ran an apiserver, the other addresses would refuse.
|
||||
#
|
||||
# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node
|
||||
# addresses from the RUNNER. That works from a host on the node LAN (verified against prod)
|
||||
# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a
|
||||
# forwarded port — i.e. it would have made the test a property of where it was launched
|
||||
# from. A Job on the cluster is on the node network in both environments, so the probe is
|
||||
# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than
|
||||
# as a fixture file because its address list is DISCOVERED from the cluster at run time —
|
||||
# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP
|
||||
# migrates, and that the API stays reachable when a server dies is an L3 fault-injection
|
||||
# claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss
|
||||
# drains a node for R-STATE-02's data-availability claim and asserts nothing about the
|
||||
# control plane). This is the largest remaining R-PLAT-01 gap.
|
||||
# - THE NEXT environment. That the repo would build an HA control plane again is the L0
|
||||
# half (tests/l0-static/ha-control-plane-as-code).
|
||||
# - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of
|
||||
# infra/PROD-PROVISIONING.md).
|
||||
# - Cilium's own `k8sServiceHost` (cluster-vars `K3S_API_HOST`) is the PRIMARY NODE's
|
||||
# address, not this VIP, in both environments. That single-node dependency in the CNI's
|
||||
# control-plane path is escalated to the PM, not asserted here (R-TEST-06).
|
||||
# - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes
|
||||
# it only on the node-local etcd metrics port, which is not reachable through the
|
||||
# Kubernetes API this test is allowed to use.
|
||||
#
|
||||
# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it
|
||||
# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test
|
||||
# creates nothing outside its own namespace and mutates no platform object.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: ha-control-plane
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver
|
||||
endpoints are probed, then every node is a k3s server that is both control-plane and a
|
||||
registered embedded-etcd member, the count is odd and quorum-capable (exactly three
|
||||
when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that
|
||||
collides with no allocator, and the apiserver answers /readyz through that VIP AND on
|
||||
every server's own address with a CA-valid cert.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 3m
|
||||
# -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE
|
||||
# CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes
|
||||
# from, so these are the same values the platform was rendered with. Test-level, so
|
||||
# they are in scope for every step (step outputs/bindings are step-scoped).
|
||||
bindings:
|
||||
- name: clusterEnv
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV)
|
||||
- name: gatewayVip
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
|
||||
- name: lbPoolStart
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START)
|
||||
- name: lbPoolStop
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP)
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane
|
||||
description: >-
|
||||
Read the Node objects: k3s on every node, every node both control-plane and a
|
||||
registered embedded-etcd member, an odd quorum-capable count, exactly three in prod.
|
||||
try:
|
||||
- script:
|
||||
env:
|
||||
- name: CLUSTER_ENV
|
||||
value: ($clusterEnv)
|
||||
content: |
|
||||
set -eu
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
total="$(kubectl get nodes -o name | grep -c . || true)"
|
||||
[ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all"
|
||||
|
||||
# --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) -----
|
||||
# A non-k3s kubelet here would mean the platform is not running on k3s.
|
||||
notk3s="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \
|
||||
| grep -v '+k3s' | grep -c . || true)"
|
||||
[ "$notk3s" = "0" ] \
|
||||
|| fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s"
|
||||
|
||||
# --- every node is BOTH control-plane AND etcd (docs/02 converged topology) -
|
||||
# Counted by label EXISTENCE, so it does not matter whether k3s writes "true"
|
||||
# or "". A control-plane node without the etcd role would be a server with an
|
||||
# external/absent datastore; an etcd node without control-plane would be an
|
||||
# isolated datastore member. Both are excluded by requiring all three counts
|
||||
# to be equal.
|
||||
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
|
||||
etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)"
|
||||
echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-<unset>}"
|
||||
[ "$cp" = "$total" ] \
|
||||
|| fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')"
|
||||
[ "$etcd" = "$total" ] \
|
||||
|| fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'"
|
||||
|
||||
# --- every node is a REGISTERED embedded-etcd member -----------------------
|
||||
# k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the
|
||||
# embedded-etcd cluster. The role label alone is a label; this annotation is
|
||||
# the datastore's own record of membership.
|
||||
members="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \
|
||||
| grep -c . || true)"
|
||||
[ "$members" = "$total" ] \
|
||||
|| fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server"
|
||||
|
||||
# --- Ready ---------------------------------------------------------------
|
||||
notready="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \
|
||||
| grep -v '=True$' | grep -c . || true)"
|
||||
[ "$notready" = "0" ] \
|
||||
|| fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test"
|
||||
|
||||
# --- quorum arithmetic ---------------------------------------------------
|
||||
# Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is
|
||||
# N - quorum; 'production-grade HA' must survive at least one loss, which
|
||||
# rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds
|
||||
# a tie, so it is also rejected.
|
||||
quorum=$(( total / 2 + 1 ))
|
||||
tolerance=$(( total - quorum ))
|
||||
echo "quorum=$quorum tolerance=$tolerance"
|
||||
[ "$tolerance" -ge 1 ] \
|
||||
|| fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server"
|
||||
[ $(( total % 2 )) -eq 1 ] \
|
||||
|| fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance"
|
||||
|
||||
# --- the count R-PLAT-01 itself fixes, per environment --------------------
|
||||
# The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a
|
||||
# flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology'.
|
||||
case "${CLUSTER_ENV:-}" in
|
||||
prod)
|
||||
[ "$total" = "3" ] \
|
||||
|| fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;;
|
||||
"")
|
||||
fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;;
|
||||
*)
|
||||
[ "$total" -ge 2 ] \
|
||||
|| fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;;
|
||||
esac
|
||||
echo "HA_TOPOLOGY_OK"
|
||||
check:
|
||||
(contains($stdout, 'HA_TOPOLOGY_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-a-control-plane-only-vip-is-floated-on-every-server
|
||||
description: >-
|
||||
Read the running kube-vip DaemonSet: Ready on every control-plane node,
|
||||
control-plane-only, and floating an address that collides with no other allocator.
|
||||
try:
|
||||
- script:
|
||||
env:
|
||||
- name: GATEWAY_VIP
|
||||
value: ($gatewayVip)
|
||||
- name: LB_POOL_START
|
||||
value: ($lbPoolStart)
|
||||
- name: LB_POOL_STOP
|
||||
value: ($lbPoolStop)
|
||||
content: |
|
||||
set -eu
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
# dotted quad -> integer, for range comparison without ipcalc/python.
|
||||
n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; }
|
||||
|
||||
# The DaemonSet is found by the label roles/kube_vip stamps on it, not by a
|
||||
# hardcoded name in a namespace this test guessed.
|
||||
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
|
||||
[ -n "$ds" ] \
|
||||
|| fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it"
|
||||
|
||||
desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')"
|
||||
ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')"
|
||||
cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
|
||||
vip="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
|
||||
cp_enable="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')"
|
||||
svc_enable="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')"
|
||||
echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable"
|
||||
|
||||
[ -n "$vip" ] \
|
||||
|| fail "the kube-vip DaemonSet announces no address — nothing is floated"
|
||||
[ "$cp_enable" = "true" ] \
|
||||
|| fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP"
|
||||
[ "$svc_enable" = "false" ] \
|
||||
|| fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)"
|
||||
[ "$desired" = "$cp" ] \
|
||||
|| fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane"
|
||||
[ "$ready" = "$desired" ] \
|
||||
|| fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over"
|
||||
|
||||
# --- the VIP collides with nothing --------------------------------------
|
||||
node_ips="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \
|
||||
| grep -c "^${vip}$" || true)"
|
||||
[ "$node_ips" = "0" ] \
|
||||
|| fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node"
|
||||
[ "$vip" != "${GATEWAY_VIP:-}" ] \
|
||||
|| fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it"
|
||||
if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then
|
||||
v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")"
|
||||
[ "$lo" -le "$hi" ] \
|
||||
|| fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it"
|
||||
if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then
|
||||
fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service"
|
||||
fi
|
||||
else
|
||||
fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)"
|
||||
fi
|
||||
echo "HA_VIP_CONTROL_PLANE_ONLY_OK"
|
||||
check:
|
||||
(contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-apiserver-answers-through-the-vip-and-on-every-server
|
||||
description: >-
|
||||
From inside the cluster, probe /readyz on the floated VIP and on every server's own
|
||||
address, validating TLS against the cluster CA for the address dialled.
|
||||
try:
|
||||
- script:
|
||||
# The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the
|
||||
# Job cannot be a static fixture — writing the addresses down is the
|
||||
# per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the
|
||||
# same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped
|
||||
# fixtures. The Job lives in chainsaw's per-test namespace and is torn down with
|
||||
# it; nothing outside that namespace is created or changed.
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
|
||||
vip="$(kubectl -n kube-system get "$ds" \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
|
||||
servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \
|
||||
-o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')"
|
||||
[ -n "$vip" ] || fail "no floated VIP to probe"
|
||||
[ -n "$servers" ] || fail "no control-plane node addresses to probe"
|
||||
echo "probing vip=$vip servers=$servers"
|
||||
|
||||
# The probe pod authenticates with its own ServiceAccount token and validates
|
||||
# TLS against the cluster CA, both from the projected SA volume. /readyz is
|
||||
# authorized for any authenticated identity via the built-in
|
||||
# system:public-info-viewer role — anonymous access is NOT relied on (verified:
|
||||
# an unauthenticated /readyz on this platform answers 401).
|
||||
kubectl -n "$ns" apply -f - <<EOF
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: ha-api-endpoint-probe
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
activeDeadlineSeconds: 150
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: probe
|
||||
image: curlimages/curl:8.11.1
|
||||
env:
|
||||
- name: VIP
|
||||
value: "${vip}"
|
||||
- name: SERVERS
|
||||
value: "${servers}"
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
SA=/var/run/secrets/kubernetes.io/serviceaccount
|
||||
probe() { # \$1 = address to dial
|
||||
curl -sS --max-time 15 \\
|
||||
--cacert "\$SA/ca.crt" \\
|
||||
-H "Authorization: Bearer \$(cat \$SA/token)" \\
|
||||
"https://\$1:6443/readyz"
|
||||
}
|
||||
rc=0
|
||||
out="\$(probe "\$VIP" 2>&1)" || rc=1
|
||||
echo "vip \$VIP -> [\$out]"
|
||||
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
|
||||
echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)"
|
||||
exit 1
|
||||
fi
|
||||
for s in \$SERVERS; do
|
||||
rc=0
|
||||
out="\$(probe "\$s" 2>&1)" || rc=1
|
||||
echo "server \$s -> [\$out]"
|
||||
if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
|
||||
echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "HA_API_ENDPOINT_OK"
|
||||
EOF
|
||||
|
||||
# Bounded wait; on failure surface the probe's own output before failing.
|
||||
if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then
|
||||
kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true
|
||||
fail "the in-cluster API-endpoint probe did not complete — see its log above"
|
||||
fi
|
||||
logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)"
|
||||
echo "$logs"
|
||||
case "$logs" in
|
||||
*HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;;
|
||||
*) fail "the probe completed without reporting success" ;;
|
||||
esac
|
||||
check:
|
||||
(contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true
|
||||
@@ -93,3 +93,20 @@ spec:
|
||||
kind: Job
|
||||
metadata: { name: tls-probe }
|
||||
status: { succeeded: 1 }
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo TLS_TRUSTED_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true
|
||||
Reference in new issue
Block a user