Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect

Browse Source 
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0
safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS.

R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control
plane" is half authored and half observable.
  l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from
  the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with
  no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init
  founder per environment; odd, quorum-capable server count with tolerance >= 1;
  tls-sans carry the VIP and every server IP; kube-vip leader-elected with
  cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside
  its environment's LB pool and not a node IP.
  l2-behavioral/ha-control-plane: control-plane AND etcd role on every node,
  every node a REGISTERED etcd member, exactly 3 when the cluster says
  CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every
  server's own address — which proves the VIP fronts the API and that every server
  runs an apiserver, not one founder plus two joiners.

  An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's
  LAN and fails against dev (nodes behind the pfSense NAT), making the test a
  property of where it was launched. Hence the in-cluster Job.

R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the
totality/disjointness of the selecting rules, not another per-service check.
  l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the
  same closed set in schema + Kyverno policy + renderer; EXACTLY ONE
  CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn;
  nothing selects `internet` (absence of a rule IS the path); RFC1918 containment
  computed with `ipaddress` plus every address each overlay assigns itself; the
  tunnel default route and blackhole confined to table 51820 behind an ip rule that
  must carry a from/fwmark/iif selector; and the nft chain `policy accept` with
  every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test
  does not assert.

R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable".
  l0-static/auditability-proofs-producible (78 assertions) drives the real
  machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer
  the spec's wording demands (fault injection and DR drill => L3), taken from
  status.py's own verdict; run.sh --requirements must resolve and pass the gate's
  --require assertion; each proof declares a sentinel record_run.py can attribute,
  computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted
  against the real tools — a green JUnit whose log holds only a SKIP token, and a
  ledger row claiming `proven` at the root commit — which must come back not-proven
  and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted,
  not hand-listed.

*** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's
default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel,
so record_run.py could never attribute a pass to them — the isolation and TLS audit
proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared,
and status.py requires all of a requirement's tests to be proven). Each now ends
with an additive terminal sentinel step reached only when every prior step passed.
Verified: all three carry it.

TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this
repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role
defaults, so flipping the template to true still passed — now parses the DaemonSet
template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja
variable while the template carries its NAME, matching nothing and passing
vacuously — templates are now rendered against role defaults first. 30 perturbations
across the three checks, all caught, plus 16 synthetic L2 perturbations via a
fake-kubectl shim.

R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's
widgets.yaml carries only datetime and search, kubernetes.yaml is empty,
automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress
only — no health source is even reachable. It has already drifted (crafty authors a
route with no tile). A test today would go red on both; it must ship WITH the
config fix.

VERIFIED: full CI gate PASS. Neither cluster mutated.

Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-06 19:35:18 -05:00
1 parent 5ec1460532
commit cb1916f092
12 files changed
+2488
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml
+94
View File
@@ -0,0 +1,94 @@
# =============================================================================
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
# routing rules ... realized as two paths — the local `internet` path and the
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
# routing: two paths, one physical WAN": "the `internet` path is simply the default
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
# the nodes themselves — egress the local/private WAN by default and **never** the
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
# kill-switch blackholes **only** that class."
#
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
# ------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
# that select among them, which nothing else asserts:
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
# absence of a matching rule IS the default path;
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
# vpn-class pod's cluster/LAN traffic still takes the local path;
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
#
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
# route in the main table). A cluster can show which path two pods took; it cannot show that
# no third rule exists or that the vocabulary is closed.
#
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
# against its own values.
#
# GIVEN the repository as git tracks it,
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
# parsed and cross-checked,
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
# selects exactly the vpn class by label; the gateway is selected by node label with a
# named egress interface; all RFC1918 space and every address any environment assigns
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
# cannot take the main table; and no nftables DROP is unscoped.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
# that generate them; a hand-added rule on the box is invisible here.
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
# * IPv6 egress.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: egress-paths-selected-by-rules
labels:
conformance.home.k3s/R-NET-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
the vpn class, local destinations stay on the local path in every environment, the
tunnel never becomes the default route, and nothing unscoped can blackhole the default
path.
spec:
steps:
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh
Executable
+513
View File
@@ -0,0 +1,513 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
# uplinks) is **out of scope** unless a future ADR reopens it."
# — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against — docs/03
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
# router -> internet |
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
# egress gateway (ADR-0013) |
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
# to policy-route across; the `internet` path is simply the default route out."
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
# themselves — egress the local/private WAN by default and **never** the VPN. Only
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
# blackholes **only** that class."
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
# only what Cilium points at `tun0` uses the VPN."
#
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
# ---------------------------------------------------------------------------------
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
# and the class can be flipped via GitOps. Those are proven by
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
#
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
# distinctive content — none of which any existing test asserts — is:
#
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
# one closed set of two values agreed by the schema, the admission policy and the
# renderer. A third value appearing in one of them but not the others would be a path
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
# class would contradict OQ-1's resolution.
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
# path is entered only via rules that select the vpn class — a Cilium
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
# to the tunnel source address. Nothing selects the `internet` class, because
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
# pod's cluster/LAN traffic still takes the local path at the same time as its
# internet traffic takes the tunnel.
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
# lives in a DEDICATED routing table reached only through the scoped rule, the
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
# from touching the main table. Otherwise there would be one path (the VPN), not two.
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
# asserts the OTHER direction, which that test does not.)
#
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
# table") are statements about ALL rules, including the ones that would have to be absent.
# A running cluster can show you which path two pods took; it cannot show you that no third
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
# tests/l2-behavioral/egress-default-path (same requirement label).
#
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
# LAN is checked against its own addresses, and the check keeps working when they change.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
# invisible here (and to every other test); drift on the box is an Ansible check-mode
# exercise for the operator.
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
# because nobody tried" — and inventing one would be a hollow assertion.
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
# this check would NOT catch its absence.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
LBL = "homelab.keithjackson.dev"
CLASS_LABEL = f"{LBL}/egress-class"
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def die():
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def docs(path: Path):
"""Every YAML document in a file, skipping empties."""
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
if ".git" not in p.parts and "_artifacts" not in p.parts)
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
RENDER = ROOT / "catalog" / "render.py"
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
CLUSTERS = ROOT / "clusters"
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
"read at all (R-NET-04)")
if FAILURES:
die()
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
import json # noqa: E402 — only needed for the schema
schema = json.loads(SCHEMA.read_text())
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
enum = egress_prop.get("enum")
check(isinstance(enum, list) and len(enum) == 2,
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
"nothing routes; one value is not 'multiple egress paths'")
PATHS = set(enum or [])
check(PATHS == {"internet", "vpn"},
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
check(egress_prop.get("default") == "internet",
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
"the default route out'. A service that says nothing must get the WAN")
# The admission policy must constrain the pod label to the SAME closed set, or a workload
# could carry a class the schema never allowed (and that no rule routes).
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
check(len(pol_files) >= 1,
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
"carrying an egress class outside the declared path set")
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
for cls in sorted(PATHS):
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
f"the egress admission policy never names the {cls!r} path — a class the policy "
"does not know cannot be validated against the path set")
# No tracked object may carry a class outside the closed set — with ONE self-policing
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
# admission test can assert the policy REJECTS it; that pod is exempt only while a
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
tests_root = fixture.parent
while tests_root != ROOT and tests_root.name != "tests":
tests_root = tests_root.parent
if tests_root == ROOT:
return False
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
for d in docs(tf):
for case in (d.get("results") or []):
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
return True
return False
stray: list[str] = []
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
for f in TRACKED_YAML:
for d in docs(f):
labels = ((d.get("metadata") or {}).get("labels") or {})
cls = labels.get(CLASS_LABEL)
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
if value is None or value in PATHS or value == "CLASS":
continue
name = ((owner.get("metadata") or {}).get("name") or "?")
if rejected_by_a_policy_test(name, f):
continue
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
# Values written as raw text (templates, docs-as-code) are caught too.
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
if value in PATHS or value == "CLASS":
continue
if any(value in s for s in stray):
continue
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
# carries this value is a policy-test fixture proven to be rejected.
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
continue
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
check(not stray,
f"tracked YAML assigns egress-class value(s) outside the declared path set "
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
# between the declaration and the path.
rtext = RENDER.read_text()
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
"two independent facts that could disagree")
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
policies: list[tuple[Path, dict]] = []
for p in TRACKED_YAML:
for d in docs(p):
if d.get("kind") == "CiliumEgressGatewayPolicy":
policies.append((p, d))
check(len(policies) == 1,
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
"something the repository states")
for path, pol in policies:
rel = str(path.relative_to(ROOT))
spec = pol.get("spec") or {}
selectors = spec.get("selectors") or []
check(len(selectors) == 1,
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
"selector widens the set of pods steered off the default path, and R-NET-04's "
"selection must be exactly 'the vpn class'")
for sel in selectors:
pod_sel = sel.get("podSelector")
check(isinstance(pod_sel, dict) and pod_sel,
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
"default path at all")
ml = (pod_sel or {}).get("matchLabels") or {}
me = (pod_sel or {}).get("matchExpressions") or []
check(not me,
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
"single label equality; expressions (In/NotIn/Exists) can select pods that "
"never declared a class, so the rule set would no longer be disjoint from the "
"default path")
check(set(ml) == {CLASS_LABEL},
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
"than the declared class (and a namespace/app key would silently exempt pods)")
check(ml.get(CLASS_LABEL) == "vpn",
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
"route the default path through the tunnel, i.e. leave one path, not two")
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
gw = spec.get("egressGateway") or {}
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
check(bool(node_sel),
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
"nodes + Egress Gateway HA)")
check("nodeName" not in gw,
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
"label so the path survives losing that node")
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
f"{rel}: the egressGateway names no egress interface — without it the steered class "
"is not actually sent down a different path from the default")
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
dests = spec.get("destinationCIDRs") or []
excl = spec.get("excludedCIDRs") or []
check([str(c) for c in dests] == ["0.0.0.0/0"],
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
"minus the exclusions below — narrowing it here would silently leave public "
"destinations on the WAN for a vpn-class service")
try:
excluded = [ipaddress.ip_network(str(c)) for c in excl]
except ValueError as e:
excluded = []
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
check(len(excluded) >= 3,
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
net = ipaddress.ip_network(rfc)
check(covered(net),
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
"coexist for the same pod and east-west traffic would break")
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
# is what makes 'local stays local' true per environment rather than by coincidence.
def overlay_addresses() -> list[tuple[str, str]]:
found: list[tuple[str, str]] = []
base = CLUSTERS / "base" / "cluster-vars.yaml"
data = {}
if base.is_file():
for d in docs(base):
if d.get("kind") == "ConfigMap":
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
env = {**data}
for m in re.finditer(
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
label = f.parent.name
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
v = env.get(key, "")
if v and not v.startswith("TBD"):
found.append((f"{label}/{key}", v))
return found
addrs = overlay_addresses()
check(len(addrs) >= 5,
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
"the 'local traffic stays local' assertion would be checking almost nothing")
for label, value in addrs:
try:
net = ipaddress.ip_network(value, strict=False)
except ValueError:
continue
check(covered(net),
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
"would try to reach its own cluster/LAN through the VPN")
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
TPL = VPN_ROLE / "templates"
up = (TPL / "vpn-egress-up.sh.j2")
down = (TPL / "vpn-egress-down.sh.j2")
conf = (TPL / "vpn-egress.conf.j2")
nft = (TPL / "vpn-egress-killswitch.nft.j2")
for f in (up, down, conf, nft):
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
"rules are not authored")
if FAILURES:
die()
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
def render(text: str) -> str:
"""Substitute `{{ var }}` with the role's own default value, for every default.
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
exactly what an earlier draft of this check did. Rendering first means the assertions
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
substituted; anything else is left as-is (and the assertions are written so an
unsubstituted expression fails rather than passes).
"""
out = text
for k, v in defaults.items():
if isinstance(v, (str, int)):
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
return out
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
"putting the tunnel's default route in any of them makes the VPN the node's default "
"route instead of a selectable second path")
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
"needs a rule, at a defined precedence")
# Every `ip route ... default` the up-script installs must name the dedicated table.
route_lines = [l.strip() for l in up_t.splitlines()
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
check(len(route_lines) >= 2,
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
"default route via the tunnel AND the fail-closed blackhole behind it")
for l in route_lines:
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
"landing in the main table changes the path every other pod and the node itself "
"uses — the VPN would stop being opt-in")
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
"be blackholed")
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
# everything into the tunnel table, which would leave one path instead of two.
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
check(len(rule_lines) >= 1,
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
for l in rule_lines:
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
"VPN is no longer opt-in")
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
f"an ip rule looks up a table other than the tunnel's: {l!r}")
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
"distinguishes the steered class from every other flow on the gateway node")
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
"the up-script does not refuse to run when the tunnel address is unknown — installing "
"a partial rule set is how a path ends up half-selected")
# The provider must not be able to make the tunnel the default route either.
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
"routes itself, in the MAIN table, and the VPN would become the node's default route "
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
"routing)')")
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
"provider could redirect the node's whole default route into the tunnel, collapsing "
"the two paths into one")
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
"default route")
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
check(bool(src_set),
"the role declares no nftables source set — the kill-switch would have no way to name "
"'the steered class' and could only match broadly")
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
check(len(policies_lines) >= 1,
"the kill-switch nft template declares no base chain — nothing would be enforced")
for l in policies_lines:
check(re.search(r"policy\s+accept\s*;", l) is not None,
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
drop_lines = [l.strip() for l in nft_t.splitlines()
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
check(len(drop_lines) >= 1,
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
for l in drop_lines:
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
check(scoped,
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
"the default `internet` path, which must never be blackholed")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 35,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
PY