jakman85andClaude Opus 5
dc2d1ce17e
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]>
522 lines
30 KiB
Bash
Executable File
522 lines
30 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
|
|
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
|
|
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
|
|
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
|
|
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
|
|
# uplinks) is **out of scope** unless a future ADR reopens it."
|
|
# — spec docs/00-vision-and-scope.md).
|
|
#
|
|
# The elaborating spec text this check quotes and asserts against — docs/03
|
|
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
|
|
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
|
|
# router -> internet |
|
|
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
|
|
# egress gateway (ADR-0013) |
|
|
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
|
|
# to policy-route across; the `internet` path is simply the default route out."
|
|
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
|
|
# themselves — egress the local/private WAN by default and **never** the VPN. Only
|
|
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
|
|
# blackholes **only** that class."
|
|
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
|
|
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
|
|
# only what Cilium points at `tun0` uses the VPN."
|
|
#
|
|
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
|
|
# ---------------------------------------------------------------------------------
|
|
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
|
|
# and the class can be flipped via GitOps. Those are proven by
|
|
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
|
|
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
|
|
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
|
|
#
|
|
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
|
|
# distinctive content — none of which any existing test asserts — is:
|
|
#
|
|
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
|
|
# one closed set of two values agreed by the schema, the admission policy and the
|
|
# renderer. A third value appearing in one of them but not the others would be a path
|
|
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
|
|
# class would contradict OQ-1's resolution.
|
|
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
|
|
# path is entered only via rules that select the vpn class — a Cilium
|
|
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
|
|
# to the tunnel source address. Nothing selects the `internet` class, because
|
|
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
|
|
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
|
|
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
|
|
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
|
|
# pod's cluster/LAN traffic still takes the local path at the same time as its
|
|
# internet traffic takes the tunnel.
|
|
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
|
|
# lives in a DEDICATED routing table reached only through the scoped rule, the
|
|
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
|
|
# from touching the main table. Otherwise there would be one path (the VPN), not two.
|
|
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
|
|
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
|
|
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
|
|
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
|
|
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
|
|
# asserts the OTHER direction, which that test does not.)
|
|
#
|
|
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
|
|
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
|
|
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
|
|
# table") are statements about ALL rules, including the ones that would have to be absent.
|
|
# A running cluster can show you which path two pods took; it cannot show you that no third
|
|
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
|
|
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
|
|
# tests/l2-behavioral/egress-default-path (same requirement label).
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
|
|
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
|
|
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
|
|
# LAN is checked against its own addresses, and the check keeps working when they change.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
|
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
|
|
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
|
|
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
|
|
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
|
|
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
|
|
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
|
|
# invisible here (and to every other test); drift on the box is an Ansible check-mode
|
|
# exercise for the operator.
|
|
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
|
|
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
|
|
# because nobody tried" — and inventing one would be a hollow assertion.
|
|
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
|
|
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
|
|
# this check would NOT catch its absence.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
|
import ipaddress
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
try:
|
|
import yaml
|
|
except ImportError: # pragma: no cover
|
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
|
file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
FAILURES: list[str] = []
|
|
CHECKS = 0
|
|
LBL = "homelab.keithjackson.dev"
|
|
CLASS_LABEL = f"{LBL}/egress-class"
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
def die():
|
|
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
|
|
"assertions failed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" - {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
|
|
def docs(path: Path):
|
|
"""Every YAML document in a file, skipping empties."""
|
|
try:
|
|
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
|
except yaml.YAMLError:
|
|
return []
|
|
|
|
|
|
# Skip dot-DIRECTORIES (.git, .claude, .cache) while still matching dot-FILES such as
|
|
# .sops.yaml. A bare rglob from ROOT also descends into NESTED WORKTREES —
|
|
# .claude/worktrees/agent-*/ created for isolated agent runs — which made this check see
|
|
# 5 CiliumEgressGatewayPolicy objects instead of the 1 authored, failing R-NET-04 for a
|
|
# reason that has nothing to do with egress. .claude/ is gitignored (.gitignore:43), so
|
|
# nothing under it is part of the authored surface by definition.
|
|
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
|
|
if "_artifacts" not in p.parts
|
|
and not any(part.startswith(".")
|
|
for part in p.relative_to(ROOT).parent.parts))
|
|
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
|
|
RENDER = ROOT / "catalog" / "render.py"
|
|
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
|
|
CLUSTERS = ROOT / "clusters"
|
|
|
|
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
|
|
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
|
|
"read at all (R-NET-04)")
|
|
if FAILURES:
|
|
die()
|
|
|
|
|
|
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
|
|
import json # noqa: E402 — only needed for the schema
|
|
|
|
schema = json.loads(SCHEMA.read_text())
|
|
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
|
|
enum = egress_prop.get("enum")
|
|
check(isinstance(enum, list) and len(enum) == 2,
|
|
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
|
|
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
|
|
"nothing routes; one value is not 'multiple egress paths'")
|
|
PATHS = set(enum or [])
|
|
check(PATHS == {"internet", "vpn"},
|
|
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
|
|
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
|
|
check(egress_prop.get("default") == "internet",
|
|
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
|
|
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
|
|
"the default route out'. A service that says nothing must get the WAN")
|
|
|
|
# The admission policy must constrain the pod label to the SAME closed set, or a workload
|
|
# could carry a class the schema never allowed (and that no rule routes).
|
|
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
|
|
check(len(pol_files) >= 1,
|
|
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
|
|
"carrying an egress class outside the declared path set")
|
|
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
|
|
for cls in sorted(PATHS):
|
|
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
|
|
f"the egress admission policy never names the {cls!r} path — a class the policy "
|
|
"does not know cannot be validated against the path set")
|
|
# No tracked object may carry a class outside the closed set — with ONE self-policing
|
|
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
|
|
# admission test can assert the policy REJECTS it; that pod is exempt only while a
|
|
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
|
|
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
|
|
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
|
|
tests_root = fixture.parent
|
|
while tests_root != ROOT and tests_root.name != "tests":
|
|
tests_root = tests_root.parent
|
|
if tests_root == ROOT:
|
|
return False
|
|
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
|
|
for d in docs(tf):
|
|
for case in (d.get("results") or []):
|
|
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
|
|
return True
|
|
return False
|
|
|
|
|
|
stray: list[str] = []
|
|
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
|
|
for f in TRACKED_YAML:
|
|
for d in docs(f):
|
|
labels = ((d.get("metadata") or {}).get("labels") or {})
|
|
cls = labels.get(CLASS_LABEL)
|
|
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
|
|
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
|
|
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
|
|
if value is None or value in PATHS or value == "CLASS":
|
|
continue
|
|
name = ((owner.get("metadata") or {}).get("name") or "?")
|
|
if rejected_by_a_policy_test(name, f):
|
|
continue
|
|
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
|
|
# Values written as raw text (templates, docs-as-code) are caught too.
|
|
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
|
|
if value in PATHS or value == "CLASS":
|
|
continue
|
|
if any(value in s for s in stray):
|
|
continue
|
|
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
|
|
# carries this value is a policy-test fixture proven to be rejected.
|
|
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
|
|
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
|
|
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
|
|
continue
|
|
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
|
|
check(not stray,
|
|
f"tracked YAML assigns egress-class value(s) outside the declared path set "
|
|
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
|
|
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
|
|
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
|
|
|
|
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
|
|
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
|
|
# between the declaration and the path.
|
|
rtext = RENDER.read_text()
|
|
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
|
|
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
|
|
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
|
|
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
|
|
"two independent facts that could disagree")
|
|
|
|
|
|
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
|
|
policies: list[tuple[Path, dict]] = []
|
|
for p in TRACKED_YAML:
|
|
for d in docs(p):
|
|
if d.get("kind") == "CiliumEgressGatewayPolicy":
|
|
policies.append((p, d))
|
|
check(len(policies) == 1,
|
|
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
|
|
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
|
|
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
|
|
"something the repository states")
|
|
|
|
for path, pol in policies:
|
|
rel = str(path.relative_to(ROOT))
|
|
spec = pol.get("spec") or {}
|
|
selectors = spec.get("selectors") or []
|
|
check(len(selectors) == 1,
|
|
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
|
|
"selector widens the set of pods steered off the default path, and R-NET-04's "
|
|
"selection must be exactly 'the vpn class'")
|
|
for sel in selectors:
|
|
pod_sel = sel.get("podSelector")
|
|
check(isinstance(pod_sel, dict) and pod_sel,
|
|
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
|
|
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
|
|
"default path at all")
|
|
ml = (pod_sel or {}).get("matchLabels") or {}
|
|
me = (pod_sel or {}).get("matchExpressions") or []
|
|
check(not me,
|
|
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
|
|
"single label equality; expressions (In/NotIn/Exists) can select pods that "
|
|
"never declared a class, so the rule set would no longer be disjoint from the "
|
|
"default path")
|
|
check(set(ml) == {CLASS_LABEL},
|
|
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
|
|
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
|
|
"than the declared class (and a namespace/app key would silently exempt pods)")
|
|
check(ml.get(CLASS_LABEL) == "vpn",
|
|
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
|
|
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
|
|
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
|
|
"route the default path through the tunnel, i.e. leave one path, not two")
|
|
|
|
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
|
|
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
|
|
gw = spec.get("egressGateway") or {}
|
|
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
|
|
check(bool(node_sel),
|
|
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
|
|
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
|
|
"nodes + Egress Gateway HA)")
|
|
check("nodeName" not in gw,
|
|
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
|
|
"label so the path survives losing that node")
|
|
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
|
|
f"{rel}: the egressGateway names no egress interface — without it the steered class "
|
|
"is not actually sent down a different path from the default")
|
|
|
|
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
|
|
dests = spec.get("destinationCIDRs") or []
|
|
excl = spec.get("excludedCIDRs") or []
|
|
check([str(c) for c in dests] == ["0.0.0.0/0"],
|
|
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
|
|
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
|
|
"minus the exclusions below — narrowing it here would silently leave public "
|
|
"destinations on the WAN for a vpn-class service")
|
|
try:
|
|
excluded = [ipaddress.ip_network(str(c)) for c in excl]
|
|
except ValueError as e:
|
|
excluded = []
|
|
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
|
|
check(len(excluded) >= 3,
|
|
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
|
|
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
|
|
|
|
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
|
|
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
|
|
|
|
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
|
|
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
|
|
net = ipaddress.ip_network(rfc)
|
|
check(covered(net),
|
|
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
|
|
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
|
|
"coexist for the same pod and east-west traffic would break")
|
|
|
|
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
|
|
# is what makes 'local stays local' true per environment rather than by coincidence.
|
|
def overlay_addresses() -> list[tuple[str, str]]:
|
|
found: list[tuple[str, str]] = []
|
|
base = CLUSTERS / "base" / "cluster-vars.yaml"
|
|
data = {}
|
|
if base.is_file():
|
|
for d in docs(base):
|
|
if d.get("kind") == "ConfigMap":
|
|
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
|
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
|
|
env = {**data}
|
|
for m in re.finditer(
|
|
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
|
|
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
|
|
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
|
label = f.parent.name
|
|
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
|
|
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
|
|
v = env.get(key, "")
|
|
if v and not v.startswith("TBD"):
|
|
found.append((f"{label}/{key}", v))
|
|
return found
|
|
|
|
addrs = overlay_addresses()
|
|
check(len(addrs) >= 5,
|
|
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
|
|
"the 'local traffic stays local' assertion would be checking almost nothing")
|
|
for label, value in addrs:
|
|
try:
|
|
net = ipaddress.ip_network(value, strict=False)
|
|
except ValueError:
|
|
continue
|
|
check(covered(net),
|
|
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
|
|
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
|
|
"would try to reach its own cluster/LAN through the VPN")
|
|
|
|
|
|
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
|
|
TPL = VPN_ROLE / "templates"
|
|
up = (TPL / "vpn-egress-up.sh.j2")
|
|
down = (TPL / "vpn-egress-down.sh.j2")
|
|
conf = (TPL / "vpn-egress.conf.j2")
|
|
nft = (TPL / "vpn-egress-killswitch.nft.j2")
|
|
for f in (up, down, conf, nft):
|
|
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
|
|
"rules are not authored")
|
|
if FAILURES:
|
|
die()
|
|
|
|
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
|
|
|
|
|
|
def render(text: str) -> str:
|
|
"""Substitute `{{ var }}` with the role's own default value, for every default.
|
|
|
|
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
|
|
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
|
|
exactly what an earlier draft of this check did. Rendering first means the assertions
|
|
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
|
|
substituted; anything else is left as-is (and the assertions are written so an
|
|
unsubstituted expression fails rather than passes).
|
|
"""
|
|
out = text
|
|
for k, v in defaults.items():
|
|
if isinstance(v, (str, int)):
|
|
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
|
|
return out
|
|
|
|
|
|
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
|
|
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
|
|
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
|
|
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
|
|
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
|
|
"putting the tunnel's default route in any of them makes the VPN the node's default "
|
|
"route instead of a selectable second path")
|
|
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
|
|
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
|
|
"needs a rule, at a defined precedence")
|
|
|
|
# Every `ip route ... default` the up-script installs must name the dedicated table.
|
|
route_lines = [l.strip() for l in up_t.splitlines()
|
|
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
|
|
check(len(route_lines) >= 2,
|
|
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
|
|
"default route via the tunnel AND the fail-closed blackhole behind it")
|
|
for l in route_lines:
|
|
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
|
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
|
|
"landing in the main table changes the path every other pod and the node itself "
|
|
"uses — the VPN would stop being opt-in")
|
|
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
|
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
|
|
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
|
|
"be blackholed")
|
|
|
|
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
|
|
# everything into the tunnel table, which would leave one path instead of two.
|
|
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
|
|
check(len(rule_lines) >= 1,
|
|
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
|
|
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
|
|
for l in rule_lines:
|
|
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
|
|
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
|
|
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
|
|
"VPN is no longer opt-in")
|
|
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
|
|
f"an ip rule looks up a table other than the tunnel's: {l!r}")
|
|
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
|
|
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
|
|
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
|
|
"distinguishes the steered class from every other flow on the gateway node")
|
|
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
|
|
"the up-script does not refuse to run when the tunnel address is unknown — installing "
|
|
"a partial rule set is how a path ends up half-selected")
|
|
|
|
# The provider must not be able to make the tunnel the default route either.
|
|
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
|
|
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
|
|
"routes itself, in the MAIN table, and the VPN would become the node's default route "
|
|
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
|
|
"routing)')")
|
|
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
|
|
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
|
|
"provider could redirect the node's whole default route into the tunnel, collapsing "
|
|
"the two paths into one")
|
|
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
|
|
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
|
|
"default route")
|
|
|
|
|
|
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
|
|
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
|
|
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
|
|
check(bool(src_set),
|
|
"the role declares no nftables source set — the kill-switch would have no way to name "
|
|
"'the steered class' and could only match broadly")
|
|
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
|
|
check(len(policies_lines) >= 1,
|
|
"the kill-switch nft template declares no base chain — nothing would be enforced")
|
|
for l in policies_lines:
|
|
check(re.search(r"policy\s+accept\s*;", l) is not None,
|
|
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
|
|
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
|
|
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
|
|
drop_lines = [l.strip() for l in nft_t.splitlines()
|
|
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
|
|
check(len(drop_lines) >= 1,
|
|
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
|
|
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
|
|
for l in drop_lines:
|
|
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
|
|
check(scoped,
|
|
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
|
|
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
|
|
"the default `internet` path, which must never be blackholed")
|
|
|
|
|
|
# --------------------------------------------------------------------------- verdict
|
|
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
|
check(CHECKS >= 35,
|
|
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
|
"would be meaningless")
|
|
|
|
if FAILURES:
|
|
die()
|
|
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
|
|
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
|
|
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
|
|
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
|
|
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
|
|
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
|
|
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
|
|
PY
|