jakman85andClaude Opus 5
cb1916f092
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
95 lines
6.1 KiB
YAML
95 lines
6.1 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
|
# routing rules ... realized as two paths — the local `internet` path and the
|
|
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
|
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
|
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
|
# the nodes themselves — egress the local/private WAN by default and **never** the
|
|
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
|
# kill-switch blackholes **only** that class."
|
|
#
|
|
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
|
# ------------------------------------------------------------------------
|
|
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
|
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
|
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
|
# that select among them, which nothing else asserts:
|
|
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
|
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
|
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
|
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
|
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
|
# absence of a matching rule IS the default path;
|
|
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
|
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
|
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
|
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
|
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
|
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
|
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
|
#
|
|
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
|
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
|
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
|
# no third rule exists or that the vocabulary is closed.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
|
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
|
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
|
# against its own values.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
|
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
|
# parsed and cross-checked,
|
|
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
|
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
|
# named egress interface; all RFC1918 space and every address any environment assigns
|
|
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
|
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
|
# cannot take the main table; and no nftables DROP is unscoped.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
|
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
|
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
|
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
|
# that generate them; a hand-added rule on the box is invisible here.
|
|
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
|
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
|
# * IPv6 egress.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: egress-paths-selected-by-rules
|
|
labels:
|
|
conformance.home.k3s/R-NET-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
|
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
|
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
|
the vpn class, local destinations stay on the local path in every environment, the
|
|
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
|
path.
|
|
spec:
|
|
steps:
|
|
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|