jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
100 lines
6.5 KiB
YAML
100 lines
6.5 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
|
# routing rules ... realized as two paths — the local `internet` path and the
|
|
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
|
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
|
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
|
# the nodes themselves — egress the local/private WAN by default and **never** the
|
|
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
|
# kill-switch blackholes **only** that class."
|
|
#
|
|
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
|
# ------------------------------------------------------------------------
|
|
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
|
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
|
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
|
# that select among them, which nothing else asserts:
|
|
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
|
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
|
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
|
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
|
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
|
# absence of a matching rule IS the default path;
|
|
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
|
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
|
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
|
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
|
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
|
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
|
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
|
#
|
|
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
|
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
|
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
|
# no third rule exists or that the vocabulary is closed.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
|
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
|
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
|
# against its own values.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
|
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
|
# parsed and cross-checked,
|
|
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
|
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
|
# named egress interface; all RFC1918 space and every address any environment assigns
|
|
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
|
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
|
# cannot take the main table; and no nftables DROP is unscoped.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
|
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
|
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
|
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
|
# that generate them; a hand-added rule on the box is invisible here.
|
|
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
|
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
|
# * IPv6 egress.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: egress-paths-selected-by-rules
|
|
labels:
|
|
conformance.home.k3s/R-NET-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
|
|
# an egress-class label/annotation must use a value the policy set actually steers. The
|
|
# file most likely to break it is a NEW workload under apps/ — which attributes no
|
|
# requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
|
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
|
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
|
the vpn class, local destinations stay on the local path in every environment, the
|
|
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
|
path.
|
|
spec:
|
|
steps:
|
|
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|