feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
5ec1460532
commit
cb1916f092
12 files changed
+2488
No files matched your search
@@ -0,0 +1,199 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE]
|
||||
# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path").
|
||||
# docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is
|
||||
# simply the default route out".
|
||||
# docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.**
|
||||
# Every pod — and the nodes themselves — egress the local/private WAN by default and
|
||||
# **never** the VPN. Only services labeled `egress-class=vpn` are steered to the
|
||||
# tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0`
|
||||
# uses the VPN."
|
||||
#
|
||||
# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT
|
||||
# --------------------------------------------------------------
|
||||
# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their
|
||||
# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing
|
||||
# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds
|
||||
# that pod, and asserts the two claims that make a rule-selected path set complete:
|
||||
#
|
||||
# TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the
|
||||
# explicitly-`internet` pod. Absence of a matching rule IS a path, and it is
|
||||
# the local one. If the steering rule ever widened (an empty selector, a
|
||||
# matchExpression, a second rule), this pod's IP would move to the tunnel and
|
||||
# this assertion is what would notice.
|
||||
# COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one
|
||||
# pass: three pods, two distinct public IPs. "Multiple egress paths" is a
|
||||
# simultaneity claim, not a claim that one switch has two positions.
|
||||
# PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918
|
||||
# destination over the LOCAL path while its public traffic is tunnelled. The
|
||||
# tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod
|
||||
# uses BOTH paths at once.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a
|
||||
# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike
|
||||
# and survives the WAN or the VPN provider rotating addresses.
|
||||
#
|
||||
# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three
|
||||
# times, differing only in the class they declare (or do not),
|
||||
# WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod
|
||||
# also resolves a name through the in-cluster resolver,
|
||||
# THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out
|
||||
# is the WAN, and the VPN is strictly opt-in);
|
||||
# AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path);
|
||||
# AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not
|
||||
# swallowed by the tunnel), so the same pod is using both paths at once.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one
|
||||
# steering rule is authored, that it selects only the vpn class, that no `ip rule` is
|
||||
# unscoped and no nftables DROP unqualified is the L0 half, in
|
||||
# tests/l0-static/egress-paths-selected-by-rules (same requirement label).
|
||||
# - WHICH INTERFACE the packets left by. The observation is the public IP an external echo
|
||||
# reports; it distinguishes the paths without naming them. Asserting "out tun0" would need
|
||||
# host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in
|
||||
# this suite does.
|
||||
# - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn`
|
||||
# fails closed. Its mirror — that `internet` is never blackholed — is asserted statically
|
||||
# by the L0 half, not here.
|
||||
# - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That
|
||||
# is a host-level fact this test cannot see from a pod.
|
||||
# - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster,
|
||||
# so a WAN outage or the echo being down fails this test. That is the same dependency
|
||||
# tests/l2-behavioral/egress-class already carries; the alternative — trusting the
|
||||
# cluster's own account of which path it used — would not be a behavioural test at all.
|
||||
#
|
||||
# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside
|
||||
# that namespace is created or changed — no label on a platform object, no policy edit.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-default-path
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public
|
||||
IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the
|
||||
default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two
|
||||
paths live at once — and the vpn-class pod still reaches in-cluster destinations over the
|
||||
local path, so one pod uses both paths simultaneously.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 2m
|
||||
exec: 3m
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: given-three-pods-differing-only-in-their-declared-egress-class
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/three-egress-pods.yaml
|
||||
# Bare pods (restartPolicy: Never) — Ready == Running here.
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-unlabelled }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-internet }
|
||||
status: { phase: Running }
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: egress-vpn }
|
||||
status: { phase: Running }
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
get_ip() {
|
||||
# Echo the pod's observed public IP, or empty on failure.
|
||||
kubectl -n "$ns" exec "$1" -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Poll until all three answer AND the vpn class has diverged from the WAN:
|
||||
# Cilium may still be programming the egress datapath just after the pods go
|
||||
# Ready. Bounded by the loop, never a bare sleep-and-hope.
|
||||
none="" inet="" vpn=""
|
||||
for i in $(seq 1 12); do
|
||||
[ -n "$none" ] || none="$(get_ip egress-unlabelled)"
|
||||
[ -n "$inet" ] || inet="$(get_ip egress-internet)"
|
||||
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
||||
if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then
|
||||
[ "$vpn" != "$inet" ] && break
|
||||
# Steering may not have converged; re-read the vpn pod only.
|
||||
vpn=""
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn"
|
||||
|
||||
[ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; }
|
||||
[ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; }
|
||||
[ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; }
|
||||
|
||||
# TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN —
|
||||
# the same WAN the explicitly-internet pod uses.
|
||||
if [ "$none" != "$inet" ]; then
|
||||
echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2
|
||||
exit 1
|
||||
fi
|
||||
# COEXISTENCE: and the vpn path is a genuinely different path, live right now.
|
||||
if [ "$vpn" = "$inet" ]; then
|
||||
echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "EGRESS_DEFAULT_PATH_IS_WAN_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true
|
||||
# -----------------------------------------------------------------------------------
|
||||
- name: then-the-vpn-class-pod-still-reaches-private-destinations-locally
|
||||
try:
|
||||
- script:
|
||||
# PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all
|
||||
# RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the
|
||||
# local datapath. The cluster resolver is an in-cluster (RFC1918) address reached
|
||||
# from the pod's own /etc/resolv.conf — no literal is written here — so a
|
||||
# successful lookup from the vpn-class pod proves the tunnel did not swallow its
|
||||
# local traffic, i.e. ONE pod is using BOTH paths at the same time.
|
||||
content: |
|
||||
set -eu
|
||||
ns="$NAMESPACE"
|
||||
resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)"
|
||||
[ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; }
|
||||
echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)"
|
||||
|
||||
# A cluster-internal name, resolvable only by the in-cluster resolver: reaching
|
||||
# it proves the private-destination path works from a vpn-class pod.
|
||||
out=""
|
||||
for i in $(seq 1 6); do
|
||||
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)"
|
||||
case "$out" in *"Address"*) break ;; esac
|
||||
sleep 5
|
||||
done
|
||||
case "$out" in
|
||||
*Address*) : ;;
|
||||
*) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;;
|
||||
esac
|
||||
echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK"
|
||||
check:
|
||||
(contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true
|
||||
@@ -0,0 +1,67 @@
|
||||
# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing
|
||||
# ONLY in how (or whether) they declare an egress class:
|
||||
#
|
||||
# egress-unlabelled NO egress-class label at all -> selected by NO routing rule
|
||||
# egress-internet egress-class: internet -> the declared default path
|
||||
# egress-vpn egress-class: vpn -> the only class any rule steers
|
||||
#
|
||||
# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress
|
||||
# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no
|
||||
# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must
|
||||
# come out the same WAN as the explicitly-`internet` pod. That is the totality half of
|
||||
# R-NET-04's "selected by routing rules" — absence of a rule is itself a path.
|
||||
#
|
||||
# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an
|
||||
# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in
|
||||
# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee
|
||||
# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe
|
||||
# the datapath's own default at all.
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-unlabelled
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-internet
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: egress-vpn
|
||||
labels:
|
||||
app: egress-default-path-test
|
||||
homelab.keithjackson.dev/egress-class: vpn
|
||||
spec:
|
||||
containers:
|
||||
- name: curl
|
||||
image: curlimages/curl:8.11.1
|
||||
command: ["sleep", "3600"]
|
||||
resources:
|
||||
requests: { cpu: 10m, memory: 16Mi }
|
||||
limits: { memory: 32Mi }
|
||||
restartPolicy: Never
|
||||
Reference in new issue
Block a user