# ============================================================================= # L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in (R-NET-04) [RUNNABLE] # Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster") # Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by # routing rules ... realized as two paths — the local `internet` path and the # in-cluster `vpn` path"). # docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is # simply the default route out". # docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.** # Every pod — and the nodes themselves — egress the local/private WAN by default and # **never** the VPN. Only services labeled `egress-class=vpn` are steered to the # tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0` # uses the VPN." # # WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT # -------------------------------------------------------------- # That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their # public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing # about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds # that pod, and asserts the two claims that make a rule-selected path set complete: # # TOTALITY — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the # explicitly-`internet` pod. Absence of a matching rule IS a path, and it is # the local one. If the steering rule ever widened (an empty selector, a # matchExpression, a second rule), this pod's IP would move to the tunnel and # this assertion is what would notice. # COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one # pass: three pods, two distinct public IPs. "Multiple egress paths" is a # simultaneity claim, not a claim that one switch has two positions. # PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918 # destination over the LOCAL path while its public traffic is tunnelled. The # tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod # uses BOTH paths at once. # # ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a # RELATION between observed values (equal / not-equal), so it holds against dev and prod alike # and survives the WAN or the VPN provider rotating addresses. # # GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three # times, differing only in the class they declare (or do not), # WHEN each asks an external echo for the public IP it egressed from, and the vpn-class pod # also resolves a name through the in-cluster resolver, # THEN the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out # is the WAN, and the VPN is strictly opt-in); # AND the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path); # AND the vpn-class pod still resolves in-cluster (its private-destination traffic was not # swallowed by the tunnel), so the same pod is using both paths at once. # # WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them): # - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one # steering rule is authored, that it selects only the vpn class, that no `ip rule` is # unscoped and no nftables DROP unqualified is the L0 half, in # tests/l0-static/egress-paths-selected-by-rules (same requirement label). # - WHICH INTERFACE the packets left by. The observation is the public IP an external echo # reports; it distinguishes the paths without naming them. Asserting "out tun0" would need # host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in # this suite does. # - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn` # fails closed. Its mirror — that `internet` is never blackholed — is asserted statically # by the L0 half, not here. # - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That # is a host-level fact this test cannot see from a pod. # - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster, # so a WAN outage or the echo being down fails this test. That is the same dependency # tests/l2-behavioral/egress-class already carries; the alternative — trusting the # cluster's own account of which path it used — would not be a behavioural test at all. # # CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside # that namespace is created or changed — no label on a platform object, no policy edit. # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: name: egress-default-path labels: conformance.home.k3s/R-NET-04: "true" annotations: conformance.home.k3s/requirements: "R-NET-04" conformance.home.k3s/layer: "L2" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two paths live at once — and the vpn-class pod still reaches in-cluster destinations over the local path, so one pod uses both paths simultaneously. spec: timeouts: apply: 60s assert: 2m exec: 3m steps: # ----------------------------------------------------------------------------------- - name: given-three-pods-differing-only-in-their-declared-egress-class try: - apply: file: fixtures/three-egress-pods.yaml # Bare pods (restartPolicy: Never) — Ready == Running here. - assert: resource: apiVersion: v1 kind: Pod metadata: { name: egress-unlabelled } status: { phase: Running } - assert: resource: apiVersion: v1 kind: Pod metadata: { name: egress-internet } status: { phase: Running } - assert: resource: apiVersion: v1 kind: Pod metadata: { name: egress-vpn } status: { phase: Running } # ----------------------------------------------------------------------------------- - name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too try: - script: content: | set -eu ns="$NAMESPACE" get_ip() { # Echo the pod's observed public IP, or empty on failure. kubectl -n "$ns" exec "$1" -c curl -- \ curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true } # Poll until all three answer AND the vpn class has diverged from the WAN: # Cilium may still be programming the egress datapath just after the pods go # Ready. Bounded by the loop, never a bare sleep-and-hope. none="" inet="" vpn="" for i in $(seq 1 12); do [ -n "$none" ] || none="$(get_ip egress-unlabelled)" [ -n "$inet" ] || inet="$(get_ip egress-internet)" [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)" if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then [ "$vpn" != "$inet" ] && break # Steering may not have converged; re-read the vpn pod only. vpn="" fi sleep 5 done echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn" [ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; } [ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; } [ -n "$vpn" ] || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; } # TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN — # the same WAN the explicitly-internet pod uses. if [ "$none" != "$inet" ]; then echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2 exit 1 fi # COEXISTENCE: and the vpn path is a genuinely different path, live right now. if [ "$vpn" = "$inet" ]; then echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2 exit 1 fi echo "EGRESS_DEFAULT_PATH_IS_WAN_OK" check: (contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true # ----------------------------------------------------------------------------------- - name: then-the-vpn-class-pod-still-reaches-private-destinations-locally try: - script: # PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all # RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the # local datapath. The cluster resolver is an in-cluster (RFC1918) address reached # from the pod's own /etc/resolv.conf — no literal is written here — so a # successful lookup from the vpn-class pod proves the tunnel did not swallow its # local traffic, i.e. ONE pod is using BOTH paths at the same time. content: | set -eu ns="$NAMESPACE" resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)" [ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; } echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)" # A cluster-internal name, resolvable only by the in-cluster resolver: reaching # it proves the private-destination path works from a vpn-class pod. out="" for i in $(seq 1 6); do out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \ nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)" case "$out" in *"Address"*) break ;; esac sleep 5 done case "$out" in *Address*) : ;; *) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;; esac echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK" check: (contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true