jakman85andClaude Opus 4.8
4c21f9f6e3
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
279 lines
13 KiB
YAML
279 lines
13 KiB
YAML
# =============================================================================
|
|
# L3 FAULT-INJECTION — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL* [RUNNABLE]
|
|
# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)".
|
|
# Spec: ADR-0015 (node-level host tunnel + Cilium Egress Gateway + host nftables
|
|
# kill-switch), docs/03 "networking isolation & egress" (fail-closed).
|
|
#
|
|
# GIVEN a `vpn`-class workload with a HEALTHY host tunnel (its egress is the PIA IP,
|
|
# distinct from the WAN),
|
|
# WHEN the tunnel is FORCED DOWN — the host `vpn-egress-tunnel` systemd unit on the
|
|
# egress-gateway node is STOPPED (fault injection) —
|
|
# THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
|
|
# and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
|
|
# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
|
|
# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
|
|
# AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
|
|
# WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
|
|
#
|
|
# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check does
|
|
# not satisfy R-NET-06.
|
|
#
|
|
# FAULT MECHANISM (ADR-0015): the tunnel is HOST infra, not a pod, so the fault is a
|
|
# host systemd action. A Chainsaw script runs in-cluster, so stopping a host service
|
|
# needs a privileged, node-pinned pod: a Job with nodeSelector egress-gateway=true,
|
|
# hostPID: true, a privileged container running
|
|
# nsenter -t 1 -m -u -i -n -p -- systemctl stop|start vpn-egress-tunnel
|
|
# (enters PID 1's namespaces to drive the host systemctl). See
|
|
# fixtures/vpn-and-internet-pods.yaml for the RBAC / PodSecurity / namespace notes.
|
|
#
|
|
# SELF-HEALING: the spec-level `finally` ALWAYS runs a `systemctl start` Job so a
|
|
# failure mid-test NEVER leaves the gateway tunnel down (idempotent — start on an
|
|
# already-running unit is a no-op success).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vpn-killswitch
|
|
labels:
|
|
conformance.home.k3s/R-NET-06: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-06"
|
|
conformance.home.k3s/layer: "L3"
|
|
conformance.home.k3s/safety-critical: "true"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Stopping the host vpn-egress-tunnel service blackholes vpn-class egress (no
|
|
leak, no WAN IP) while internet-class is unaffected; restarting recovers it.
|
|
spec:
|
|
timeouts:
|
|
apply: 90s
|
|
assert: 180s
|
|
exec: 180s
|
|
steps:
|
|
# ---------------------------------------------------------------------------
|
|
- name: given-healthy-tunnel-and-two-workloads
|
|
try:
|
|
# The fault Jobs below need privileged + hostPID, which the platform-default
|
|
# `restricted` PSA on the ephemeral test namespace forbids. Relabel THIS
|
|
# per-test namespace only (torn down with it) so the blast radius is scoped.
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
kubectl label namespace "$NAMESPACE" \
|
|
pod-security.kubernetes.io/enforce=privileged \
|
|
pod-security.kubernetes.io/warn=privileged \
|
|
pod-security.kubernetes.io/audit=privileged \
|
|
--overwrite
|
|
echo "NS_PRIVILEGED_OK"
|
|
check:
|
|
(contains($stdout, 'NS_PRIVILEGED_OK')): true
|
|
- apply:
|
|
file: fixtures/vpn-and-internet-pods.yaml
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-internet }
|
|
status: { phase: Running }
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-vpn }
|
|
status: { phase: Running }
|
|
- script:
|
|
# Baseline: tunnel HEALTHY. vpn-class egress works AND differs from WAN
|
|
# (proves it is steered through tun0, not leaking). Record the WAN IP for
|
|
# the leak check below. IP-agnostic (compares classes, no hardcoded IPs).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
get_ip() {
|
|
kubectl -n "$ns" exec "$1" -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
|
}
|
|
wan="" vpn=""
|
|
for i in $(seq 1 12); do
|
|
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
|
|
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
|
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
|
|
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
|
|
sleep 5
|
|
done
|
|
echo "wan=$wan vpn=$vpn"
|
|
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
|
|
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_HEALTHY_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_HEALTHY_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-force-tunnel-down-then-no-leak
|
|
try:
|
|
# -- FAULT INJECTION: stop the host `vpn-egress-tunnel` unit on the gateway --
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-stop
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true # see PID 1 to enter its namespaces
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true" # pin to gateway node
|
|
tolerations:
|
|
- operator: Exists # schedule even if the gateway node is tainted
|
|
containers:
|
|
- name: nsenter
|
|
# busybox ships an `nsenter` applet; if a build lacks it, swap
|
|
# for a util-linux image (e.g. debian:12-slim + util-linux).
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true # CAP_SYS_ADMIN for setns()
|
|
command:
|
|
- nsenter
|
|
- "-t"
|
|
- "1"
|
|
- "-m"
|
|
- "-u"
|
|
- "-i"
|
|
- "-n"
|
|
- "-p"
|
|
- "--"
|
|
- systemctl
|
|
- stop
|
|
- vpn-egress-tunnel
|
|
- assert:
|
|
# The stop actually ran on the host (Job completed).
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tunnel-stop }
|
|
status: { succeeded: 1 }
|
|
- script:
|
|
# THEN: vpn-class egress must reach NOTHING. Poll — ANY successful response
|
|
# is a LEAK (with tun0 down the kill-switch blackholes the class), and a
|
|
# response equal to the WAN IP is the specific leak we must never see.
|
|
content: |
|
|
set -u
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
for i in $(seq 1 8); do
|
|
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$out" ]; then
|
|
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
|
|
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "attempt $i: vpn-class blackholed (no response), good"
|
|
sleep 3
|
|
done
|
|
echo "NO_LEAK_OK"
|
|
check:
|
|
(contains($stdout, 'NO_LEAK_OK')): true
|
|
- script:
|
|
# AND: internet-class egress is UNAFFECTED (fails OPEN — kill-switch is
|
|
# scoped to vpn-class only, ADR-0015 invariant 3).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
echo "internet_ip=$out"
|
|
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
|
|
echo "INTERNET_UNAFFECTED_OK"
|
|
check:
|
|
(contains($stdout, 'INTERNET_UNAFFECTED_OK')): true
|
|
# -- SELF-HEALING (STEP-LEVEL finally — chainsaw v0.2.12 has NO test-level
|
|
# spec.finally; finally lives on the step, and it only accepts catch-style ops
|
|
# like `script`, NOT `apply`/`assert`). This runs ALWAYS, even if a `try`
|
|
# assertion above FAILED, so a failed no-leak / internet-unaffected assertion
|
|
# can never leave the gateway tunnel down. This is the ONLY step that stops the
|
|
# tunnel, so restarting it here closes every path to a leaked-down gateway.
|
|
# We create the privileged restart Job with kubectl and `kubectl wait` for it —
|
|
# the equivalent of an `assert succeeded:1`: teardown blocks until the host
|
|
# confirms the unit is back up. Idempotent: `systemctl start` on an already-
|
|
# running unit is a no-op success.
|
|
finally:
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
# Distinct Job name (tunnel-restart) so it never collides with the
|
|
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
|
|
# valid YAML at column 0 at runtime.
|
|
cat <<'JOB' | kubectl -n "$ns" apply -f -
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-restart
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 3
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true"
|
|
tolerations:
|
|
- operator: Exists
|
|
containers:
|
|
- name: nsenter
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true
|
|
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
|
|
JOB
|
|
# Block teardown until the restart actually completed on the host — a
|
|
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
|
|
# or timeout, which (set -e) fails the step and surfaces the problem.
|
|
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
|
|
echo "TUNNEL_SELF_HEAL_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_SELF_HEAL_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-tunnel-recovers-connectivity-returns
|
|
# The tunnel was already restarted by the previous step's self-healing
|
|
# `finally`; here we prove the EFFECT — vpn-class egress returns to a NON-WAN
|
|
# (PIA) IP within a bounded window (tunnel reconnect + datapath convergence).
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
vpn=""
|
|
for i in $(seq 1 24); do
|
|
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
|
|
vpn=""
|
|
sleep 5
|
|
done
|
|
echo "recovered_vpn=$vpn wan=$wan"
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
|
|
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_RECOVERED_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_RECOVERED_OK')): true
|
|
# Fault Jobs + fixtures are torn down automatically by Chainsaw's per-test
|
|
# namespace deletion (config/chainsaw-config.yaml: skipDelete: false).
|