Files
home-network-k3s-impl/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml
T
jakman85andClaude Opus 4.8 0e69c63b55 conformance: complete vpn-killswitch L3 test + ADR-0015 RBAC/fixtures
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.

- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
  RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
  WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
  hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
  fails, never returns the WAN IP = no leak) while internet-class is unaffected;
  WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
  idempotent start Job (self-healing) so a mid-test failure never leaves the
  gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
  (namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
  pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
  ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).

Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 13:58:40 -05:00

330 lines
14 KiB
YAML

# =============================================================================
# L3 FAULT-INJECTION — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL* [RUNNABLE]
# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)".
# Spec: ADR-0015 (node-level host tunnel + Cilium Egress Gateway + host nftables
# kill-switch), docs/03 "networking isolation & egress" (fail-closed).
#
# GIVEN a `vpn`-class workload with a HEALTHY host tunnel (its egress is the PIA IP,
# distinct from the WAN),
# WHEN the tunnel is FORCED DOWN — the host `vpn-egress-tunnel` systemd unit on the
# egress-gateway node is STOPPED (fault injection) —
# THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
# and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
# AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
# WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
#
# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check does
# not satisfy R-NET-06.
#
# FAULT MECHANISM (ADR-0015): the tunnel is HOST infra, not a pod, so the fault is a
# host systemd action. A Chainsaw script runs in-cluster, so stopping a host service
# needs a privileged, node-pinned pod: a Job with nodeSelector egress-gateway=true,
# hostPID: true, a privileged container running
# nsenter -t 1 -m -u -i -n -p -- systemctl stop|start vpn-egress-tunnel
# (enters PID 1's namespaces to drive the host systemctl). See
# fixtures/vpn-and-internet-pods.yaml for the RBAC / PodSecurity / namespace notes.
#
# SELF-HEALING: the spec-level `finally` ALWAYS runs a `systemctl start` Job so a
# failure mid-test NEVER leaves the gateway tunnel down (idempotent — start on an
# already-running unit is a no-op success).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vpn-killswitch
labels:
conformance.home.k3s/R-NET-06: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-06"
conformance.home.k3s/layer: "L3"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Stopping the host vpn-egress-tunnel service blackholes vpn-class egress (no
leak, no WAN IP) while internet-class is unaffected; restarting recovers it.
spec:
timeouts:
apply: 90s
assert: 180s
exec: 180s
steps:
# ---------------------------------------------------------------------------
- name: given-healthy-tunnel-and-two-workloads
try:
# The fault Jobs below need privileged + hostPID, which the platform-default
# `restricted` PSA on the ephemeral test namespace forbids. Relabel THIS
# per-test namespace only (torn down with it) so the blast radius is scoped.
- script:
content: |
set -eu
kubectl label namespace "$NAMESPACE" \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/warn=privileged \
pod-security.kubernetes.io/audit=privileged \
--overwrite
echo "NS_PRIVILEGED_OK"
check:
(contains($stdout, 'NS_PRIVILEGED_OK')): true
- apply:
file: fixtures/vpn-and-internet-pods.yaml
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: egress-internet }
status: { phase: Running }
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: egress-vpn }
status: { phase: Running }
- script:
# Baseline: tunnel HEALTHY. vpn-class egress works AND differs from WAN
# (proves it is steered through tun0, not leaking). Record the WAN IP for
# the leak check below. IP-agnostic (compares classes, no hardcoded IPs).
content: |
set -eu
ns="$NAMESPACE"
get_ip() {
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
wan="" vpn=""
for i in $(seq 1 12); do
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
sleep 5
done
echo "wan=$wan vpn=$vpn"
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
echo "TUNNEL_HEALTHY_OK"
check:
(contains($stdout, 'TUNNEL_HEALTHY_OK')): true
# ---------------------------------------------------------------------------
- name: when-force-tunnel-down-then-no-leak
try:
# -- FAULT INJECTION: stop the host `vpn-egress-tunnel` unit on the gateway --
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-stop
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 2
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true # see PID 1 to enter its namespaces
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true" # pin to gateway node
tolerations:
- operator: Exists # schedule even if the gateway node is tainted
containers:
- name: nsenter
# busybox ships an `nsenter` applet; if a build lacks it, swap
# for a util-linux image (e.g. debian:12-slim + util-linux).
image: busybox:1.37.0
securityContext:
privileged: true # CAP_SYS_ADMIN for setns()
command:
- nsenter
- "-t"
- "1"
- "-m"
- "-u"
- "-i"
- "-n"
- "-p"
- "--"
- systemctl
- stop
- vpn-egress-tunnel
- assert:
# The stop actually ran on the host (Job completed).
resource:
apiVersion: batch/v1
kind: Job
metadata: { name: tunnel-stop }
status: { succeeded: 1 }
- script:
# THEN: vpn-class egress must reach NOTHING. Poll — ANY successful response
# is a LEAK (with tun0 down the kill-switch blackholes the class), and a
# response equal to the WAN IP is the specific leak we must never see.
content: |
set -u
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
for i in $(seq 1 8); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$out" ]; then
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "attempt $i: vpn-class blackholed (no response), good"
sleep 3
done
echo "NO_LEAK_OK"
check:
(contains($stdout, 'NO_LEAK_OK')): true
- script:
# AND: internet-class egress is UNAFFECTED (fails OPEN — kill-switch is
# scoped to vpn-class only, ADR-0015 invariant 3).
content: |
set -eu
ns="$NAMESPACE"
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
echo "internet_ip=$out"
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
echo "INTERNET_UNAFFECTED_OK"
check:
(contains($stdout, 'INTERNET_UNAFFECTED_OK')): true
# ---------------------------------------------------------------------------
- name: when-tunnel-recovers-connectivity-returns
try:
# -- Restart the host tunnel service and prove vpn-class egress comes back. --
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-start
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 2
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command:
- nsenter
- "-t"
- "1"
- "-m"
- "-u"
- "-i"
- "-n"
- "-p"
- "--"
- systemctl
- start
- vpn-egress-tunnel
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata: { name: tunnel-start }
status: { succeeded: 1 }
- script:
# Poll (bounded) until the tunnel reconnects and vpn-class egress returns
# to a NON-WAN (PIA) IP — connectivity restored, still tunneled.
content: |
set -eu
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
vpn=""
for i in $(seq 1 24); do
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
vpn=""
sleep 5
done
echo "recovered_vpn=$vpn wan=$wan"
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
echo "TUNNEL_RECOVERED_OK"
check:
(contains($stdout, 'TUNNEL_RECOVERED_OK')): true
# ---------------------------------------------------------------------------
# SELF-HEALING: no matter how any step above ended, ALWAYS start the tunnel so a
# mid-test failure never leaves the gateway down. Idempotent: `systemctl start` on
# an already-running unit is a no-op success. Distinct Job name so it never
# collides with the recovery-step Job.
finally:
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-start-finally
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 3
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command:
- nsenter
- "-t"
- "1"
- "-m"
- "-u"
- "-i"
- "-n"
- "-p"
- "--"
- systemctl
- start
- vpn-egress-tunnel
# Block until the self-healing start actually completed on the host before the
# namespace (and this Job) is torn down — a leaked-down tunnel is unacceptable.
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata: { name: tunnel-start-finally }
status: { succeeded: 1 }
# Clean up the fault Jobs (belt-and-suspenders; ns teardown also removes them).
- delete:
ref:
apiVersion: batch/v1
kind: Job
labels:
app: vpn-killswitch-fault