Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress Gateway with a host nftables kill-switch), replacing the obsolete in-cluster gluetun design. - fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented RBAC/PodSecurity/namespace requirements for node-level fault injection. - vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN); WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged, hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl fails, never returns the WAN IP = no leak) while internet-class is unaffected; WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an idempotent start Job (self-healing) so a mid-test failure never leaves the gateway tunnel down. - incluster/rbac.yaml: grant the runner least-privilege for the delivered tests (namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/ pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC). Offline validation: python yaml parse (all docs), `run.sh --lint` (structural), `run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/ 07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run. Co-Authored-By: Claude Opus 4.8 <[email protected]>
330 lines
14 KiB
YAML
330 lines
14 KiB
YAML
# =============================================================================
|
|
# L3 FAULT-INJECTION — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL* [RUNNABLE]
|
|
# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)".
|
|
# Spec: ADR-0015 (node-level host tunnel + Cilium Egress Gateway + host nftables
|
|
# kill-switch), docs/03 "networking isolation & egress" (fail-closed).
|
|
#
|
|
# GIVEN a `vpn`-class workload with a HEALTHY host tunnel (its egress is the PIA IP,
|
|
# distinct from the WAN),
|
|
# WHEN the tunnel is FORCED DOWN — the host `vpn-egress-tunnel` systemd unit on the
|
|
# egress-gateway node is STOPPED (fault injection) —
|
|
# THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
|
|
# and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
|
|
# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
|
|
# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
|
|
# AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
|
|
# WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
|
|
#
|
|
# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check does
|
|
# not satisfy R-NET-06.
|
|
#
|
|
# FAULT MECHANISM (ADR-0015): the tunnel is HOST infra, not a pod, so the fault is a
|
|
# host systemd action. A Chainsaw script runs in-cluster, so stopping a host service
|
|
# needs a privileged, node-pinned pod: a Job with nodeSelector egress-gateway=true,
|
|
# hostPID: true, a privileged container running
|
|
# nsenter -t 1 -m -u -i -n -p -- systemctl stop|start vpn-egress-tunnel
|
|
# (enters PID 1's namespaces to drive the host systemctl). See
|
|
# fixtures/vpn-and-internet-pods.yaml for the RBAC / PodSecurity / namespace notes.
|
|
#
|
|
# SELF-HEALING: the spec-level `finally` ALWAYS runs a `systemctl start` Job so a
|
|
# failure mid-test NEVER leaves the gateway tunnel down (idempotent — start on an
|
|
# already-running unit is a no-op success).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vpn-killswitch
|
|
labels:
|
|
conformance.home.k3s/R-NET-06: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-06"
|
|
conformance.home.k3s/layer: "L3"
|
|
conformance.home.k3s/safety-critical: "true"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Stopping the host vpn-egress-tunnel service blackholes vpn-class egress (no
|
|
leak, no WAN IP) while internet-class is unaffected; restarting recovers it.
|
|
spec:
|
|
timeouts:
|
|
apply: 90s
|
|
assert: 180s
|
|
exec: 180s
|
|
steps:
|
|
# ---------------------------------------------------------------------------
|
|
- name: given-healthy-tunnel-and-two-workloads
|
|
try:
|
|
# The fault Jobs below need privileged + hostPID, which the platform-default
|
|
# `restricted` PSA on the ephemeral test namespace forbids. Relabel THIS
|
|
# per-test namespace only (torn down with it) so the blast radius is scoped.
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
kubectl label namespace "$NAMESPACE" \
|
|
pod-security.kubernetes.io/enforce=privileged \
|
|
pod-security.kubernetes.io/warn=privileged \
|
|
pod-security.kubernetes.io/audit=privileged \
|
|
--overwrite
|
|
echo "NS_PRIVILEGED_OK"
|
|
check:
|
|
(contains($stdout, 'NS_PRIVILEGED_OK')): true
|
|
- apply:
|
|
file: fixtures/vpn-and-internet-pods.yaml
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-internet }
|
|
status: { phase: Running }
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-vpn }
|
|
status: { phase: Running }
|
|
- script:
|
|
# Baseline: tunnel HEALTHY. vpn-class egress works AND differs from WAN
|
|
# (proves it is steered through tun0, not leaking). Record the WAN IP for
|
|
# the leak check below. IP-agnostic (compares classes, no hardcoded IPs).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
get_ip() {
|
|
kubectl -n "$ns" exec "$1" -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
|
}
|
|
wan="" vpn=""
|
|
for i in $(seq 1 12); do
|
|
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
|
|
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
|
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
|
|
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
|
|
sleep 5
|
|
done
|
|
echo "wan=$wan vpn=$vpn"
|
|
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
|
|
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_HEALTHY_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_HEALTHY_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-force-tunnel-down-then-no-leak
|
|
try:
|
|
# -- FAULT INJECTION: stop the host `vpn-egress-tunnel` unit on the gateway --
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-stop
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true # see PID 1 to enter its namespaces
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true" # pin to gateway node
|
|
tolerations:
|
|
- operator: Exists # schedule even if the gateway node is tainted
|
|
containers:
|
|
- name: nsenter
|
|
# busybox ships an `nsenter` applet; if a build lacks it, swap
|
|
# for a util-linux image (e.g. debian:12-slim + util-linux).
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true # CAP_SYS_ADMIN for setns()
|
|
command:
|
|
- nsenter
|
|
- "-t"
|
|
- "1"
|
|
- "-m"
|
|
- "-u"
|
|
- "-i"
|
|
- "-n"
|
|
- "-p"
|
|
- "--"
|
|
- systemctl
|
|
- stop
|
|
- vpn-egress-tunnel
|
|
- assert:
|
|
# The stop actually ran on the host (Job completed).
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tunnel-stop }
|
|
status: { succeeded: 1 }
|
|
- script:
|
|
# THEN: vpn-class egress must reach NOTHING. Poll — ANY successful response
|
|
# is a LEAK (with tun0 down the kill-switch blackholes the class), and a
|
|
# response equal to the WAN IP is the specific leak we must never see.
|
|
content: |
|
|
set -u
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
for i in $(seq 1 8); do
|
|
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$out" ]; then
|
|
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
|
|
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "attempt $i: vpn-class blackholed (no response), good"
|
|
sleep 3
|
|
done
|
|
echo "NO_LEAK_OK"
|
|
check:
|
|
(contains($stdout, 'NO_LEAK_OK')): true
|
|
- script:
|
|
# AND: internet-class egress is UNAFFECTED (fails OPEN — kill-switch is
|
|
# scoped to vpn-class only, ADR-0015 invariant 3).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
echo "internet_ip=$out"
|
|
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
|
|
echo "INTERNET_UNAFFECTED_OK"
|
|
check:
|
|
(contains($stdout, 'INTERNET_UNAFFECTED_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-tunnel-recovers-connectivity-returns
|
|
try:
|
|
# -- Restart the host tunnel service and prove vpn-class egress comes back. --
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-start
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true"
|
|
tolerations:
|
|
- operator: Exists
|
|
containers:
|
|
- name: nsenter
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true
|
|
command:
|
|
- nsenter
|
|
- "-t"
|
|
- "1"
|
|
- "-m"
|
|
- "-u"
|
|
- "-i"
|
|
- "-n"
|
|
- "-p"
|
|
- "--"
|
|
- systemctl
|
|
- start
|
|
- vpn-egress-tunnel
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tunnel-start }
|
|
status: { succeeded: 1 }
|
|
- script:
|
|
# Poll (bounded) until the tunnel reconnects and vpn-class egress returns
|
|
# to a NON-WAN (PIA) IP — connectivity restored, still tunneled.
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
vpn=""
|
|
for i in $(seq 1 24); do
|
|
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
|
|
vpn=""
|
|
sleep 5
|
|
done
|
|
echo "recovered_vpn=$vpn wan=$wan"
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
|
|
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_RECOVERED_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_RECOVERED_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
# SELF-HEALING: no matter how any step above ended, ALWAYS start the tunnel so a
|
|
# mid-test failure never leaves the gateway down. Idempotent: `systemctl start` on
|
|
# an already-running unit is a no-op success. Distinct Job name so it never
|
|
# collides with the recovery-step Job.
|
|
finally:
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-start-finally
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 3
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true"
|
|
tolerations:
|
|
- operator: Exists
|
|
containers:
|
|
- name: nsenter
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true
|
|
command:
|
|
- nsenter
|
|
- "-t"
|
|
- "1"
|
|
- "-m"
|
|
- "-u"
|
|
- "-i"
|
|
- "-n"
|
|
- "-p"
|
|
- "--"
|
|
- systemctl
|
|
- start
|
|
- vpn-egress-tunnel
|
|
# Block until the self-healing start actually completed on the host before the
|
|
# namespace (and this Job) is torn down — a leaked-down tunnel is unacceptable.
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tunnel-start-finally }
|
|
status: { succeeded: 1 }
|
|
# Clean up the fault Jobs (belt-and-suspenders; ns teardown also removes them).
|
|
- delete:
|
|
ref:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
labels:
|
|
app: vpn-killswitch-fault
|