jakman85andClaude Opus 5
04606d48b2
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]>
302 lines
15 KiB
YAML
302 lines
15 KiB
YAML
# =============================================================================
|
|
# L3 FAULT-INJECTION — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL* [RUNNABLE]
|
|
# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)".
|
|
# Spec: ADR-0015 (node-level host tunnel + Cilium Egress Gateway + host nftables
|
|
# kill-switch), docs/03 "networking isolation & egress" (fail-closed).
|
|
#
|
|
# GIVEN a `vpn`-class workload with a HEALTHY host tunnel (its egress is the PIA IP,
|
|
# distinct from the WAN),
|
|
# WHEN the tunnel is FORCED DOWN — the host `vpn-egress-tunnel` systemd unit on the
|
|
# egress-gateway node is STOPPED (fault injection) —
|
|
# THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
|
|
# and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
|
|
# vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4:
|
|
# vpn fails CLOSED).
|
|
#
|
|
# MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables
|
|
# kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway
|
|
# datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device
|
|
# without invoking netfilter POSTROUTING, measured with counters on the gateway
|
|
# node (0 packets for the pod CIDR and the PIA range in forward and postrouting,
|
|
# against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's
|
|
# own drop when it cannot use the cached egress interface; tcpdump on both the WAN
|
|
# and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and
|
|
# still the right one — only the explanation was wrong.
|
|
#
|
|
# PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node
|
|
# carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to
|
|
# it by nodeSelector, and with no gateway there is no tunnel to stop).
|
|
#
|
|
# RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives
|
|
# tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own,
|
|
# so before 2026-09-12 the recovery step could not pass: measured, the map stayed
|
|
# on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway
|
|
# now publishes the live uplink as a node label on every connect, which makes
|
|
# Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP
|
|
# returns). If this test's recovery step ever fails again, check
|
|
# `journalctl -u vpn-egress-resteer` on the gateway node first.
|
|
# AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
|
|
# WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
|
|
#
|
|
# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check does
|
|
# not satisfy R-NET-06.
|
|
#
|
|
# FAULT MECHANISM (ADR-0015): the tunnel is HOST infra, not a pod, so the fault is a
|
|
# host systemd action. A Chainsaw script runs in-cluster, so stopping a host service
|
|
# needs a privileged, node-pinned pod: a Job with nodeSelector egress-gateway=true,
|
|
# hostPID: true, a privileged container running
|
|
# nsenter -t 1 -m -u -i -n -p -- systemctl stop|start vpn-egress-tunnel
|
|
# (enters PID 1's namespaces to drive the host systemctl). See
|
|
# fixtures/vpn-and-internet-pods.yaml for the RBAC / PodSecurity / namespace notes.
|
|
#
|
|
# SELF-HEALING: the spec-level `finally` ALWAYS runs a `systemctl start` Job so a
|
|
# failure mid-test NEVER leaves the gateway tunnel down (idempotent — start on an
|
|
# already-running unit is a no-op success).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vpn-killswitch
|
|
labels:
|
|
conformance.home.k3s/R-NET-06: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-06"
|
|
conformance.home.k3s/layer: "L3"
|
|
conformance.home.k3s/safety-critical: "true"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Stopping the host vpn-egress-tunnel service blackholes vpn-class egress (no
|
|
leak, no WAN IP) while internet-class is unaffected; restarting recovers it.
|
|
spec:
|
|
timeouts:
|
|
apply: 90s
|
|
assert: 180s
|
|
exec: 180s
|
|
steps:
|
|
# ---------------------------------------------------------------------------
|
|
- name: given-healthy-tunnel-and-two-workloads
|
|
try:
|
|
# The fault Jobs below need privileged + hostPID, which the platform-default
|
|
# `restricted` PSA on the ephemeral test namespace forbids. Relabel THIS
|
|
# per-test namespace only (torn down with it) so the blast radius is scoped.
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
kubectl label namespace "$NAMESPACE" \
|
|
pod-security.kubernetes.io/enforce=privileged \
|
|
pod-security.kubernetes.io/warn=privileged \
|
|
pod-security.kubernetes.io/audit=privileged \
|
|
--overwrite
|
|
echo "NS_PRIVILEGED_OK"
|
|
check:
|
|
(contains($stdout, 'NS_PRIVILEGED_OK')): true
|
|
- apply:
|
|
file: fixtures/vpn-and-internet-pods.yaml
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-internet }
|
|
status: { phase: Running }
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: { name: egress-vpn }
|
|
status: { phase: Running }
|
|
- script:
|
|
# Baseline: tunnel HEALTHY. vpn-class egress works AND differs from WAN
|
|
# (proves it is steered through tun0, not leaking). Record the WAN IP for
|
|
# the leak check below. IP-agnostic (compares classes, no hardcoded IPs).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
get_ip() {
|
|
kubectl -n "$ns" exec "$1" -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
|
}
|
|
wan="" vpn=""
|
|
for i in $(seq 1 12); do
|
|
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
|
|
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
|
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
|
|
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
|
|
sleep 5
|
|
done
|
|
echo "wan=$wan vpn=$vpn"
|
|
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
|
|
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_HEALTHY_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_HEALTHY_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-force-tunnel-down-then-no-leak
|
|
try:
|
|
# -- FAULT INJECTION: stop the host `vpn-egress-tunnel` unit on the gateway --
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-stop
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true # see PID 1 to enter its namespaces
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true" # pin to gateway node
|
|
tolerations:
|
|
- operator: Exists # schedule even if the gateway node is tainted
|
|
containers:
|
|
- name: nsenter
|
|
# busybox ships an `nsenter` applet; if a build lacks it, swap
|
|
# for a util-linux image (e.g. debian:12-slim + util-linux).
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true # CAP_SYS_ADMIN for setns()
|
|
command:
|
|
- nsenter
|
|
- "-t"
|
|
- "1"
|
|
- "-m"
|
|
- "-u"
|
|
- "-i"
|
|
- "-n"
|
|
- "-p"
|
|
- "--"
|
|
- systemctl
|
|
- stop
|
|
- vpn-egress-tunnel
|
|
- assert:
|
|
# The stop actually ran on the host (Job completed).
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tunnel-stop }
|
|
status: { succeeded: 1 }
|
|
- script:
|
|
# THEN: vpn-class egress must reach NOTHING. Poll — ANY successful response
|
|
# is a LEAK (with tun0 down the kill-switch blackholes the class), and a
|
|
# response equal to the WAN IP is the specific leak we must never see.
|
|
content: |
|
|
set -u
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
for i in $(seq 1 8); do
|
|
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$out" ]; then
|
|
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
|
|
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
|
|
fi
|
|
echo "attempt $i: vpn-class blackholed (no response), good"
|
|
sleep 3
|
|
done
|
|
echo "NO_LEAK_OK"
|
|
check:
|
|
(contains($stdout, 'NO_LEAK_OK')): true
|
|
- script:
|
|
# AND: internet-class egress is UNAFFECTED (fails OPEN — kill-switch is
|
|
# scoped to vpn-class only, ADR-0015 invariant 3).
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
echo "internet_ip=$out"
|
|
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
|
|
echo "INTERNET_UNAFFECTED_OK"
|
|
check:
|
|
(contains($stdout, 'INTERNET_UNAFFECTED_OK')): true
|
|
# -- SELF-HEALING (STEP-LEVEL finally — chainsaw v0.2.12 has NO test-level
|
|
# spec.finally; finally lives on the step, and it only accepts catch-style ops
|
|
# like `script`, NOT `apply`/`assert`). This runs ALWAYS, even if a `try`
|
|
# assertion above FAILED, so a failed no-leak / internet-unaffected assertion
|
|
# can never leave the gateway tunnel down. This is the ONLY step that stops the
|
|
# tunnel, so restarting it here closes every path to a leaked-down gateway.
|
|
# We create the privileged restart Job with kubectl and `kubectl wait` for it —
|
|
# the equivalent of an `assert succeeded:1`: teardown blocks until the host
|
|
# confirms the unit is back up. Idempotent: `systemctl start` on an already-
|
|
# running unit is a no-op success.
|
|
finally:
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
# Distinct Job name (tunnel-restart) so it never collides with the
|
|
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
|
|
# valid YAML at column 0 at runtime.
|
|
cat <<'JOB' | kubectl -n "$ns" apply -f -
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tunnel-restart
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
backoffLimit: 3
|
|
activeDeadlineSeconds: 90
|
|
ttlSecondsAfterFinished: 600
|
|
template:
|
|
metadata:
|
|
labels: { app: vpn-killswitch-fault }
|
|
spec:
|
|
restartPolicy: Never
|
|
hostPID: true
|
|
nodeSelector:
|
|
homelab.keithjackson.dev/egress-gateway: "true"
|
|
tolerations:
|
|
- operator: Exists
|
|
containers:
|
|
- name: nsenter
|
|
image: busybox:1.37.0
|
|
securityContext:
|
|
privileged: true
|
|
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
|
|
JOB
|
|
# Block teardown until the restart actually completed on the host — a
|
|
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
|
|
# or timeout, which (set -e) fails the step and surfaces the problem.
|
|
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
|
|
echo "TUNNEL_SELF_HEAL_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_SELF_HEAL_OK')): true
|
|
# ---------------------------------------------------------------------------
|
|
- name: when-tunnel-recovers-connectivity-returns
|
|
# The tunnel was already restarted by the previous step's self-healing
|
|
# `finally`; here we prove the EFFECT — vpn-class egress returns to a NON-WAN
|
|
# (PIA) IP within a bounded window (tunnel reconnect + datapath convergence).
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
ns="$NAMESPACE"
|
|
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
|
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
|
vpn=""
|
|
for i in $(seq 1 24); do
|
|
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
|
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
|
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
|
|
vpn=""
|
|
sleep 5
|
|
done
|
|
echo "recovered_vpn=$vpn wan=$wan"
|
|
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
|
|
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
|
|
echo "TUNNEL_RECOVERED_OK"
|
|
check:
|
|
(contains($stdout, 'TUNNEL_RECOVERED_OK')): true
|
|
# Fault Jobs + fixtures are torn down automatically by Chainsaw's per-test
|
|
# namespace deletion (config/chainsaw-config.yaml: skipDelete: false).
|