Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.
What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).
Also recorded:
* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
gateway node is what tells Cilium it changed (the egress-uplink node label). A
future reader's instinct will be to have the node write to that object; the
comment says not to, because it is Flux-managed, and points at the label and
`cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
not have passed before it existed. Its no-leak assertion was always right; only
the explanation was wrong. Also states the prerequisite plainly: the test cannot
run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
escalation is closed as not implementable on Cilium OSS, rather than left open
for someone to attempt.
DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.
Co-Authored-By: Claude Opus 5 <[email protected]>
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.
- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
fails, never returns the WAN IP = no leak) while internet-class is unaffected;
WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
idempotent start Job (self-healing) so a mid-test failure never leaves the
gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
(namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).
Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
My previous commit converted =~ to contains() but kept flow style
'check: { (contains($stdout, 'X')): true }' — the comma inside contains() is a
YAML flow-map separator, so 9 tests failed yamllint ('expected , or }'). Convert
those checks to block style (matching data-survival), where a comma in the key
scalar is fine. Gate PASS; data-survival still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Every chainsaw `check` used the regex operator `=~` (`($stdout =~ 'X'): true`),
which chainsaw's expression engine rejects ('SyntaxError: Unknown char: ~') — so
EVERY such test failed on the assertion even when the behavior passed. Proven with
data-survival (R-STATE-01): the Longhorn canary genuinely SURVIVED pod destruction
(DATA_SURVIVED_OK) but the test FAILED on the `=~` syntax. Converted all 10 tests to
JMESPath `contains($stdout, 'X')` (data-survival's alternation → contains||contains).
data-survival now PASSES; the other 9 (later-phase components) are syntactically fixed
for when their phases deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5