docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
60e3f09c67
commit
04606d48b2
5 files changed
+479
-53
No files matched your search
@@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015
|
||||
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
|
||||
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
|
||||
|
||||
> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
|
||||
> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
|
||||
> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
|
||||
> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
|
||||
> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
|
||||
> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
|
||||
> egress ifindex — see below), so the test's recovery step could not have passed on the
|
||||
> implementation as it stood. Whatever produced the earlier green result, re-run it.
|
||||
|
||||
## How it works (ADR-0015)
|
||||
Three layers, two owners:
|
||||
@@ -24,9 +28,101 @@ Three layers, two owners:
|
||||
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
|
||||
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
|
||||
datapath; only public destinations leave via the tunnel.
|
||||
- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
|
||||
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
|
||||
`internet`-class and node traffic are unaffected.
|
||||
- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
|
||||
closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
|
||||
nftables table is retained for a narrower job. See "The kill-switch does not do what this file
|
||||
used to claim" below — that section is the measurement, not an opinion.
|
||||
|
||||
## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
|
||||
|
||||
| | dev | prod |
|
||||
|---|---|---|
|
||||
| gateway node | `k3s-dev-3` | `k3s-prod-3` |
|
||||
| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
|
||||
| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
|
||||
| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
|
||||
|
||||
Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
|
||||
nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
|
||||
WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
|
||||
reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
|
||||
`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
|
||||
gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
|
||||
means nothing had been leaking in the meantime.
|
||||
|
||||
Both environments currently share one PIA account and, as it happens, one public egress IP.
|
||||
That is inside the account's connection limit, but it means the two environments are not
|
||||
distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
|
||||
matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
|
||||
|
||||
## The kill-switch does not do what this file used to claim (measured 2026-09-12)
|
||||
|
||||
This README, `egress-steering.yaml`, and the role all used to say the **host nftables
|
||||
kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
|
||||
With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
|
||||
eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
|
||||
never invoked for those packets.
|
||||
|
||||
A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
|
||||
pod egressed:
|
||||
|
||||
| counter | packets |
|
||||
|---|---|
|
||||
| `forward`, saddr = pod CIDR | **0** |
|
||||
| `forward`, saddr = PIA tunnel range | **0** |
|
||||
| `postrouting`, oif tun0, saddr = tunnel range | **0** |
|
||||
| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
|
||||
| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
|
||||
| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
|
||||
|
||||
**What actually fails closed is Cilium itself.** When it cannot use its cached egress
|
||||
interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
|
||||
tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
|
||||
guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
|
||||
test passes because of it, not because of the nft rules.
|
||||
|
||||
The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
|
||||
does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
|
||||
`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
|
||||
replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
|
||||
`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
|
||||
If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
|
||||
that as news, not reassurance.
|
||||
|
||||
Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
|
||||
mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
|
||||
2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
|
||||
a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
|
||||
variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
|
||||
role README is **closed as not-implementable** rather than left open.
|
||||
|
||||
## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
|
||||
|
||||
Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
|
||||
revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
|
||||
stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
|
||||
dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
|
||||
|
||||
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
|
||||
setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
|
||||
reconciliations that k8s object events already triggered — netlink link events are not among them.
|
||||
|
||||
The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
|
||||
the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
|
||||
Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
|
||||
within 10s, egress recovered immediately). A node label rather than an annotation on this
|
||||
Flux-managed policy, so GitOps state is never written to from a node.
|
||||
|
||||
**Debugging a dead gateway starts here:**
|
||||
|
||||
```bash
|
||||
kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
|
||||
kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0`
|
||||
```
|
||||
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
|
||||
Reference in new issue
Block a user