Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

docs(vpn): stop crediting the nftables kill-switch for a drop it never makes

Browse Source 
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 17:57:52 -05:00
1 parent 60e3f09c67
commit 04606d48b2
5 files changed
+479 -53
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/ansible/roles/vpn_egress_gateway/README.md
+112 -40
View File
@@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below.
> once the tunnel is live in dev/test.
## Interface contract (shared with the Cilium-policy agent)
- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`).
- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod`
(prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the
flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the
first node in lexical ordering", so a second labelled node does not add failover — it silently
steals the role.
- **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
`CiliumEgressGatewayPolicy` selects the gateway node by it.
`CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the
datapath**, which is why the role applies it LAST.
- **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side
MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.
- **k8s node label** `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` — set by this
role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what
makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it
on its own, so without this the gateway is dead after its first reconnect. See "Re-steering".
## How the four ADR-0015 invariants are met
@@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff
2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
narrow matches, never break anything). In `postrouting`:
- `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
- `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even
while the tunnel is down** because the source set is retained on tunnel-down (see below).
- `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.
- `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let
tunnel-sourced traffic out.
- `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a
tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed
even while the tunnel is down, because the source set is retained on tunnel-down (below).
`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then
re-adds** the current tunnel source on every connect; the down-script **intentionally retains
it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are
never in the set, never carry the mark, and are never targeted — so they always hit
**Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a
source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and
moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet
would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252
bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next
paragraph for what it can and cannot reach.
**SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic
table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward`
and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on
that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0.
Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs
straight to the egress device, so netfilter POSTROUTING is never invoked for them. What
makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is
unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving
either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic
sourced from a tunnel address.
`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds**
the current tunnel source on every connect and never flushes (it used to flush first, which left
a brief moment where a packet still sourced from the previous address matched neither the
ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and
`internet`-class traffic are never in the set and are never targeted — so they always hit
`policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
directions are structural, not incidental.
A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush
table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot
empty the set mid-flight.
### Explicit, edge-independent fail-closed backstop (tunnel-down)
Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
`ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
@@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s
vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
the host nft table — defense-in-depth, independent of edge-router behavior.
This is safe precisely because the **up-script flushes the set and re-adds the (possibly new)
tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained
stale address self-corrects the instant the tunnel comes back. The set remains scoped to
vpn-class only; node and internet-class traffic are never members, so **no invariant is
weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in
table `51820` remains as the second, independent fail-closed layer.
The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live
source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect
and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address
matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just
been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra
element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and
internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4
upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed
layer.
## Why the node default route is provably unaffected (design-level verification)
- OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
@@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer.
ip-rules, never `table main` / `table default`.
- The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
from — so node-originated and internet-class packets never enter the tunnel table.
- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark.
- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`.
- Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
`curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.
Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up,
node curl = the WAN IP, k3s `/readyz` = ok.
## Kill-switch layering & ESCALATION (honest limit)
- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the
tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when
its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no
valid egress path rather than falling back to the node's default route) **plus** the routing
blackhole. That is the ADR's expected behavior.
- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when
the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class
packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the
*Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium
Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for
R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy
marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop.
This is flagged rather than worked around — no spec was weakened to reach done.
## Re-steering — Cilium does not follow tun0, and that is the whole ball game
Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex)
pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it
does not automatically respond to a change in the gateway node's network configuration... You can
force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`:
| what | observed |
|---|---|
| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 |
| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) |
| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them |
| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) |
| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately |
So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix
is `templates/vpn-egress-resteer.sh.j2`, which publishes `<tun0 addr>-<ifindex>` as the
`egress-uplink` node label:
- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and
writing to it from a node invites drift/ownership churn; node labels are already this role's
territory, and the label value doubles as the diagnostic (`kubectl get nodes -L
homelab.keithjackson.dev/egress-uplink`);
- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can
never block or fail the tunnel and does not need the kube-API to be up at that instant;
- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where
the API is not up yet and every in-script retry is exhausted;
- **idempotent** — if the label already records the live uplink it exits without writing, so the
timer is free to run often.
Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map
1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely —
which also means the **recovery half of the `vpn-killswitch` conformance test could not have
passed** on the pre-fix implementation.
## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
while internet-class is unaffected — the R-NET-06 fault-injection gate.
4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
rules actually see the packets (eBPF datapath interaction — see escalation).
4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does
NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE,
HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended.
5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.
## Files
- `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label.
- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label
(kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering).
- `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units.
- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`.
- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and
tunnel units, and `vpn-egress-resteer.{sh,service,timer}`.
- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys:
`inventory/group_vars/vpn_egress_gateway/README.md` (dev) and
`inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod).
## Running it against prod
```bash
cd infra/ansible
ansible-playbook playbooks/vpn.egress.gateway.yml \
-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
# confirm it says exactly `k3s-prod-3`, then drop --list-hosts
```
The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply
the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or
kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is
not ready to carry it.