diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 30f11df..c2df2c5 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2089,3 +2089,200 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP** (`placeholder-guard-coupling`, pre-existing), 0 FAIL. + +## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`) + +The operator asked for VPN egress on prod. An earlier assessment this session had recommended +holding off over three suspected defects; the operator decided, so the defects were fixed first +and then the gateway was stood up. All three were settled by **measurement on the dev gateway +node**, not by reading code — and two of the three turned out to be different from what was +suspected, in one case much worse. + +### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes + +`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on +that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex +1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and +the ledger still recorded the gateway as conformance-green. + +**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an +(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit: +*"it does not automatically respond to a change in the gateway node's network configuration… You +can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart +re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.** +`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that +setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that +k8s object events already triggered, and netlink link events are not among them. + +Measured, in order: + +| step | observation | +|---|---| +| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out | +| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately | +| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again | +| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later | +| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers | + +**Fix:** `roles/vpn_egress_gateway` now publishes +`homelab.keithjackson.dev/egress-uplink=-` on the gateway node — from the +OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or +fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence +timer for the boot case. A **node label, not an annotation on the policy**: the policy is +Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are +already this role's territory — and the label's value doubles as the diagnostic. After the fix, +stop->start recovers unaided (map 1989 -> 1990, PIA IP returns). + +**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the +tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified +twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and +remains sound. + +### Defect as reported vs. as measured + +**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps +SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match +neither the ip-rule nor the drop rule and **leave via the WAN**. + +**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing +lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in +place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the +old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on +the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out +tun0 — the redirect to a dead ifindex simply fails and the packet is dropped. + +The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the +down-script already retained it). It cannot leak pod traffic — see the next section for why — but +node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch +reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which +clears rules but leaves a named set's elements intact. + +### The nftables kill-switch cannot see the traffic it was written to guard + +Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table +on the gateway node, while a vpn-class pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any — control, the hook is live | 45166 | +| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 | + +With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF, +which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The +hook works and the matching works — the steered pod packets are simply never presented to it. + +**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a +tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes +because of that, not because of the nft rules. Every comment in the tree that credited the nft +table has been corrected rather than left to mislead the next reader. + +**The table is kept, not deleted**, because it does cover something real — node-originated traffic +sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`: + +- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms, + `killswitch_accept` advances; +- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop` + 0 -> 3 packets / 252 bytes**, 100% loss. + +Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever +counts in normal operation, the datapath has changed. + +Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps +that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule +to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary +mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the +role README's standing escalation ("add the shared fwmark to the interface contract so the +CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there +is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in. + +### Prod stand-up + +`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its +own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of +`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing +the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run +as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` +first (confirmed: exactly `k3s-prod-3`). + +Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries +the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical +ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway +role, and the public egress IP, away from `k3s-prod-3`. + +The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and +copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The +two files are independent from here on; rotating means re-encrypting both, which is correct for two +environments. + +**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said +node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink +in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211` +device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is +assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory +comment is corrected. + +**Result on prod:** + +| | before | after | +|---|---|---| +| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) | +| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged | +| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok | + +Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from +the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going +out. + +### The NFS media mounts are unaffected, verified rather than assumed + +sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and +`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was +checked before and after arming: + +- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node + kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches + it. No `excludedCIDRs` entry is needed for that. +- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing + `192.168.0.0/16` exclusion anyway. +- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway + node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own + `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS + media mount in this repo — only plex, sonarr, radarr and nextcloud do. + +### Left for the operator / not settled here + +1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can + only run where a node carries the gateway label, and its recovery step is only passable with + this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE + cluster, so it was not fired at prod unasked. +2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside + the connection limit, but the two environments are indistinguishable from outside. A separate + prod PIA account is the clean fix if that ever matters. +3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the + ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the + single-gateway choice is now recorded in the inventory itself, not just in prose. +4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but + nothing watches the invariant "the gateway node's `egress-uplink` label matches + `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one, + and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small + node-exporter textfile from the re-steer script would close it. +5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a + commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert + (wireless is gone) and not printed here. Worth scrubbing on all three nodes. +6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class + fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect: + > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress + > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be + > used; this was verified with packet capture on both the WAN and tunnel devices. The host + > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel + > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod + > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark + > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.* + > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a + > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every + > interface change. This is a standing operational requirement of the ADR-0015 design, not an + > implementation detail.* diff --git a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml index e89bc3b..7b317af 100644 --- a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml +++ b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml @@ -10,8 +10,31 @@ # egress-gateway node is STOPPED (fault injection) — # THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS # and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS). -# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0 -# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED). +# vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4: +# vpn fails CLOSED). +# +# MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables +# kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway +# datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device +# without invoking netfilter POSTROUTING, measured with counters on the gateway +# node (0 packets for the pod CIDR and the PIA range in forward and postrouting, +# against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's +# own drop when it cannot use the cached egress interface; tcpdump on both the WAN +# and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and +# still the right one — only the explanation was wrong. +# +# PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node +# carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to +# it by nodeSelector, and with no gateway there is no tunnel to stop). +# +# RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives +# tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own, +# so before 2026-09-12 the recovery step could not pass: measured, the map stayed +# on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway +# now publishes the live uplink as a node label on every connect, which makes +# Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP +# returns). If this test's recovery step ever fails again, check +# `journalctl -u vpn-egress-resteer` on the gateway node first. # AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN). # WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP). # diff --git a/infra/ansible/roles/vpn_egress_gateway/README.md b/infra/ansible/roles/vpn_egress_gateway/README.md index 59992c8..eb34984 100644 --- a/infra/ansible/roles/vpn_egress_gateway/README.md +++ b/infra/ansible/roles/vpn_egress_gateway/README.md @@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below. > once the tunnel is live in dev/test. ## Interface contract (shared with the Cilium-policy agent) -- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`). +- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod` + (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the + flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the + first node in lexical ordering", so a second labelled node does not add failover — it silently + steals the role. - **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the - `CiliumEgressGatewayPolicy` selects the gateway node by it. + `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the + datapath**, which is why the role applies it LAST. - **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it. -- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side - MAY stamp on vpn-class packets for a tunnel-down-independent host backstop. +- **k8s node label** `homelab.keithjackson.dev/egress-uplink=-` — set by this + role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what + makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it + on its own, so without this the gateway is dead after its first reconnect. See "Re-steering". ## How the four ADR-0015 invariants are met @@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff 2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the narrow matches, never break anything). In `postrouting`: - - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out. - - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet - trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even - while the tunnel is down** because the source set is retained on tunnel-down (see below). - - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop. + - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let + tunnel-sourced traffic out. + - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a + tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed + even while the tunnel is down, because the source set is retained on tunnel-down (below). -`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then -re-adds** the current tunnel source on every connect; the down-script **intentionally retains -it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are -never in the set, never carry the mark, and are never targeted — so they always hit + **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a + source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and + moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet + would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252 + bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next + paragraph for what it can and cannot reach. + + **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic + table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward` + and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on + that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0. + Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs + straight to the egress device, so netfilter POSTROUTING is never invoked for them. What + makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is + unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving + either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic + sourced from a tunnel address. + +`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds** +the current tunnel source on every connect and never flushes (it used to flush first, which left +a brief moment where a packet still sourced from the previous address matched neither the +ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and +`internet`-class traffic are never in the set and are never targeted — so they always hit `policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental. +A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush +table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot +empty the set mid-flight. + ### Explicit, edge-independent fail-closed backstop (tunnel-down) Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was @@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in the host nft table — defense-in-depth, independent of edge-router behavior. -This is safe precisely because the **up-script flushes the set and re-adds the (possibly new) -tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained -stale address self-corrects the instant the tunnel comes back. The set remains scoped to -vpn-class only; node and internet-class traffic are never members, so **no invariant is -weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in -table `51820` remains as the second, independent fail-closed layer. +The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live +source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect +and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address +matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just +been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra +element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and +internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4 +upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed +layer. ## Why the node default route is provably unaffected (design-level verification) - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero** @@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer. ip-rules, never `table main` / `table default`. - The ip-rule matches **`from `** — an address the node itself never sources traffic from — so node-originated and internet-class packets never enter the tunnel table. -- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark. +- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`. - Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down. + Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up, + node curl = the WAN IP, k3s `/readyz` = ok. -## Kill-switch layering & ESCALATION (honest limit) -- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the - tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps. -- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when - its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no - valid egress path rather than falling back to the node's default route) **plus** the routing - blackhole. That is the ADR's expected behavior. -- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when - the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class - packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the - *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium - Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for - R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy - marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop. - This is flagged rather than worked around — no spec was weakened to reach done. +## Re-steering — Cilium does not follow tun0, and that is the whole ball game + +Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex) +pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it +does not automatically respond to a change in the gateway node's network configuration... You can +force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`: + +| what | observed | +|---|---| +| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 | +| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) | +| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them | +| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) | +| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately | + +So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix +is `templates/vpn-egress-resteer.sh.j2`, which publishes `-` as the +`egress-uplink` node label: + +- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and + writing to it from a node invites drift/ownership churn; node labels are already this role's + territory, and the label value doubles as the diagnostic (`kubectl get nodes -L + homelab.keithjackson.dev/egress-uplink`); +- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can + never block or fail the tunnel and does not need the kube-API to be up at that instant; +- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where + the API is not up yet and every in-script retry is exhausted; +- **idempotent** — if the label already records the live uplink it exits without writing, so the + timer is free to run often. + +Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map +1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely — +which also means the **recovery half of the `vpn-killswitch` conformance test could not have +passed** on the pre-fix implementation. ## What requires a LIVE tunnel to validate (cannot be tested without operator creds) 1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface. 2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP. 3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak) while internet-class is unaffected — the R-NET-06 fault-injection gate. -4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped - rules actually see the packets (eBPF datapath interaction — see escalation). +4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped + rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does + NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE, + HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended. 5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live). 6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role. ## Files - `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars. -- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label. +- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label + (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering). - `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart. -- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units. -- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`. +- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and + tunnel units, and `vpn-egress-resteer.{sh,service,timer}`. +- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: + `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and + `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod). + +## Running it against prod +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# confirm it says exactly `k3s-prod-3`, then drop --list-hosts +``` +The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply +the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or +kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is +not ready to carry it. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index cbd3e58..907e29a 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015 by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster gateway pod** — the tunnel is host infrastructure, not a Flux workload. -> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class` -> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster. +> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`. +> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev. +> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the +> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale +> egress ifindex — see below), so the test's recovery step could not have passed on the +> implementation as it stood. Whatever produced the earlier green result, re-run it. ## How it works (ADR-0015) Three layers, two owners: @@ -24,9 +28,101 @@ Three layers, two owners: (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel. -- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered - `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so - `internet`-class and node traffic are unaffected. +- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails + closed because Cilium drops the traffic when it cannot use its cached egress interface. The host + nftables table is retained for a narrower job. See "The kill-switch does not do what this file + used to claim" below — that section is the measurement, not an opinion. + +## LIVE ON PROD since 2026-09-12 — `k3s-prod-3` + +| | dev | prod | +|---|---|---| +| gateway node | `k3s-dev-3` | `k3s-prod-3` | +| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | +| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | +| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | + +Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three +nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home +WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was +reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed +`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the +gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also +means nothing had been leaking in the meantime. + +Both environments currently share one PIA account and, as it happens, one public egress IP. +That is inside the account's connection limit, but it means the two environments are not +distinguishable from the outside; a separate prod PIA account is the clean fix if that ever +matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). + +## The kill-switch does not do what this file used to claim (measured 2026-09-12) + +This README, `egress-steering.yaml`, and the role all used to say the **host nftables +kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.** +With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in +eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is +never invoked for those packets. + +A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class +pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 | +| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 | + +**What actually fails closed is Cilium itself.** When it cannot use its cached egress +interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a +tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the +guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance +test passes because of it, not because of the nft rules. + +The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely +does cover: **node-originated** traffic sourced from a tunnel address. That was proven on +`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 +replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out +`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back. +If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read +that as news, not reassurance. + +Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that +mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table +2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is +a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark` +variable is gone with it, and the "add the fwmark to the interface contract" escalation in the +role README is **closed as not-implementable** rather than left open. + +## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help + +Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not +revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes +stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been +dead for ~80 minutes when this work started, with no alert and nothing in the ledger. + +`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that +setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces +reconciliations that k8s object events already triggered — netlink link events are not among them. + +The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=-` on +the gateway node on every connect, plus a 2-minute convergence timer. That node update makes +Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 +within 10s, egress recovered immediately). A node label rather than an annotation on this +Flux-managed policy, so GitOps state is never written to from a node. + +**Debugging a dead gateway starts here:** + +```bash +kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink +kubectl -n kube-system exec -c cilium-agent -- \ + cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0` +``` + +If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the +gateway node is the next stop. ## Resilience — there is NO gateway failover (spec correction, escalated) diff --git a/platform/vpn-egress/egress-steering.yaml b/platform/vpn-egress/egress-steering.yaml index 1cabfa8..bd06f2b 100644 --- a/platform/vpn-egress/egress-steering.yaml +++ b/platform/vpn-egress/egress-steering.yaml @@ -59,16 +59,38 @@ spec: # Excluded flows fall through to Cilium's normal local datapath (invariant 1). destinationCIDRs: - "0.0.0.0/0" + # + # BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media + # mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both + # inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason + # is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted + # by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never + # matches it and no pod-scoped egress steering can touch it. Verified after arming the prod + # gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list + # AND write the export, and the node's own mounts are intact. No RFC1918 destination needs + # adding here for the media path. excludedCIDRs: - "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net - "172.16.0.0/12" # RFC1918 - - "192.168.0.0/16" # RFC1918 (home LAN / node network family) + - "192.168.0.0/16" # RFC1918 (home LAN / node network family / the bulk NFS server) # --- Where it egresses -------------------------------------------------- - # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When - # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible, - # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to - # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED, - # internet fails OPEN). + # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. + # + # CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the + # authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot + # be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the + # skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never + # see this traffic. Measured with a counters-only nft table on the gateway node while a + # vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and + # the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live) + # and 16 for a node-originated curl out tun0 (so the matching works). + # WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used + # it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a + # tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak); + # internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold). + # The host nft table is retained, with counters, for the narrower case it CAN see: + # node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's + # counter moved 0 -> 3 packets when such a packet was routed at the WAN). # # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby. # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD @@ -89,6 +111,22 @@ spec: # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)" # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06. + # + # `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes. + # Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the + # first resolution and does NOT re-resolve on a netlink change. `systemctl restart + # vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class + # egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but + # dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events. + # Upstream: "it does not automatically respond to a change in the gateway node's network + # configuration... You can force a fresh selection by re-applying the Egress Gateway policy." + # The Ansible role closes this: roles/vpn_egress_gateway publishes + # `homelab.keithjackson.dev/egress-uplink=-` on the gateway node on every + # connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this + # policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s. + # DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node + # writing to GitOps state is a drift fight. Check the label, not this file, when debugging: + # kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink egressGateway: nodeSelector: matchLabels: