diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 30f11df..b20b43f 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2089,3 +2089,245 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP** (`placeholder-guard-coupling`, pre-existing), 0 FAIL. + +## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`) + +The operator asked for VPN egress on prod. An earlier assessment this session had recommended +holding off over three suspected defects; the operator decided, so the defects were fixed first +and then the gateway was stood up. All three were settled by **measurement on the dev gateway +node**, not by reading code — and two of the three turned out to be different from what was +suspected, in one case much worse. + +### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes + +`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on +that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex +1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and +the ledger still recorded the gateway as conformance-green. + +**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an +(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit: +*"it does not automatically respond to a change in the gateway node's network configuration… You +can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart +re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.** +`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that +setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that +k8s object events already triggered, and netlink link events are not among them. + +Measured, in order: + +| step | observation | +|---|---| +| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out | +| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately | +| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again | +| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later | +| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers | + +**Fix:** `roles/vpn_egress_gateway` now publishes +`homelab.keithjackson.dev/egress-uplink=-` on the gateway node — from the +OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or +fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence +timer for the boot case. A **node label, not an annotation on the policy**: the policy is +Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are +already this role's territory — and the label's value doubles as the diagnostic. After the fix, +stop->start recovers unaided (map 1989 -> 1990, PIA IP returns). + +**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the +tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified +twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and +remains sound. + +### Defect as reported vs. as measured + +**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps +SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match +neither the ip-rule nor the drop rule and **leave via the WAN**. + +**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing +lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in +place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the +old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on +the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out +tun0 — the redirect to a dead ifindex simply fails and the packet is dropped. + +The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the +down-script already retained it). It cannot leak pod traffic — see the next section for why — but +node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch +reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which +clears rules but leaves a named set's elements intact. + +### The nftables kill-switch cannot see the traffic it was written to guard + +Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table +on the gateway node, while a vpn-class pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any — control, the hook is live | 45166 | +| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 | + +With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF, +which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The +hook works and the matching works — the steered pod packets are simply never presented to it. + +**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a +tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes +because of that, not because of the nft rules. Every comment in the tree that credited the nft +table has been corrected rather than left to mislead the next reader. + +**The table is kept, not deleted**, because it does cover something real — node-originated traffic +sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`: + +- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms, + `killswitch_accept` advances; +- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop` + 0 -> 3 packets / 252 bytes**, 100% loss. + +Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever +counts in normal operation, the datapath has changed. + +Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps +that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule +to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary +mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the +role README's standing escalation ("add the shared fwmark to the interface contract so the +CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there +is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in. + +### Prod stand-up + +`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its +own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of +`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing +the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run +as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` +first (confirmed: exactly `k3s-prod-3`). + +Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries +the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical +ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway +role, and the public egress IP, away from `k3s-prod-3`. + +The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and +copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The +two files are independent from here on; rotating means re-encrypting both, which is correct for two +environments. + +**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said +node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink +in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211` +device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is +assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory +comment is corrected. + +**Result on prod:** + +| | before | after | +|---|---|---| +| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) | +| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged | +| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok | + +Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from +the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going +out. + +### The NFS media mounts are unaffected, verified rather than assumed + +sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and +`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was +checked before and after arming: + +- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node + kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches + it. No `excludedCIDRs` entry is needed for that. +- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing + `192.168.0.0/16` exclusion anyway. +- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway + node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own + `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS + media mount in this repo — only plex, sonarr, radarr and nextcloud do. + +### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways + +Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account, +well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled +it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips +`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05, +22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute +mutual-eviction cycle with no end. Prod's first probes looked green only because they landed +inside an up-window; that is a good reminder that a single green probe is not a stability check. + +**Fix:** a new role var `vpn_egress_gateway_pia_remote` (" ", empty = honour the +.ovpn) rewrites only the `remote` line of the deployed .ovpn, and +`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another +member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to +`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and +the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is +why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server +address is not a credential, so it sits in plain group_vars next to the vault, reviewable. + +**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that +re-pins, must take an unused address from the pool. + +**Dev recovered the moment prod left the shared server.** Dev's last `Inactivity timeout` is +23:05:06 UTC — the minute prod switched to `.135`. Re-checked at 23:12:03: the 23:05:07 session is +still the live one, seven minutes unbroken against a previous cycle of two, and a source-bound ping +through the tunnel returns 4/4 at 146 ms with 0% loss. vpn-class pod egress from dev is back to +`158.173.241.36` on three consecutive probes. + +**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to +`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) -> +`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what +defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium +re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual +step, no CEGP edit, no operator. Before this branch that address change would have left prod's +vpn-class egress dead until someone noticed. + +### Left for the operator / not settled here + +1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can + only run where a node carries the gateway label, and its recovery step is only passable with + this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE + cluster, so it was not fired at prod unasked. +2. **Decide whether prod gets its own PIA account.** After the server split the two environments do + have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is + solved — but they still share one account, and the per-environment PIA server is now a + hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with + dev's. A separate prod account removes that coupling permanently. +3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the + ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the + single-gateway choice is now recorded in the inventory itself, not just in prose. +4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but + nothing watches the invariant "the gateway node's `egress-uplink` label matches + `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one, + and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small + node-exporter textfile from the re-steer script would close it. +5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a + commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert + (wireless is gone) and not printed here. Worth scrubbing on all three nodes. +6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class + fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect: + > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress + > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be + > used; this was verified with packet capture on both the WAN and tunnel devices. The host + > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel + > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod + > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark + > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.* + > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a + > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every + > interface change. This is a standing operational requirement of the ADR-0015 design, not an + > implementation detail.* diff --git a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml index e89bc3b..7b317af 100644 --- a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml +++ b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml @@ -10,8 +10,31 @@ # egress-gateway node is STOPPED (fault injection) — # THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS # and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS). -# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0 -# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED). +# vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4: +# vpn fails CLOSED). +# +# MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables +# kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway +# datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device +# without invoking netfilter POSTROUTING, measured with counters on the gateway +# node (0 packets for the pod CIDR and the PIA range in forward and postrouting, +# against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's +# own drop when it cannot use the cached egress interface; tcpdump on both the WAN +# and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and +# still the right one — only the explanation was wrong. +# +# PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node +# carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to +# it by nodeSelector, and with no gateway there is no tunnel to stop). +# +# RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives +# tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own, +# so before 2026-09-12 the recovery step could not pass: measured, the map stayed +# on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway +# now publishes the live uplink as a node label on every connect, which makes +# Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP +# returns). If this test's recovery step ever fails again, check +# `journalctl -u vpn-egress-resteer` on the gateway node first. # AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN). # WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP). # diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md new file mode 100644 index 0000000..f111110 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md @@ -0,0 +1,49 @@ +# group_vars for the `vpn_egress_gateway_prod` node — ADR-0015 host tunnel secrets (PROD) + +Prod's egress gateway is `k3s-prod-3`, in its **own** inventory group +(`[vpn_egress_gateway_prod]`) so that the flagless +`ansible-playbook playbooks/vpn.egress.gateway.yml` — which defaults to the **dev** group — +can never touch a prod node. Prod is reached only by asking for it: + +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# then drop --list-hosts +``` + +A separate group needs its own `group_vars`, because the dev group's vault is attached to the +dev group and is not visible to a prod host. The required keys are **identical** to dev's — +see `../vpn_egress_gateway/README.md` for what each one is: + +| Vault key | +|---| +| `vault_vpn_egress_gateway_pia_username` | +| `vault_vpn_egress_gateway_pia_password` | +| `vault_vpn_egress_gateway_pia_ovpn` | + +## How this file was created (2026-09-12) + +`vault.yml` here is a **verbatim copy of the dev group's encrypted blob**. That is deliberate: + +- the operator confirmed the **same PIA account** applies to prod and is well inside its + connection limit, so the values are the same values; +- copying the *ciphertext* means the plaintext was never decrypted to disk, never passed + through a shell, and never appeared in an agent transcript. The repo-wide vault passphrase + (`ansible.cfg` → `vault_password_file`) decrypts both files identically. + +The two files are now **independent**: rotating the PIA credentials means re-encrypting +*both*, and that is the intended behaviour — prod and dev are separate environments and a +credential change should be a deliberate act per environment, not an accident of sharing a +file. If prod is ever moved to its own PIA account (recommended if the connection limit ever +bites, or to keep the public egress IPs of the two environments independent), re-create this +file alone: + +```bash +cd infra/ansible +ansible-vault create inventory/group_vars/vpn_egress_gateway_prod/vault.yml +``` + +## Notes +- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`). +- **Never** run `ansible-inventory --list`: it prints DECRYPTED vault values. diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml new file mode 100644 index 0000000..19e2ed4 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml @@ -0,0 +1,22 @@ +--- +# Non-secret group_vars for the PROD egress gateway (k3s-prod-3). Credentials live in the +# vaulted vault.yml beside this file. + +# PROD DIALS ITS OWN PIA SERVER — and it has to. +# +# The vaulted .ovpn pins ONE server, 158.173.241.112. Measured 2026-09-12: with dev +# (k3s-dev-3) and prod (k3s-prod-3) both dialling it, PIA handed BOTH sessions the SAME +# tunnel address — `Server-10779-2a ... ifconfig 10.72.192.7` in both journals. Each +# reconnect steals that address from the other; the loser then sees no data for 60s, hits +# `--ping-restart`, reconnects and steals it back. Both tunnels flapped on a ~2-minute cycle +# (dev 22:57 / 22:59 / 23:01 UTC, prod interleaved a minute off) and neither environment had +# reliable vpn-class egress. This is NOT the account's connection limit — that has room. +# +# 158.173.241.135 is another member of the same PIA Stockholm LB pool: `sweden.privacy.network` +# resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239}. Any server in the pool accepts +# the same credentials and the same inline CA, so only the address differs. Dev keeps the +# .ovpn's pinned .112 untouched, which is why the override lives HERE and not in the role. +# +# A public server address is not a credential, so this is plain group_vars, not vault. +# If prod is ever moved to its own PIA account, this can go away again. +vpn_egress_gateway_pia_remote: "158.173.241.135 1198" diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml new file mode 100644 index 0000000..1948dd7 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml @@ -0,0 +1,131 @@ +$ANSIBLE_VAULT;1.1;AES256 +38306636313935333030303664653739316233663139616563663233643262373637306331653264 +6538363539383464353964663632633561323562383038340a396233633232356137313435336231 +36333663363130643532613566643339373365626462313537653033306363336564343139373537 +3164323463616632630a373732333536613936366463313439343664363530383664636535616261 +31306332666666643637346364386537343062383033663530346163373536323639373366333661 +36303831323662333232633034393362333561643637623033656565303461336465333664366332 +36336466656437343431363735646661643837666335633464666130643866313830366465323166 +65616537343630303431623537303766383762646533616539613235343938656237613934623565 +35373835633539653333343232626336326638303961373230663561313330313632393066323130 +31383838386262393730666231376362383132646230383964313439663935613861323137646331 +36626365323566316436386662383434366664356166636431663332393665323734623735303239 +62383031613237333136613837356166343436306137353935613661613264326466346337346530 +65356630373235313131656131383965313366613965386662343961653864373964346337363562 +66626635363438656430306231323764346564666132633966353861643262353131623134656137 +39306134626163346336613732326264396438623630656534396565343763303761333538316661 +30623466366232323531633262623434643366613833353564386163303966316537623934336163 +66323933653036366535663266666531666236646431383066616262353733316165666434616534 +35386633313464363233326663363634626239616364383866653135386435636130316335666430 +38323363386462373862366230393764386338643466363161333733653065383966663935303964 +64333461333437653433373131343736333831313236306262373631353461356237386163623166 +30663761306534316439356365396663336135333537663366353866336536663934646133333332 +65353236366435326630653830346631643236326337626435333236386634333137623138316131 +61323365303461353932626336323534633533646233653135396665373765666337333934356662 +36646339376564366439646137306364623165626263383634326337333566663236346366366334 +37316166393763663139336538663266333736623661383062383732303133326233313232323836 +34356130396131346533633336373736663366366231343162633064343565613937366434613664 +31656365303439643837323135613936646133306163663265373933633231616135353335376435 +61633965356538613636663564346566313261306235313438316639626131303566363365663764 +64316562383361373663653034393333653831666239363037343636666364666536343534613066 +61373335353464633963623161353866333038643062303931653134626334393864663461653531 +65373766336437376435616138666563623337373038623566666136626637633463323965666139 +38336139613635383066336164373061663266326235363731663565613336623936373531383966 +62323330636263306665343462666639643530336633346531636263616436356631343763343432 +61636264316566663933346163373163653734623031393962343635653765653964633137346433 +62616138333536336438363564356636316531346566663066376234356230626165386165336165 +38656166633839656438323966376561306338623237303530376231393663663661343362653664 +37613433653634363534306432383061373165613266386562323632636537373736656137633639 +33666634316465383066333737616636363866626330313232326434666164316439333433633461 +35336161313039646264613835613336326332326464313339636532653837653430393939633462 +66656563626163376134623437633332633535336233656237303962623063363134616431623235 +62323265666136323366343565656236646430393237356333613836343438353164313535616138 +34613134663465333734663337303663316161313439626332326334363835376361366463363631 +30396336303761383465373439313666656237356433303532373162306166356165316164656334 +37326163393230306161653332323462333238313166373134343839323461363936353737613662 +61643938333464366433306363633531383264613030316631303034376131323135313136316461 +31666632393631376133373533616639396361633762636663323463363433626537656333653837 +38633632623139356630616637663331616461383361383733613563383165363139343035383333 +32653563303038336137336333653561363037633634323662616666356534323138353438616463 +64383639353733333537616632346230383330356166663433373734386435376563333038333934 +37616536333132303166366235613265336432616239393730353036616337386463363330306532 +65613066363434346562633630336135313736393137656230306163383730356133326561333431 +35393466323936636562616336353934626262336363306335636331666633623432343134396334 +37343765646134306638313636326531616665323136643264353666323236343239633639356537 +61343037666232386135303363363938353864396461363830373633363730323165303165633430 +32656532633761316132613463616365353066393165353331646665613239333738653861363134 +30343939333633366532656430303133643337356336336638653266653766353637356561663636 +61626632333438313436323465643434666334393034366166343930323661386664623663396435 +61633334656464623330623333346165363563356363656663373335646433343762663964323735 +34316239663164623431353164306235386661666632383730376139626666313935663462326433 +36313039326635396435646466306366343438656334346231393066326638313337373933393231 +35373731336164396165303935316366343565363631353137393333613363656564343063373933 +61643033373839656532373330666436396632626663353436356231323166326365323832346461 +31366630626635323465333238613734313265356231313338356431393338353166633939383334 +64333634373638383937663933336563663132366534643434373032333934383533383063643036 +39323534326139633931623831656565333930323362366539346239646464353536346639343436 +37663337396565363065343532616437333666316266633632653661396535636461653335363538 +37303263383030646233313239643536326434323933616135373835393965346462373739383230 +33383863353761346232643835633139363231663162353937343363653339393838616535393436 +38303737633231626236316336353832306232633032653032633365386563626434646139633063 +31313436343063643766663465626330643366313663633733663762653631633966383736336138 +36353738383865613537366364666665326464626365643032666630643039326538663531353964 +33646561303435666237303763663634613136633662633536303066363764383039373236666135 +30626639313531353134353730323334623631396463323035313733333965623665333466366164 +62646539393966626337333637646438313635373666306538356631396237323739613032353832 +31383439343531373561353038363535353266333265336434303934343239633334383865333436 +31616537666332373332623932623162333138373563363336303735396431396464613364643162 +61323737643134653763373232383134643931343366353437633632326138353237336333616238 +32613239316666343761363863376563613638366432313461643535353731383631633233366234 +39663632613062656337616561636533633264373930376437373132303830313863303734346632 +33313431336661663634663861643364363834626638666565383635343433323961623765633265 +34386631653833653865383865636364653265363162323462376364313964626633313635313765 +38633131653836396463393136333932346532333631626461373739346139306465653532366632 +65363333633964616632613830386566623865356131646338653236373861346430363164393634 +64373434663531643436616664393936396564666561313862636133306537373137636234626232 +61313861626235636235353431353738303161373939373966383139323833346662373731646434 +31356133623331323661666234613466366130336662653764363738626531346531333339396266 +32636564653732396339383533343266633730666434303161626362366134353361643362626533 +32333132656636333736616661323664316234393937313666653762316463306437633163383030 +65383962346233663061356338306164343363323736646165613730373166333533373635373039 +39343262393031626637623030373535663539613361613366333863336531663637323035376564 +30626232623661623065643136353237316333323764663633643132653264303162366134616564 +31366665313830323263373835613964386132333538396165663538653865356236646234373465 +30383533316532363133323465363063373962353763346366383563613862336465366565653637 +62346365373030383135323933366138656463323864663363626330653036623739343035336135 +62643262336436336462386430363430363139396566343963373664373334366430643830313439 +66613362396137313661666635643962346662376137393066653765636330366462633565346663 +38633864366666316534653665303664393863633464373638393463343537386437383039373530 +66623063653536666364303531373931613433383139376664633934376662336231643762353830 +31653065303430393864383532333766613965383831616563653233313338343861663334373465 +39363361313532333034613034666265386430333466396163303263636133316263313538656634 +65313464316463623134356562366261663635323530393137313233316638643364646639343935 +38313131633132636463323231393332363766653935323965373134653336376131636163376566 +33623934346431313433386430656264316566383436663535343532303763316437303436336465 +39626231303931303130643237383663666338313662663231313935643334393362323130356530 +39656330616138373962633732313436666662616165316236323930613134643161613464636536 +36613735636233616164396630333634333935393938623061313963636461316564313662353034 +33666434326165643964303561303062323061633766353761313464376262333361323532616361 +39313832396134643966353239333066346239626534303830396165396530623135333834313439 +37373661643731623935666437313465633565303232366638393837303462613035393031333061 +39666433373662633730363665313666346165366138306636623263633537346631666132306238 +34393363373335663631393135646231313064383837396164613536643533383464313536663561 +38633131303966356534636539333733336464393236623238373766616537613437343232303437 +64323636613862663135323034636239333433333165313239336239666665616631623934316637 +61303237336334316265643964663830336539353164343063636664633430326139356534313036 +65393464653330333365336335363531303536313439363066376438666133376664346336633937 +66346165666337666134333963613662303534656339373731663534656533316338353763643930 +38313239313637326435323463333863663433623366316532333065396635663966343835636630 +34643631613431333431636330623564663662383063353138656436336333333262306633666163 +38303032316133666533393663343366333162626436333762306263616362643936383637653330 +31643334613766636231333461643863333838343961383762316162636634323134313461313962 +35363062663934336632383931333262643534336563376537343265633434336439623131613337 +32626461643263623664353639613361393735663266373930366134633534343466346461313032 +34363162343464353037616637316636393139343563303331376362653238346631333137393164 +30306636366636366562656361346637356364666136623131316138393262633963356138383232 +64383932623831376632336331343333393739613663613763303035643136653965386435366663 +35313161383236356563653931313933633737646363363339336639613961343934373637643034 +63333036343532323166366132656538643331303131393737656338323939396461393764393134 +33663539633930636265393363326638636130313036333536306362663633616563316465616330 +35353539303335336565373962393433333539613439353765643061383132386133396339616137 +3265326634643062346666356263326639373130333362333064 diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 1d0f330..4b419ce 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -45,6 +45,34 @@ localhost ansible_connection=local [vpn_egress_gateway] k3s-dev-3 +# PROD's egress-gateway node — a SEPARATE group on purpose (2026-09-12). +# +# It is NOT a member of [vpn_egress_gateway] above, and k3s-prod-3 must never be added +# there: that group is the DEFAULT target of playbooks/vpn.egress.gateway.yml, so a +# flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` would then silently +# reconfigure a prod node's kill-switch datapath. Prod is reached only by asking for it: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 +# +# (`--list-hosts` first, every time. The play's target var is +# `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`.) +# +# WHY NODE 3, AND WHY EXACTLY ONE: +# * node 1's SSD was replaced 2026-09-11 and it has the least operational history; +# * node 2 carries the §0b memattr history; +# * node 3 has a single uplink (eno1 192.168.2.12 — its wireless is gone: no wl* netdev +# exists, the /etc/network/interfaces wlp1s0 stanza is commented out, and the +# 192.168.2.195 address the earlier note below claimed is NOT present. Verified +# 2026-09-12) which is what a kill-switch datapath wants. +# * LEXICAL ORDERING HAZARD: Cilium OSS picks "the first node in lexical ordering" when +# several match the policy's nodeSelector, so labelling k3s-prod-1 would SILENTLY take +# the gateway role from k3s-prod-3. Exactly one host here, forever. +# Its PIA credentials are its own vaulted group_vars/vpn_egress_gateway_prod/vault.yml — +# the dev group's vault does not apply to a prod host. +[vpn_egress_gateway_prod] +k3s-prod-3 + # The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml, # which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY # over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so @@ -116,8 +144,13 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 # needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA # SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now # three-way, not a node-1 outlier. -# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately -# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2. +# NOTE (CORRECTED 2026-09-12): this used to read "node 3 also has wlp1s0 up on +# 192.168.2.195 (wifi)". It does not — re-verified before making node 3 the VPN egress +# gateway, because a second live uplink in a kill-switch datapath is not a detail: there is +# NO wl* netdev on the node at all, no phy80211 device, the wlp1s0 stanza in +# /etc/network/interfaces is commented out, 192.168.2.195 is assigned nowhere, and the only +# IPv4 addresses are lo and eno1 192.168.2.12. `ansible_host` is that WIRED eno1 address — +# the same directly-routable 192.168.2.x LAN as nodes 1 and 2. k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] diff --git a/infra/ansible/playbooks/vpn.egress.gateway.yml b/infra/ansible/playbooks/vpn.egress.gateway.yml index 92d4a79..751e0c6 100644 --- a/infra/ansible/playbooks/vpn.egress.gateway.yml +++ b/infra/ansible/playbooks/vpn.egress.gateway.yml @@ -11,13 +11,25 @@ # vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn # Run: ansible-playbook playbooks/vpn.egress.gateway.yml # -# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod -# has NOT decided whether it runs an egress gateway at all, nor on which node -# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress -# group in the inventory yet. When there is, point this at it with -# -e vpn_egress_gateway_hosts= -# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to -# the dev group. +# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. PROD has +# its own group as of 2026-09-12 — `vpn_egress_gateway_prod` (k3s-prod-3), with its own +# vaulted credentials in group_vars/vpn_egress_gateway_prod/: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# +# The var is `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`. Prod is +# deliberately NOT a member of the default group, so a flagless run cannot reach it — and +# `--list-hosts` before every run is the habit that catches a mistyped flag, which in this +# repo has silently targeted all three dev nodes before. +# +# THE LABEL IS WHAT ARMS THE DATAPATH. The CiliumEgressGatewayPolicy is already reconciled +# on both clusters, so the instant a node carries +# `homelab.keithjackson.dev/egress-gateway=true` steering goes live. The role sequences +# kill-switch -> tunnel -> re-steer -> label, with the label LAST, on purpose. Never apply +# that label by hand, and never run this play with tags that skip the tunnel or kill-switch +# tasks. Exactly ONE node per cluster may carry it (Cilium OSS picks the lexically first +# match, so a labelled k3s-prod-1 would silently steal the role from k3s-prod-3). - name: Configure the node-level VPN egress gateway hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}" become: true diff --git a/infra/ansible/roles/vpn_egress_gateway/README.md b/infra/ansible/roles/vpn_egress_gateway/README.md index 59992c8..eb34984 100644 --- a/infra/ansible/roles/vpn_egress_gateway/README.md +++ b/infra/ansible/roles/vpn_egress_gateway/README.md @@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below. > once the tunnel is live in dev/test. ## Interface contract (shared with the Cilium-policy agent) -- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`). +- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod` + (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the + flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the + first node in lexical ordering", so a second labelled node does not add failover — it silently + steals the role. - **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the - `CiliumEgressGatewayPolicy` selects the gateway node by it. + `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the + datapath**, which is why the role applies it LAST. - **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it. -- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side - MAY stamp on vpn-class packets for a tunnel-down-independent host backstop. +- **k8s node label** `homelab.keithjackson.dev/egress-uplink=-` — set by this + role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what + makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it + on its own, so without this the gateway is dead after its first reconnect. See "Re-steering". ## How the four ADR-0015 invariants are met @@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff 2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the narrow matches, never break anything). In `postrouting`: - - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out. - - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet - trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even - while the tunnel is down** because the source set is retained on tunnel-down (see below). - - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop. + - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let + tunnel-sourced traffic out. + - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a + tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed + even while the tunnel is down, because the source set is retained on tunnel-down (below). -`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then -re-adds** the current tunnel source on every connect; the down-script **intentionally retains -it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are -never in the set, never carry the mark, and are never targeted — so they always hit + **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a + source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and + moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet + would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252 + bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next + paragraph for what it can and cannot reach. + + **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic + table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward` + and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on + that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0. + Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs + straight to the egress device, so netfilter POSTROUTING is never invoked for them. What + makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is + unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving + either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic + sourced from a tunnel address. + +`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds** +the current tunnel source on every connect and never flushes (it used to flush first, which left +a brief moment where a packet still sourced from the previous address matched neither the +ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and +`internet`-class traffic are never in the set and are never targeted — so they always hit `policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental. +A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush +table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot +empty the set mid-flight. + ### Explicit, edge-independent fail-closed backstop (tunnel-down) Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was @@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in the host nft table — defense-in-depth, independent of edge-router behavior. -This is safe precisely because the **up-script flushes the set and re-adds the (possibly new) -tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained -stale address self-corrects the instant the tunnel comes back. The set remains scoped to -vpn-class only; node and internet-class traffic are never members, so **no invariant is -weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in -table `51820` remains as the second, independent fail-closed layer. +The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live +source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect +and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address +matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just +been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra +element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and +internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4 +upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed +layer. ## Why the node default route is provably unaffected (design-level verification) - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero** @@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer. ip-rules, never `table main` / `table default`. - The ip-rule matches **`from `** — an address the node itself never sources traffic from — so node-originated and internet-class packets never enter the tunnel table. -- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark. +- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`. - Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down. + Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up, + node curl = the WAN IP, k3s `/readyz` = ok. -## Kill-switch layering & ESCALATION (honest limit) -- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the - tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps. -- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when - its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no - valid egress path rather than falling back to the node's default route) **plus** the routing - blackhole. That is the ADR's expected behavior. -- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when - the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class - packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the - *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium - Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for - R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy - marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop. - This is flagged rather than worked around — no spec was weakened to reach done. +## Re-steering — Cilium does not follow tun0, and that is the whole ball game + +Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex) +pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it +does not automatically respond to a change in the gateway node's network configuration... You can +force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`: + +| what | observed | +|---|---| +| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 | +| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) | +| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them | +| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) | +| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately | + +So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix +is `templates/vpn-egress-resteer.sh.j2`, which publishes `-` as the +`egress-uplink` node label: + +- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and + writing to it from a node invites drift/ownership churn; node labels are already this role's + territory, and the label value doubles as the diagnostic (`kubectl get nodes -L + homelab.keithjackson.dev/egress-uplink`); +- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can + never block or fail the tunnel and does not need the kube-API to be up at that instant; +- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where + the API is not up yet and every in-script retry is exhausted; +- **idempotent** — if the label already records the live uplink it exits without writing, so the + timer is free to run often. + +Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map +1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely — +which also means the **recovery half of the `vpn-killswitch` conformance test could not have +passed** on the pre-fix implementation. ## What requires a LIVE tunnel to validate (cannot be tested without operator creds) 1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface. 2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP. 3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak) while internet-class is unaffected — the R-NET-06 fault-injection gate. -4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped - rules actually see the packets (eBPF datapath interaction — see escalation). +4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped + rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does + NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE, + HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended. 5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live). 6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role. ## Files - `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars. -- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label. +- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label + (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering). - `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart. -- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units. -- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`. +- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and + tunnel units, and `vpn-egress-resteer.{sh,service,timer}`. +- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: + `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and + `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod). + +## Running it against prod +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# confirm it says exactly `k3s-prod-3`, then drop --list-hosts +``` +The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply +the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or +kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is +not ready to carry it. diff --git a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml index 8e82ef0..817ad04 100644 --- a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml @@ -20,6 +20,23 @@ vpn_egress_gateway_pia_password: "{{ vault_vpn_egress_gateway_pia_password }}" # The FULL text of the PIA .ovpn (inline /, cipher/auth, remote line). vpn_egress_gateway_pia_ovpn: "{{ vault_vpn_egress_gateway_pia_ovpn }}" +# OVERRIDE THE .ovpn's PINNED `remote` — " ", empty to honour the .ovpn. +# +# WHY THIS EXISTS (measured 2026-09-12): the operator's .ovpn pins ONE PIA server IP. Point +# two gateways at the same vaulted .ovpn and both dial the SAME server, and PIA hands BOTH +# THE SAME tunnel address — dev and prod were both `Server-10779-2a` +# (158.173.241.112:1198) with `ifconfig 10.72.192.7`. Each reconnect then steals that +# address from the other, the loser sees no data for 60s, hits `--ping-restart`, reconnects +# and steals it back: both tunnels flapped on a ~2-minute cycle and neither was reliably +# up. It is NOT a connection-LIMIT problem (the account has room) — it is one pinned server. +# +# So each environment's gateway needs its own server. The PIA Stockholm LB pool +# (`sweden.privacy.network`) resolved to 158.173.241.{34,55,112,135,239}; any PIA server in +# the pool accepts the same credentials and CA, so this is a per-environment knob rather +# than a second vaulted .ovpn. NOT a secret — a public server address — so it belongs in +# group_vars//vars.yml, not the vault. +vpn_egress_gateway_pia_remote: "" + # --- on-disk layout ------------------------------------------------------------- vpn_egress_gateway_config_dir: /etc/openvpn/client vpn_egress_gateway_scripts_dir: /etc/openvpn/scripts @@ -46,17 +63,39 @@ vpn_egress_gateway_blackhole_metric: 4000 # (Cilium VXLAN 1450 → needed 1300) no shrink is required by default. vpn_egress_gateway_tun_mtu: "" -# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ---------- +# --- nftables kill-switch ------------------------------------------------------- +# SCOPE, HONESTLY (measured 2026-09-12, see the .nft template header): this table guards +# NODE-ORIGINATED traffic sourced from a tunnel address. It does NOT and cannot guard the +# egress-class=vpn POD traffic, because Cilium's eBPF egress-gateway datapath +# (bpf.masquerade) redirects those packets to the egress device without ever invoking +# netfilter POSTROUTING. vpn-class fails closed via Cilium's own drop, not via this table. vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch -# Named set holding the LIVE tun0 source address (maintained by the up/down scripts). -# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway -# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never -# in this set and is never matched (fails OPEN toward the WAN). +# Named set of tunnel source addresses, ADDITIVE (the up-script adds, never flushes; the +# down-script retains). Every element only tightens the `oifname != tun0 drop` rule. vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src -# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets -# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an -# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation". -vpn_egress_gateway_fwmark: "0x1c8" +# REMOVED 2026-09-12: vpn_egress_gateway_fwmark ("0x1c8"). It backed a kill-switch rule +# for a mark that nothing stamps — Cilium OSS does not mark egress-gateway traffic with +# it (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule), and the +# traffic never reaches netfilter anyway. A dead rule matching an arbitrary mark is a +# hazard, not a backstop, so both the rule and the variable are gone. + +# --- re-steer: keep Cilium's cached egress interface in sync with tun0 ----------- +# Cilium resolves `interface: tun0` ONCE into a cached (egress IP, egress ifindex) and does +# not re-resolve when tun0 changes; every tunnel restart gives tun0 a NEW ifindex, which +# silently blackholes vpn-class egress. The re-steer script publishes the live uplink as a +# node label; that node update makes the gateway node re-process the policy. Verified on +# k3s-dev-3: cached ifindex moved 1986 -> 1987 within 10s and egress recovered. +# The label VALUE is "-", so `kubectl get nodes -L ` shows +# which uplink Cilium was last told about. +vpn_egress_gateway_uplink_label_key: homelab.keithjackson.dev/egress-uplink +# The kube-API is routinely not reachable yet when OpenVPN connects during boot, so the +# script retries. Bounded so a genuinely broken API does not hang a systemd job forever. +vpn_egress_gateway_resteer_retries: 30 +vpn_egress_gateway_resteer_retry_seconds: 10 +# Convergence backstop if the event-driven trigger is ever lost (no-op when already in +# sync, so it is cheap to run often). +vpn_egress_gateway_resteer_timer_boot: 2min +vpn_egress_gateway_resteer_timer_interval: 2min # --- kubernetes node label (steering contract) ---------------------------------- # The label the parallel CiliumEgressGatewayPolicy selects the gateway node by. diff --git a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml index 206786d..f9a7761 100644 --- a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml @@ -28,9 +28,18 @@ - "{{ vpn_egress_gateway_config_dir }}" - "{{ vpn_egress_gateway_scripts_dir }}" +# `vpn_egress_gateway_pia_remote`, when set, rewrites ONLY the `remote ` line so +# this environment's gateway dials its own PIA server. Two gateways sharing the .ovpn's single +# pinned server are handed the SAME tunnel address and evict each other every ~2 minutes — see +# the variable's comment in defaults/main.yml. The regex is anchored to a line that STARTS with +# `remote` followed by whitespace, so `remote-cert-tls` / `remote-random` are untouched. - name: Deploy the operator's proven PIA .ovpn (from Ansible vault) ansible.builtin.copy: - content: "{{ vpn_egress_gateway_pia_ovpn }}" + content: >- + {{ vpn_egress_gateway_pia_ovpn if (vpn_egress_gateway_pia_remote | length == 0) + else (vpn_egress_gateway_pia_ovpn + | regex_replace('(?m)^[ \t]*remote[ \t]+\S+.*$', + 'remote ' ~ vpn_egress_gateway_pia_remote)) }} dest: "{{ vpn_egress_gateway_ovpn_path }}" owner: root group: root @@ -69,6 +78,17 @@ - vpn-egress-down.sh notify: Restart vpn-egress tunnel +# The re-steer script is what keeps Cilium's cached egress interface in sync with tun0. +# It is NOT notified to the tunnel: it is triggered BY the tunnel's up-script and by its +# own timer, and re-deploying it must never bounce the tunnel. +- name: Deploy the Cilium egress-gateway re-steer script + ansible.builtin.template: + src: vpn-egress-resteer.sh.j2 + dest: "{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh" + owner: root + group: root + mode: "0750" + - name: Deploy the scoped nftables kill-switch ruleset ansible.builtin.template: src: vpn-egress-killswitch.nft.j2 @@ -100,6 +120,18 @@ - Reload systemd - Restart vpn-egress tunnel +- name: Deploy the egress-gateway re-steer unit and its convergence timer + ansible.builtin.template: + src: "{{ item }}.j2" + dest: "/etc/systemd/system/{{ item }}" + owner: root + group: root + mode: "0644" + loop: + - vpn-egress-resteer.service + - vpn-egress-resteer.timer + notify: Reload systemd + - name: Flush handlers so units/routing exist before enable+start ansible.builtin.meta: flush_handlers @@ -115,6 +147,27 @@ enabled: true state: started +# Enabled BEFORE the node label below: the label is what arms steering, and by then the +# re-steer path must already exist so Cilium is told about the live tun0 rather than +# caching whatever it resolves once and never revisiting it. +- name: Enable and start the re-steer convergence timer + ansible.builtin.systemd_service: + name: vpn-egress-resteer.timer + enabled: true + state: started + +# Publish the live uplink immediately rather than waiting for the timer's first tick, so a +# first-time stand-up (or a re-run after a tunnel restart) converges within seconds. +# `started`, NOT `restarted`: the tunnel's up-script fires this same unit on connect, and +# `restarted` SIGTERMs that in-flight run (logging a spurious unit failure) instead of +# letting it finish. The unit is a oneshot without RemainAfterExit, so it is `inactive` +# once it has run and `started` genuinely re-runs it. +- name: Re-steer Cilium onto the live tun0 now + ansible.builtin.systemd_service: + name: vpn-egress-resteer.service + state: started + changed_when: false + # --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ---- - name: Read the current egress-gateway node label ansible.builtin.command: diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 index 21ec2ba..14cd8ff 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 @@ -8,15 +8,20 @@ # then (2) remove the tunnel default from the dedicated table. The permanent # `blackhole default` in that table is LEFT in place as one fail-closed fallback. # -# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED -# here (we do NOT flush it). Keeping the last live tun0 address in +# FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED here (we do +# NOT flush it). Keeping every tunnel address this node has had in # `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule # `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the -# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped -# — defense-in-depth that does not depend on the stale SNAT source being non-routable or -# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then -# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects. -# The set is scoped to vpn-class only; node and internet-class traffic are never in it. +# tunnel is down. The up-script is ADDITIVE (2026-09-12 — it used to flush first, which +# briefly left a previous tunnel address matched by nothing), so the set only ever grows +# and only ever tightens the rule. +# +# SCOPE, HONESTLY: that rule covers NODE-ORIGINATED traffic sourced from a tunnel address. +# It does NOT cover the egress-class=vpn POD traffic — Cilium's eBPF egress-gateway +# datapath never presents those packets to netfilter (measured; see the kill-switch .nft +# header). vpn-class pods fail closed because Cilium drops them when it cannot reach the +# cached egress interface — verified with tcpdump on both the WAN and tun0 during a tunnel +# stop and a tunnel restart: zero packets left either device. # The node's main-table WAN default is never touched here. set -uo pipefail @@ -30,8 +35,8 @@ SRC="${ifconfig_local:-}" # 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule. # NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the -# last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit, -# edge-independent fail-closed backstop until the up-script re-registers on reconnect. +# tun0 addresses keeps the `oifname != tun0 drop` rule armed for node-originated +# tunnel-sourced traffic until the up-script re-registers on reconnect. if [ -n "${SRC}" ]; then while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done fi diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 index 134c6ad..d3faa48 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 @@ -1,40 +1,93 @@ #!/usr/sbin/nft -f # {{ ansible_managed }} -# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED. +# ADR-0015 kill-switch — SCOPED TO tunnel-sourced traffic ONLY, fails CLOSED. +# +# ============================================================================= +# READ THIS BEFORE TRUSTING THIS FILE (measured 2026-09-12 on k3s-dev-3) +# ============================================================================= +# This table does NOT, and CANNOT, guard the `egress-class=vpn` POD traffic that +# ADR-0015 asks it to guard. It was measured, not reasoned about: +# +# A diagnostic nft table (counters only) was installed on the gateway node +# alongside this one, counting every packet in `forward` and `postrouting` by +# (oifname, saddr). While a vpn-class pod egressed through the gateway: +# forward saddr=pod CIDR ................ 0 packets +# forward saddr=PIA tunnel range ........ 0 packets +# postrouting oif tun0 saddr=tunnel ...... 0 packets +# postrouting oif WAN saddr=pod CIDR .... 0 packets +# postrouting oif WAN (any) ............. 45166 packets <- hook IS live +# and, as a positive control, a curl from the NODE itself out tun0 counted +# postrouting oif tun0 saddr=tunnel ...... 16 packets +# +# So the hook is traversed and the matching works — but Cilium's egress-gateway +# datapath never presents the steered pod packets to netfilter at all. With +# `bpf.masquerade: true` the SNAT and the egress happen in eBPF, which +# bpf_redirect()s the skb straight to the egress device: POSTROUTING is not +# invoked for those packets, in either direction. +# +# WHAT ACTUALLY MAKES vpn-class FAIL CLOSED (also measured): +# Cilium caches the gateway's (egress IP, egress ifindex) when it resolves this +# policy. If the cached ifindex no longer exists — which is exactly what a tunnel +# restart produces, tun0 being re-created with a NEW ifindex — the eBPF redirect +# fails and the packet is DROPPED. Verified with tcpdump on BOTH the WAN and tun0 +# during a tunnel restart and during a full stop: 0 packets left either device. +# There is no leak, but the mechanism is Cilium's, not this table's. +# +# WHAT THIS TABLE IS STILL GOOD FOR: +# NODE-ORIGINATED traffic sourced from a tunnel address (the 16-packet control +# above) — e.g. a host-side helper bound to tun0 — which DOES traverse netfilter. +# That is a real, if narrow, guarantee, and it is the only one claimed here. +# Every rule carries a `counter` so the claim is falsifiable at any moment: +# nft list table inet {{ vpn_egress_gateway_nft_table }} +# If `killswitch_drop` ever becomes non-zero, something (a Cilium datapath +# change, bpf.masquerade turned off) has put this traffic back into netfilter — +# read that as new information, not as reassurance. +# +# See platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-12. +# ============================================================================= # # Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules. -# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class -# traffic matched below; node traffic and internet-class traffic are untouched and keep -# egressing the WAN (fail OPEN — invariant 4). It matches on: -# * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs -# vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script). -# This IS the egress-class=vpn traffic and nothing else. -# * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent -# backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark. +# The base chain is policy ACCEPT: this table can ONLY ever drop traffic whose source is +# in the set below; node traffic and internet-class traffic are untouched and keep +# egressing the WAN (fail OPEN — invariant 4). + +# Idempotent reload. `table` + `flush table` clears the RULES only — a named set and its +# elements are not chains, so the live tunnel address SURVIVES a reload and the guard is +# not disarmed by an Ansible run. (A plain `nft -f` without this would append a second +# copy of every rule on each load.) +table inet {{ vpn_egress_gateway_nft_table }} { } +flush table inet {{ vpn_egress_gateway_nft_table }} table inet {{ vpn_egress_gateway_nft_table }} { - # Live tunnel source address(es). Empty whenever the tunnel is down => the two - # saddr rules below match nothing => vpn-class has no path at all (fail closed). + # Every tunnel source address this gateway has had. The OpenVPN up-script ADDS the + # live address on each connect and never flushes (see vpn-egress-up.sh): an element + # can only ever tighten the guard, and retaining previous addresses means a packet + # still sourced from an older tunnel address is dropped rather than unmatched. + # Empty before the very first successful connect => the rules below match nothing, + # and vpn-class has no path at all (fail closed, by Cilium's drop — see above). set {{ vpn_egress_gateway_nft_src_set }} { type ipv4_addr } chain postrouting { - # oifname is resolved at postrouting; SNAT/masquerade has already applied here, - # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address. + # oifname is resolved at postrouting, after any netfilter SNAT. type filter hook postrouting priority 100; policy accept; - # Tunnel UP: let the steered vpn-class traffic out tun0. - oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept + # Tunnel UP: let tunnel-sourced traffic out tun0. + oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} counter accept comment "killswitch_accept" - # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other - # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it - # can never match node or internet-class traffic. - ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # FAIL CLOSED: tunnel-sourced traffic trying to leave ANY other interface (i.e. + # the WAN) is dropped — never leaks. Scoped by the set, so it can never match + # node or internet-class traffic. The counter is the proof-of-life for the whole + # table: see the header for why it reads 0 for POD traffic by construction. + ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" counter drop comment "killswitch_drop" - # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class - # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set - # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark. - meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # REMOVED 2026-09-12 — the `meta mark 0x1c8 oifname != tun0 drop` rule. It was a + # "contract hook" for a fwmark that nothing stamps: Cilium OSS does not mark + # egress-gateway traffic with 0x1c8 (the only Cilium mark visible on the gateway + # node is 0x200/0xf00, which is the L7-proxy rule -> table 2004, unrelated), and + # the traffic does not reach netfilter anyway. A dead rule matching an arbitrary + # mark is a hazard, not a backstop: any future component that happens to use + # 0x1c8 would have had its egress silently dropped. } } diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 new file mode 100644 index 0000000..179cc54 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 @@ -0,0 +1,25 @@ +# {{ ansible_managed }} +# Publishes this node's CURRENT tun0 (address + ifindex) as a node label so Cilium +# re-resolves the CiliumEgressGatewayPolicy's `interface: tun0` (ADR-0015). Without this, +# Cilium keeps a cached (egress IP, egress ifindex) from the FIRST resolution and vpn-class +# egress is silently blackholed after every tunnel restart or reboot — measured, see the +# script header and the DECISIONS-LOG entry of 2026-09-12. +# +# Triggered by the OpenVPN up-script (`systemctl start --no-block`) on every connect, and +# by vpn-egress-resteer.timer as the convergence backstop. Deliberately NOT ordered +# Before/After the tunnel: it must never be able to block or fail the tunnel. +[Unit] +Description=Re-steer Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +After=network-online.target +# The kill-switch owns the nft table this script re-registers the tunnel source into. +After=vpn-egress-killswitch.service + +[Service] +Type=oneshot +ExecStart={{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +# Bounded: {{ vpn_egress_gateway_resteer_retries }} attempts x {{ vpn_egress_gateway_resteer_retry_seconds }}s waiting for the kube-API, plus slack. +TimeoutStartSec={{ (vpn_egress_gateway_resteer_retries | int * vpn_egress_gateway_resteer_retry_seconds | int) + 60 }} +# A failure here means "the kube-API was unreachable"; the timer retries. Do not let a +# failed re-steer mark the whole gateway degraded. +SuccessExitStatus=0 1 diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 new file mode 100644 index 0000000..bdd62dc --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# {{ ansible_managed }} +# Re-steer the Cilium Egress Gateway onto THIS node's CURRENT tun0 (ADR-0015). +# +# WHY THIS EXISTS (measured on k3s-dev-3, 2026-09-12 — see DECISIONS-LOG): +# Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` once, caching an +# (egress IP, egress ifindex) pair in its BPF egress map, and does NOT re-resolve when +# tun0 changes. Observed directly: after `systemctl restart vpn-egress-tunnel` the node +# had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986, and +# vpn-class pod egress was blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — no +# leak, but no service). `egress-gateway-reconciliation-trigger-interval: 1s` does NOT +# cover netlink link events; it only coalesces reconciliations that k8s object events +# already triggered. +# +# Upstream remedy: "You can force a fresh selection by re-applying the Egress Gateway +# policy." Any k8s event that makes the gateway node re-process the policy does it. We +# use a NODE LABEL rather than touching the policy object, because: +# * the policy is Flux-managed GitOps — writing to it from a node invites drift +# fights and ownership churn; the node's labels are already this role's territory; +# * it was verified to work: setting an unrelated label on k3s-dev-3 moved the cached +# ifindex 1986 -> 1987 within 10s and vpn-class egress recovered (PIA IP returned); +# * the label VALUE is the uplink identity, so `kubectl get node -L ...` shows at a +# glance which tun0 Cilium was last told about. Diagnosis needs no exec into a pod. +# +# Idempotent: if the label already records the live uplink, this exits without touching +# anything, so the timer can run as often as it likes. +set -uo pipefail + +IFACE="{{ vpn_egress_gateway_tun_iface }}" +TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" +RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" +TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}" +NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" +NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +NODE="{{ inventory_hostname }}" +UPLINK_LABEL="{{ vpn_egress_gateway_uplink_label_key }}" +RETRIES="{{ vpn_egress_gateway_resteer_retries }}" +RETRY_SLEEP="{{ vpn_egress_gateway_resteer_retry_seconds }}" +export KUBECONFIG="{{ vpn_egress_gateway_kubeconfig }}" + +log() { echo "vpn-egress-resteer: $*"; } + +# --- 1) what is the live uplink? ------------------------------------------------- +addr="$(ip -4 -o addr show dev "${IFACE}" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)" +idx="$(cat "/sys/class/net/${IFACE}/ifindex" 2>/dev/null || true)" +if [ -z "${addr}" ] || [ -z "${idx}" ]; then + # Tunnel down. Nothing to publish — and deliberately NOTHING is torn down here either: + # vpn-class is already fail-closed (Cilium drops when the cached ifindex is gone), and + # the kill-switch set is retained on purpose. The next successful connect re-runs this. + log "${IFACE} has no address yet (tunnel down) — nothing to re-steer" + exit 0 +fi +# A k8s label value must be <=63 chars of [A-Za-z0-9] plus - _ . in the middle. An IPv4 +# address and an ifindex satisfy that; the separator is a dash. +want="${addr}-${idx}" + +# --- 2) re-assert the host side (idempotent; repairs a kill-switch reload) -------- +# `flush table` in the .nft file keeps set elements, but a full ExecStop/ExecStart cycle +# (or a hand-run `nft delete table`) does not, so re-register unconditionally. +nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${addr} }" 2>/dev/null || \ + log "WARNING could not add ${addr} to ${NFT_TABLE}/${NFT_SET} (is the kill-switch loaded?)" +if ! grep -q "from ${addr} lookup" <<<"$(ip rule show)"; then + ip rule add from "${addr}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null || true +fi +ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" 2>/dev/null || true + +# --- 3) has Cilium already been told about THIS uplink? --------------------------- +# The label is the record of what we last published. Retried because at boot the kube-API +# is routinely not up yet when OpenVPN connects. +i=0 +while :; do + i=$((i + 1)) + # -o json + an exact-string match rather than jsonpath: the label key contains dots and + # a slash, which jsonpath needs escaped differently across kubectl versions, and getting + # that wrong would silently read "" and re-label on every timer tick. + node_json="$(k3s kubectl get node "${NODE}" -o json 2>/dev/null)" + if [ -n "${node_json}" ]; then + # Here-strings, NOT pipes: with `set -o pipefail`, `grep -q` exiting early on a match + # SIGPIPEs the writer and the pipeline reports 141 — i.e. a successful match would read + # as a failure and this script would re-label on every single run. + if grep -qF "\"${UPLINK_LABEL}\": \"${want}\"" <<<"${node_json}"; then + log "uplink unchanged (${want}) — Cilium already re-steered, nothing to do" + exit 0 + fi + have="$(grep -F "\"${UPLINK_LABEL}\": " <<<"${node_json}" | head -1 | cut -d'"' -f4)" + log "uplink changed: label='${have:-}' live='${want}' — re-steering" + if k3s kubectl label node "${NODE}" "${UPLINK_LABEL}=${want}" --overwrite >/dev/null 2>&1; then + log "re-steered: ${NODE} ${UPLINK_LABEL}=${want}" + exit 0 + fi + fi + if [ "${i}" -ge "${RETRIES}" ]; then + log "ERROR kube-API unreachable or label write failed after ${i} attempts — the timer will retry" + exit 1 + fi + sleep "${RETRY_SLEEP}" +done diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 new file mode 100644 index 0000000..f2fb55a --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 @@ -0,0 +1,19 @@ +# {{ ansible_managed }} +# Convergence backstop for the event-driven re-steer. The OpenVPN up-script fires +# vpn-egress-resteer.service on every connect, but that call can be lost (kube-API not up +# yet at boot and all in-script retries exhausted, an OpenVPN soft restart whose up-script +# ran before k3s, a hand-run `nft delete table`). This timer makes the gateway converge on +# its own instead of needing an operator. The script is a no-op when the published label +# already matches the live tun0, so running it often is free. +[Unit] +Description=Periodic re-steer of the Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh + +[Timer] +OnBootSec={{ vpn_egress_gateway_resteer_timer_boot }} +OnUnitInactiveSec={{ vpn_egress_gateway_resteer_timer_interval }} +AccuracySec=15s +Unit=vpn-egress-resteer.service + +[Install] +WantedBy=timers.target diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 index d108dbc..36a5f87 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 @@ -35,9 +35,34 @@ ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}" while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" -# Register the live tunnel source with the kill-switch set so the nft backstop can scope -# to exactly this traffic. Flush any stale members first (address changes on reconnect). -nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +# Register the live tunnel source with the kill-switch set. ADDITIVE ON PURPOSE — the set +# is NEVER flushed here (it used to be, see the DECISIONS-LOG entry of 2026-09-12). +# `nft add element` is idempotent, and every address that has ever been this tunnel's +# source can only ever TIGHTEN the `oifname != tun0 drop` rule. Flushing first meant that +# between a reconnect and Cilium re-resolving its cached egress IP there was a moment +# where a packet still sourced from the PREVIOUS address matched neither the ip-rule nor +# the drop rule. (Measurement since showed that window cannot leak for POD traffic, +# because that traffic never enters netfilter at all — see the kill-switch .nft header — +# but node-originated tunnel traffic DOES, and additive costs nothing.) nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }" +# THE LOAD-BEARING FIX (2026-09-12): tell Cilium that this gateway's uplink changed. +# +# Cilium resolves the CiliumEgressGatewayPolicy's `interface: tun0` ONCE, into a cached +# (egress IP, egress ifindex) pair, and — measured on k3s-dev-3 — does NOT re-resolve it +# when tun0 changes underneath, not even with `egress-gateway-reconciliation-trigger- +# interval: 1s`. Upstream says as much: "it does not automatically respond to a change in +# the gateway node's network configuration... You can force a fresh selection by +# re-applying the Egress Gateway policy." +# +# Every tunnel restart re-creates tun0 with a NEW ifindex, so WITHOUT this the gateway is +# silently dead after its first reconnect or reboot: vpn-class egress is blackholed (no +# leak — the eBPF redirect to the dead ifindex just fails — but no service, and no alert). +# Re-steering is done out-of-band by vpn-egress-resteer.service so this hook can never +# block or fail the tunnel: OpenVPN's up-script must stay fast and must not depend on the +# kube-API being reachable yet (at boot it usually is not — the resteer unit retries, and +# its timer converges anyway). +systemctl start --no-block vpn-egress-resteer.service || \ + echo "vpn-egress-up: WARNING could not trigger vpn-egress-resteer.service (timer will converge)" >&2 + echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active" diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index cbd3e58..6883332 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015 by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster gateway pod** — the tunnel is host infrastructure, not a Flux workload. -> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class` -> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster. +> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`. +> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev. +> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the +> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale +> egress ifindex — see below), so the test's recovery step could not have passed on the +> implementation as it stood. Whatever produced the earlier green result, re-run it. ## How it works (ADR-0015) Three layers, two owners: @@ -24,9 +28,134 @@ Three layers, two owners: (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel. -- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered - `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so - `internet`-class and node traffic are unaffected. +- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails + closed because Cilium drops the traffic when it cannot use its cached egress interface. The host + nftables table is retained for a narrower job. See "The kill-switch does not do what this file + used to claim" below — that section is the measurement, not an opinion. + +## LIVE ON PROD since 2026-09-12 — `k3s-prod-3` + +| | dev | prod | +|---|---|---| +| gateway node | `k3s-dev-3` | `k3s-prod-3` | +| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | +| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) | +| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` | +| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | + +Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three +nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home +WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was +reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed +`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the +gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also +means nothing had been leaking in the meantime. + +### One PIA account is fine. One pinned PIA *server* is not. + +Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn` +pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same +tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips +`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute +off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's +first probes looked green only because they landed inside an up-window. + +Dev recovered the minute prod left `.112`: its last `Inactivity timeout` is 23:05:06 UTC, and at +23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with +a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to `158.173.241.36`. + +**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` +rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the +same PIA Stockholm LB pool (`sweden.privacy.network` resolves to +`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev +keeps `.112`. + +**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must +pick an unused address from the pool. A separate prod PIA account would also solve it — see +`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`. + +That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's +tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real +new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink` +label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing, +now via `158.173.241.11`. No manual step, no CEGP edit. + +## The kill-switch does not do what this file used to claim (measured 2026-09-12) + +This README, `egress-steering.yaml`, and the role all used to say the **host nftables +kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.** +With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in +eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is +never invoked for those packets. + +A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class +pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 | +| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 | + +**What actually fails closed is Cilium itself.** When it cannot use its cached egress +interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a +tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the +guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance +test passes because of it, not because of the nft rules. + +The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely +does cover: **node-originated** traffic sourced from a tunnel address. That was proven on +`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 +replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out +`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back. +If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read +that as news, not reassurance. + +Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that +mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table +2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is +a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark` +variable is gone with it, and the "add the fwmark to the interface contract" escalation in the +role README is **closed as not-implementable** rather than left open. + +## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help + +Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not +revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes +stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been +dead for ~80 minutes when this work started, with no alert and nothing in the ledger. + +`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that +setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces +reconciliations that k8s object events already triggered — netlink link events are not among them. + +The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=-` on +the gateway node on every connect, plus a 2-minute convergence timer. That node update makes +Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 +within 10s, egress recovered immediately). A node label rather than an annotation on this +Flux-managed policy, so GitOps state is never written to from a node. + +**Debugging a dead gateway starts here:** + +```bash +kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink +kubectl -n kube-system exec -c cilium-agent -- \ + cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0` +``` + +If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the +gateway node is the next stop. ## Resilience — there is NO gateway failover (spec correction, escalated) diff --git a/platform/vpn-egress/egress-steering.yaml b/platform/vpn-egress/egress-steering.yaml index 1cabfa8..bd06f2b 100644 --- a/platform/vpn-egress/egress-steering.yaml +++ b/platform/vpn-egress/egress-steering.yaml @@ -59,16 +59,38 @@ spec: # Excluded flows fall through to Cilium's normal local datapath (invariant 1). destinationCIDRs: - "0.0.0.0/0" + # + # BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media + # mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both + # inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason + # is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted + # by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never + # matches it and no pod-scoped egress steering can touch it. Verified after arming the prod + # gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list + # AND write the export, and the node's own mounts are intact. No RFC1918 destination needs + # adding here for the media path. excludedCIDRs: - "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net - "172.16.0.0/12" # RFC1918 - - "192.168.0.0/16" # RFC1918 (home LAN / node network family) + - "192.168.0.0/16" # RFC1918 (home LAN / node network family / the bulk NFS server) # --- Where it egresses -------------------------------------------------- - # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When - # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible, - # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to - # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED, - # internet fails OPEN). + # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. + # + # CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the + # authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot + # be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the + # skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never + # see this traffic. Measured with a counters-only nft table on the gateway node while a + # vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and + # the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live) + # and 16 for a node-originated curl out tun0 (so the matching works). + # WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used + # it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a + # tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak); + # internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold). + # The host nft table is retained, with counters, for the narrower case it CAN see: + # node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's + # counter moved 0 -> 3 packets when such a packet was routed at the WAN). # # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby. # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD @@ -89,6 +111,22 @@ spec: # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)" # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06. + # + # `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes. + # Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the + # first resolution and does NOT re-resolve on a netlink change. `systemctl restart + # vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class + # egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but + # dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events. + # Upstream: "it does not automatically respond to a change in the gateway node's network + # configuration... You can force a fresh selection by re-applying the Egress Gateway policy." + # The Ansible role closes this: roles/vpn_egress_gateway publishes + # `homelab.keithjackson.dev/egress-uplink=-` on the gateway node on every + # connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this + # policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s. + # DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node + # writing to GitOps state is a drift fight. Check the label, not this file, when debugging: + # kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink egressGateway: nodeSelector: matchLabels: