From 6f698f59af2835edf412575500d2e7bb28a4cfdd Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:38:27 -0500 Subject: [PATCH] =?UTF-8?q?fix(vpn):=20Cilium=20never=20re-resolves=20tun0?= =?UTF-8?q?=20=E2=80=94=20the=20gateway=20dies=20on=20every=20reconnect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "-" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 --- .../vpn_egress_gateway/defaults/main.yml | 40 ++++++-- .../roles/vpn_egress_gateway/tasks/main.yml | 44 +++++++++ .../templates/vpn-egress-down.sh.j2 | 23 +++-- .../templates/vpn-egress-killswitch.nft.j2 | 99 ++++++++++++++----- .../templates/vpn-egress-resteer.service.j2 | 25 +++++ .../templates/vpn-egress-resteer.sh.j2 | 97 ++++++++++++++++++ .../templates/vpn-egress-resteer.timer.j2 | 19 ++++ .../templates/vpn-egress-up.sh.j2 | 31 +++++- 8 files changed, 334 insertions(+), 44 deletions(-) create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 diff --git a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml index 8e82ef0..359d1ea 100644 --- a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml @@ -46,17 +46,39 @@ vpn_egress_gateway_blackhole_metric: 4000 # (Cilium VXLAN 1450 → needed 1300) no shrink is required by default. vpn_egress_gateway_tun_mtu: "" -# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ---------- +# --- nftables kill-switch ------------------------------------------------------- +# SCOPE, HONESTLY (measured 2026-09-12, see the .nft template header): this table guards +# NODE-ORIGINATED traffic sourced from a tunnel address. It does NOT and cannot guard the +# egress-class=vpn POD traffic, because Cilium's eBPF egress-gateway datapath +# (bpf.masquerade) redirects those packets to the egress device without ever invoking +# netfilter POSTROUTING. vpn-class fails closed via Cilium's own drop, not via this table. vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch -# Named set holding the LIVE tun0 source address (maintained by the up/down scripts). -# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway -# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never -# in this set and is never matched (fails OPEN toward the WAN). +# Named set of tunnel source addresses, ADDITIVE (the up-script adds, never flushes; the +# down-script retains). Every element only tightens the `oifname != tun0 drop` rule. vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src -# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets -# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an -# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation". -vpn_egress_gateway_fwmark: "0x1c8" +# REMOVED 2026-09-12: vpn_egress_gateway_fwmark ("0x1c8"). It backed a kill-switch rule +# for a mark that nothing stamps — Cilium OSS does not mark egress-gateway traffic with +# it (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule), and the +# traffic never reaches netfilter anyway. A dead rule matching an arbitrary mark is a +# hazard, not a backstop, so both the rule and the variable are gone. + +# --- re-steer: keep Cilium's cached egress interface in sync with tun0 ----------- +# Cilium resolves `interface: tun0` ONCE into a cached (egress IP, egress ifindex) and does +# not re-resolve when tun0 changes; every tunnel restart gives tun0 a NEW ifindex, which +# silently blackholes vpn-class egress. The re-steer script publishes the live uplink as a +# node label; that node update makes the gateway node re-process the policy. Verified on +# k3s-dev-3: cached ifindex moved 1986 -> 1987 within 10s and egress recovered. +# The label VALUE is "-", so `kubectl get nodes -L ` shows +# which uplink Cilium was last told about. +vpn_egress_gateway_uplink_label_key: homelab.keithjackson.dev/egress-uplink +# The kube-API is routinely not reachable yet when OpenVPN connects during boot, so the +# script retries. Bounded so a genuinely broken API does not hang a systemd job forever. +vpn_egress_gateway_resteer_retries: 30 +vpn_egress_gateway_resteer_retry_seconds: 10 +# Convergence backstop if the event-driven trigger is ever lost (no-op when already in +# sync, so it is cheap to run often). +vpn_egress_gateway_resteer_timer_boot: 2min +vpn_egress_gateway_resteer_timer_interval: 2min # --- kubernetes node label (steering contract) ---------------------------------- # The label the parallel CiliumEgressGatewayPolicy selects the gateway node by. diff --git a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml index 206786d..401a13f 100644 --- a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml @@ -69,6 +69,17 @@ - vpn-egress-down.sh notify: Restart vpn-egress tunnel +# The re-steer script is what keeps Cilium's cached egress interface in sync with tun0. +# It is NOT notified to the tunnel: it is triggered BY the tunnel's up-script and by its +# own timer, and re-deploying it must never bounce the tunnel. +- name: Deploy the Cilium egress-gateway re-steer script + ansible.builtin.template: + src: vpn-egress-resteer.sh.j2 + dest: "{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh" + owner: root + group: root + mode: "0750" + - name: Deploy the scoped nftables kill-switch ruleset ansible.builtin.template: src: vpn-egress-killswitch.nft.j2 @@ -100,6 +111,18 @@ - Reload systemd - Restart vpn-egress tunnel +- name: Deploy the egress-gateway re-steer unit and its convergence timer + ansible.builtin.template: + src: "{{ item }}.j2" + dest: "/etc/systemd/system/{{ item }}" + owner: root + group: root + mode: "0644" + loop: + - vpn-egress-resteer.service + - vpn-egress-resteer.timer + notify: Reload systemd + - name: Flush handlers so units/routing exist before enable+start ansible.builtin.meta: flush_handlers @@ -115,6 +138,27 @@ enabled: true state: started +# Enabled BEFORE the node label below: the label is what arms steering, and by then the +# re-steer path must already exist so Cilium is told about the live tun0 rather than +# caching whatever it resolves once and never revisiting it. +- name: Enable and start the re-steer convergence timer + ansible.builtin.systemd_service: + name: vpn-egress-resteer.timer + enabled: true + state: started + +# Publish the live uplink immediately rather than waiting for the timer's first tick, so a +# first-time stand-up (or a re-run after a tunnel restart) converges within seconds. +# `started`, NOT `restarted`: the tunnel's up-script fires this same unit on connect, and +# `restarted` SIGTERMs that in-flight run (logging a spurious unit failure) instead of +# letting it finish. The unit is a oneshot without RemainAfterExit, so it is `inactive` +# once it has run and `started` genuinely re-runs it. +- name: Re-steer Cilium onto the live tun0 now + ansible.builtin.systemd_service: + name: vpn-egress-resteer.service + state: started + changed_when: false + # --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ---- - name: Read the current egress-gateway node label ansible.builtin.command: diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 index 21ec2ba..14cd8ff 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 @@ -8,15 +8,20 @@ # then (2) remove the tunnel default from the dedicated table. The permanent # `blackhole default` in that table is LEFT in place as one fail-closed fallback. # -# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED -# here (we do NOT flush it). Keeping the last live tun0 address in +# FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED here (we do +# NOT flush it). Keeping every tunnel address this node has had in # `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule # `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the -# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped -# — defense-in-depth that does not depend on the stale SNAT source being non-routable or -# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then -# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects. -# The set is scoped to vpn-class only; node and internet-class traffic are never in it. +# tunnel is down. The up-script is ADDITIVE (2026-09-12 — it used to flush first, which +# briefly left a previous tunnel address matched by nothing), so the set only ever grows +# and only ever tightens the rule. +# +# SCOPE, HONESTLY: that rule covers NODE-ORIGINATED traffic sourced from a tunnel address. +# It does NOT cover the egress-class=vpn POD traffic — Cilium's eBPF egress-gateway +# datapath never presents those packets to netfilter (measured; see the kill-switch .nft +# header). vpn-class pods fail closed because Cilium drops them when it cannot reach the +# cached egress interface — verified with tcpdump on both the WAN and tun0 during a tunnel +# stop and a tunnel restart: zero packets left either device. # The node's main-table WAN default is never touched here. set -uo pipefail @@ -30,8 +35,8 @@ SRC="${ifconfig_local:-}" # 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule. # NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the -# last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit, -# edge-independent fail-closed backstop until the up-script re-registers on reconnect. +# tun0 addresses keeps the `oifname != tun0 drop` rule armed for node-originated +# tunnel-sourced traffic until the up-script re-registers on reconnect. if [ -n "${SRC}" ]; then while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done fi diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 index 134c6ad..d3faa48 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 @@ -1,40 +1,93 @@ #!/usr/sbin/nft -f # {{ ansible_managed }} -# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED. +# ADR-0015 kill-switch — SCOPED TO tunnel-sourced traffic ONLY, fails CLOSED. +# +# ============================================================================= +# READ THIS BEFORE TRUSTING THIS FILE (measured 2026-09-12 on k3s-dev-3) +# ============================================================================= +# This table does NOT, and CANNOT, guard the `egress-class=vpn` POD traffic that +# ADR-0015 asks it to guard. It was measured, not reasoned about: +# +# A diagnostic nft table (counters only) was installed on the gateway node +# alongside this one, counting every packet in `forward` and `postrouting` by +# (oifname, saddr). While a vpn-class pod egressed through the gateway: +# forward saddr=pod CIDR ................ 0 packets +# forward saddr=PIA tunnel range ........ 0 packets +# postrouting oif tun0 saddr=tunnel ...... 0 packets +# postrouting oif WAN saddr=pod CIDR .... 0 packets +# postrouting oif WAN (any) ............. 45166 packets <- hook IS live +# and, as a positive control, a curl from the NODE itself out tun0 counted +# postrouting oif tun0 saddr=tunnel ...... 16 packets +# +# So the hook is traversed and the matching works — but Cilium's egress-gateway +# datapath never presents the steered pod packets to netfilter at all. With +# `bpf.masquerade: true` the SNAT and the egress happen in eBPF, which +# bpf_redirect()s the skb straight to the egress device: POSTROUTING is not +# invoked for those packets, in either direction. +# +# WHAT ACTUALLY MAKES vpn-class FAIL CLOSED (also measured): +# Cilium caches the gateway's (egress IP, egress ifindex) when it resolves this +# policy. If the cached ifindex no longer exists — which is exactly what a tunnel +# restart produces, tun0 being re-created with a NEW ifindex — the eBPF redirect +# fails and the packet is DROPPED. Verified with tcpdump on BOTH the WAN and tun0 +# during a tunnel restart and during a full stop: 0 packets left either device. +# There is no leak, but the mechanism is Cilium's, not this table's. +# +# WHAT THIS TABLE IS STILL GOOD FOR: +# NODE-ORIGINATED traffic sourced from a tunnel address (the 16-packet control +# above) — e.g. a host-side helper bound to tun0 — which DOES traverse netfilter. +# That is a real, if narrow, guarantee, and it is the only one claimed here. +# Every rule carries a `counter` so the claim is falsifiable at any moment: +# nft list table inet {{ vpn_egress_gateway_nft_table }} +# If `killswitch_drop` ever becomes non-zero, something (a Cilium datapath +# change, bpf.masquerade turned off) has put this traffic back into netfilter — +# read that as new information, not as reassurance. +# +# See platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-12. +# ============================================================================= # # Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules. -# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class -# traffic matched below; node traffic and internet-class traffic are untouched and keep -# egressing the WAN (fail OPEN — invariant 4). It matches on: -# * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs -# vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script). -# This IS the egress-class=vpn traffic and nothing else. -# * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent -# backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark. +# The base chain is policy ACCEPT: this table can ONLY ever drop traffic whose source is +# in the set below; node traffic and internet-class traffic are untouched and keep +# egressing the WAN (fail OPEN — invariant 4). + +# Idempotent reload. `table` + `flush table` clears the RULES only — a named set and its +# elements are not chains, so the live tunnel address SURVIVES a reload and the guard is +# not disarmed by an Ansible run. (A plain `nft -f` without this would append a second +# copy of every rule on each load.) +table inet {{ vpn_egress_gateway_nft_table }} { } +flush table inet {{ vpn_egress_gateway_nft_table }} table inet {{ vpn_egress_gateway_nft_table }} { - # Live tunnel source address(es). Empty whenever the tunnel is down => the two - # saddr rules below match nothing => vpn-class has no path at all (fail closed). + # Every tunnel source address this gateway has had. The OpenVPN up-script ADDS the + # live address on each connect and never flushes (see vpn-egress-up.sh): an element + # can only ever tighten the guard, and retaining previous addresses means a packet + # still sourced from an older tunnel address is dropped rather than unmatched. + # Empty before the very first successful connect => the rules below match nothing, + # and vpn-class has no path at all (fail closed, by Cilium's drop — see above). set {{ vpn_egress_gateway_nft_src_set }} { type ipv4_addr } chain postrouting { - # oifname is resolved at postrouting; SNAT/masquerade has already applied here, - # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address. + # oifname is resolved at postrouting, after any netfilter SNAT. type filter hook postrouting priority 100; policy accept; - # Tunnel UP: let the steered vpn-class traffic out tun0. - oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept + # Tunnel UP: let tunnel-sourced traffic out tun0. + oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} counter accept comment "killswitch_accept" - # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other - # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it - # can never match node or internet-class traffic. - ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # FAIL CLOSED: tunnel-sourced traffic trying to leave ANY other interface (i.e. + # the WAN) is dropped — never leaks. Scoped by the set, so it can never match + # node or internet-class traffic. The counter is the proof-of-life for the whole + # table: see the header for why it reads 0 for POD traffic by construction. + ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" counter drop comment "killswitch_drop" - # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class - # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set - # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark. - meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # REMOVED 2026-09-12 — the `meta mark 0x1c8 oifname != tun0 drop` rule. It was a + # "contract hook" for a fwmark that nothing stamps: Cilium OSS does not mark + # egress-gateway traffic with 0x1c8 (the only Cilium mark visible on the gateway + # node is 0x200/0xf00, which is the L7-proxy rule -> table 2004, unrelated), and + # the traffic does not reach netfilter anyway. A dead rule matching an arbitrary + # mark is a hazard, not a backstop: any future component that happens to use + # 0x1c8 would have had its egress silently dropped. } } diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 new file mode 100644 index 0000000..179cc54 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 @@ -0,0 +1,25 @@ +# {{ ansible_managed }} +# Publishes this node's CURRENT tun0 (address + ifindex) as a node label so Cilium +# re-resolves the CiliumEgressGatewayPolicy's `interface: tun0` (ADR-0015). Without this, +# Cilium keeps a cached (egress IP, egress ifindex) from the FIRST resolution and vpn-class +# egress is silently blackholed after every tunnel restart or reboot — measured, see the +# script header and the DECISIONS-LOG entry of 2026-09-12. +# +# Triggered by the OpenVPN up-script (`systemctl start --no-block`) on every connect, and +# by vpn-egress-resteer.timer as the convergence backstop. Deliberately NOT ordered +# Before/After the tunnel: it must never be able to block or fail the tunnel. +[Unit] +Description=Re-steer Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +After=network-online.target +# The kill-switch owns the nft table this script re-registers the tunnel source into. +After=vpn-egress-killswitch.service + +[Service] +Type=oneshot +ExecStart={{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +# Bounded: {{ vpn_egress_gateway_resteer_retries }} attempts x {{ vpn_egress_gateway_resteer_retry_seconds }}s waiting for the kube-API, plus slack. +TimeoutStartSec={{ (vpn_egress_gateway_resteer_retries | int * vpn_egress_gateway_resteer_retry_seconds | int) + 60 }} +# A failure here means "the kube-API was unreachable"; the timer retries. Do not let a +# failed re-steer mark the whole gateway degraded. +SuccessExitStatus=0 1 diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 new file mode 100644 index 0000000..bdd62dc --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# {{ ansible_managed }} +# Re-steer the Cilium Egress Gateway onto THIS node's CURRENT tun0 (ADR-0015). +# +# WHY THIS EXISTS (measured on k3s-dev-3, 2026-09-12 — see DECISIONS-LOG): +# Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` once, caching an +# (egress IP, egress ifindex) pair in its BPF egress map, and does NOT re-resolve when +# tun0 changes. Observed directly: after `systemctl restart vpn-egress-tunnel` the node +# had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986, and +# vpn-class pod egress was blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — no +# leak, but no service). `egress-gateway-reconciliation-trigger-interval: 1s` does NOT +# cover netlink link events; it only coalesces reconciliations that k8s object events +# already triggered. +# +# Upstream remedy: "You can force a fresh selection by re-applying the Egress Gateway +# policy." Any k8s event that makes the gateway node re-process the policy does it. We +# use a NODE LABEL rather than touching the policy object, because: +# * the policy is Flux-managed GitOps — writing to it from a node invites drift +# fights and ownership churn; the node's labels are already this role's territory; +# * it was verified to work: setting an unrelated label on k3s-dev-3 moved the cached +# ifindex 1986 -> 1987 within 10s and vpn-class egress recovered (PIA IP returned); +# * the label VALUE is the uplink identity, so `kubectl get node -L ...` shows at a +# glance which tun0 Cilium was last told about. Diagnosis needs no exec into a pod. +# +# Idempotent: if the label already records the live uplink, this exits without touching +# anything, so the timer can run as often as it likes. +set -uo pipefail + +IFACE="{{ vpn_egress_gateway_tun_iface }}" +TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" +RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" +TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}" +NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" +NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +NODE="{{ inventory_hostname }}" +UPLINK_LABEL="{{ vpn_egress_gateway_uplink_label_key }}" +RETRIES="{{ vpn_egress_gateway_resteer_retries }}" +RETRY_SLEEP="{{ vpn_egress_gateway_resteer_retry_seconds }}" +export KUBECONFIG="{{ vpn_egress_gateway_kubeconfig }}" + +log() { echo "vpn-egress-resteer: $*"; } + +# --- 1) what is the live uplink? ------------------------------------------------- +addr="$(ip -4 -o addr show dev "${IFACE}" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)" +idx="$(cat "/sys/class/net/${IFACE}/ifindex" 2>/dev/null || true)" +if [ -z "${addr}" ] || [ -z "${idx}" ]; then + # Tunnel down. Nothing to publish — and deliberately NOTHING is torn down here either: + # vpn-class is already fail-closed (Cilium drops when the cached ifindex is gone), and + # the kill-switch set is retained on purpose. The next successful connect re-runs this. + log "${IFACE} has no address yet (tunnel down) — nothing to re-steer" + exit 0 +fi +# A k8s label value must be <=63 chars of [A-Za-z0-9] plus - _ . in the middle. An IPv4 +# address and an ifindex satisfy that; the separator is a dash. +want="${addr}-${idx}" + +# --- 2) re-assert the host side (idempotent; repairs a kill-switch reload) -------- +# `flush table` in the .nft file keeps set elements, but a full ExecStop/ExecStart cycle +# (or a hand-run `nft delete table`) does not, so re-register unconditionally. +nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${addr} }" 2>/dev/null || \ + log "WARNING could not add ${addr} to ${NFT_TABLE}/${NFT_SET} (is the kill-switch loaded?)" +if ! grep -q "from ${addr} lookup" <<<"$(ip rule show)"; then + ip rule add from "${addr}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null || true +fi +ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" 2>/dev/null || true + +# --- 3) has Cilium already been told about THIS uplink? --------------------------- +# The label is the record of what we last published. Retried because at boot the kube-API +# is routinely not up yet when OpenVPN connects. +i=0 +while :; do + i=$((i + 1)) + # -o json + an exact-string match rather than jsonpath: the label key contains dots and + # a slash, which jsonpath needs escaped differently across kubectl versions, and getting + # that wrong would silently read "" and re-label on every timer tick. + node_json="$(k3s kubectl get node "${NODE}" -o json 2>/dev/null)" + if [ -n "${node_json}" ]; then + # Here-strings, NOT pipes: with `set -o pipefail`, `grep -q` exiting early on a match + # SIGPIPEs the writer and the pipeline reports 141 — i.e. a successful match would read + # as a failure and this script would re-label on every single run. + if grep -qF "\"${UPLINK_LABEL}\": \"${want}\"" <<<"${node_json}"; then + log "uplink unchanged (${want}) — Cilium already re-steered, nothing to do" + exit 0 + fi + have="$(grep -F "\"${UPLINK_LABEL}\": " <<<"${node_json}" | head -1 | cut -d'"' -f4)" + log "uplink changed: label='${have:-}' live='${want}' — re-steering" + if k3s kubectl label node "${NODE}" "${UPLINK_LABEL}=${want}" --overwrite >/dev/null 2>&1; then + log "re-steered: ${NODE} ${UPLINK_LABEL}=${want}" + exit 0 + fi + fi + if [ "${i}" -ge "${RETRIES}" ]; then + log "ERROR kube-API unreachable or label write failed after ${i} attempts — the timer will retry" + exit 1 + fi + sleep "${RETRY_SLEEP}" +done diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 new file mode 100644 index 0000000..f2fb55a --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 @@ -0,0 +1,19 @@ +# {{ ansible_managed }} +# Convergence backstop for the event-driven re-steer. The OpenVPN up-script fires +# vpn-egress-resteer.service on every connect, but that call can be lost (kube-API not up +# yet at boot and all in-script retries exhausted, an OpenVPN soft restart whose up-script +# ran before k3s, a hand-run `nft delete table`). This timer makes the gateway converge on +# its own instead of needing an operator. The script is a no-op when the published label +# already matches the live tun0, so running it often is free. +[Unit] +Description=Periodic re-steer of the Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh + +[Timer] +OnBootSec={{ vpn_egress_gateway_resteer_timer_boot }} +OnUnitInactiveSec={{ vpn_egress_gateway_resteer_timer_interval }} +AccuracySec=15s +Unit=vpn-egress-resteer.service + +[Install] +WantedBy=timers.target diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 index d108dbc..36a5f87 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 @@ -35,9 +35,34 @@ ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}" while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" -# Register the live tunnel source with the kill-switch set so the nft backstop can scope -# to exactly this traffic. Flush any stale members first (address changes on reconnect). -nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +# Register the live tunnel source with the kill-switch set. ADDITIVE ON PURPOSE — the set +# is NEVER flushed here (it used to be, see the DECISIONS-LOG entry of 2026-09-12). +# `nft add element` is idempotent, and every address that has ever been this tunnel's +# source can only ever TIGHTEN the `oifname != tun0 drop` rule. Flushing first meant that +# between a reconnect and Cilium re-resolving its cached egress IP there was a moment +# where a packet still sourced from the PREVIOUS address matched neither the ip-rule nor +# the drop rule. (Measurement since showed that window cannot leak for POD traffic, +# because that traffic never enters netfilter at all — see the kill-switch .nft header — +# but node-originated tunnel traffic DOES, and additive costs nothing.) nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }" +# THE LOAD-BEARING FIX (2026-09-12): tell Cilium that this gateway's uplink changed. +# +# Cilium resolves the CiliumEgressGatewayPolicy's `interface: tun0` ONCE, into a cached +# (egress IP, egress ifindex) pair, and — measured on k3s-dev-3 — does NOT re-resolve it +# when tun0 changes underneath, not even with `egress-gateway-reconciliation-trigger- +# interval: 1s`. Upstream says as much: "it does not automatically respond to a change in +# the gateway node's network configuration... You can force a fresh selection by +# re-applying the Egress Gateway policy." +# +# Every tunnel restart re-creates tun0 with a NEW ifindex, so WITHOUT this the gateway is +# silently dead after its first reconnect or reboot: vpn-class egress is blackholed (no +# leak — the eBPF redirect to the dead ifindex just fails — but no service, and no alert). +# Re-steering is done out-of-band by vpn-egress-resteer.service so this hook can never +# block or fail the tunnel: OpenVPN's up-script must stay fast and must not depend on the +# kube-API being reachable yet (at boot it usually is not — the resteer unit retries, and +# its timer converges anyway). +systemctl start --no-block vpn-egress-resteer.service || \ + echo "vpn-egress-up: WARNING could not trigger vpn-egress-resteer.service (timer will converge)" >&2 + echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active"