fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
6b953cc56b
commit
6f698f59af
8 files changed
+334
-44
No files matched your search
@@ -46,17 +46,39 @@ vpn_egress_gateway_blackhole_metric: 4000
|
||||
# (Cilium VXLAN 1450 → needed 1300) no shrink is required by default.
|
||||
vpn_egress_gateway_tun_mtu: ""
|
||||
|
||||
# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ----------
|
||||
# --- nftables kill-switch -------------------------------------------------------
|
||||
# SCOPE, HONESTLY (measured 2026-09-12, see the .nft template header): this table guards
|
||||
# NODE-ORIGINATED traffic sourced from a tunnel address. It does NOT and cannot guard the
|
||||
# egress-class=vpn POD traffic, because Cilium's eBPF egress-gateway datapath
|
||||
# (bpf.masquerade) redirects those packets to the egress device without ever invoking
|
||||
# netfilter POSTROUTING. vpn-class fails closed via Cilium's own drop, not via this table.
|
||||
vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch
|
||||
# Named set holding the LIVE tun0 source address (maintained by the up/down scripts).
|
||||
# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway
|
||||
# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never
|
||||
# in this set and is never matched (fails OPEN toward the WAN).
|
||||
# Named set of tunnel source addresses, ADDITIVE (the up-script adds, never flushes; the
|
||||
# down-script retains). Every element only tightens the `oifname != tun0 drop` rule.
|
||||
vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src
|
||||
# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets
|
||||
# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an
|
||||
# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation".
|
||||
vpn_egress_gateway_fwmark: "0x1c8"
|
||||
# REMOVED 2026-09-12: vpn_egress_gateway_fwmark ("0x1c8"). It backed a kill-switch rule
|
||||
# for a mark that nothing stamps — Cilium OSS does not mark egress-gateway traffic with
|
||||
# it (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule), and the
|
||||
# traffic never reaches netfilter anyway. A dead rule matching an arbitrary mark is a
|
||||
# hazard, not a backstop, so both the rule and the variable are gone.
|
||||
|
||||
# --- re-steer: keep Cilium's cached egress interface in sync with tun0 -----------
|
||||
# Cilium resolves `interface: tun0` ONCE into a cached (egress IP, egress ifindex) and does
|
||||
# not re-resolve when tun0 changes; every tunnel restart gives tun0 a NEW ifindex, which
|
||||
# silently blackholes vpn-class egress. The re-steer script publishes the live uplink as a
|
||||
# node label; that node update makes the gateway node re-process the policy. Verified on
|
||||
# k3s-dev-3: cached ifindex moved 1986 -> 1987 within 10s and egress recovered.
|
||||
# The label VALUE is "<tun0 address>-<ifindex>", so `kubectl get nodes -L <this key>` shows
|
||||
# which uplink Cilium was last told about.
|
||||
vpn_egress_gateway_uplink_label_key: homelab.keithjackson.dev/egress-uplink
|
||||
# The kube-API is routinely not reachable yet when OpenVPN connects during boot, so the
|
||||
# script retries. Bounded so a genuinely broken API does not hang a systemd job forever.
|
||||
vpn_egress_gateway_resteer_retries: 30
|
||||
vpn_egress_gateway_resteer_retry_seconds: 10
|
||||
# Convergence backstop if the event-driven trigger is ever lost (no-op when already in
|
||||
# sync, so it is cheap to run often).
|
||||
vpn_egress_gateway_resteer_timer_boot: 2min
|
||||
vpn_egress_gateway_resteer_timer_interval: 2min
|
||||
|
||||
# --- kubernetes node label (steering contract) ----------------------------------
|
||||
# The label the parallel CiliumEgressGatewayPolicy selects the gateway node by.
|
||||
|
||||
@@ -69,6 +69,17 @@
|
||||
- vpn-egress-down.sh
|
||||
notify: Restart vpn-egress tunnel
|
||||
|
||||
# The re-steer script is what keeps Cilium's cached egress interface in sync with tun0.
|
||||
# It is NOT notified to the tunnel: it is triggered BY the tunnel's up-script and by its
|
||||
# own timer, and re-deploying it must never bounce the tunnel.
|
||||
- name: Deploy the Cilium egress-gateway re-steer script
|
||||
ansible.builtin.template:
|
||||
src: vpn-egress-resteer.sh.j2
|
||||
dest: "{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0750"
|
||||
|
||||
- name: Deploy the scoped nftables kill-switch ruleset
|
||||
ansible.builtin.template:
|
||||
src: vpn-egress-killswitch.nft.j2
|
||||
@@ -100,6 +111,18 @@
|
||||
- Reload systemd
|
||||
- Restart vpn-egress tunnel
|
||||
|
||||
- name: Deploy the egress-gateway re-steer unit and its convergence timer
|
||||
ansible.builtin.template:
|
||||
src: "{{ item }}.j2"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- vpn-egress-resteer.service
|
||||
- vpn-egress-resteer.timer
|
||||
notify: Reload systemd
|
||||
|
||||
- name: Flush handlers so units/routing exist before enable+start
|
||||
ansible.builtin.meta: flush_handlers
|
||||
|
||||
@@ -115,6 +138,27 @@
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
# Enabled BEFORE the node label below: the label is what arms steering, and by then the
|
||||
# re-steer path must already exist so Cilium is told about the live tun0 rather than
|
||||
# caching whatever it resolves once and never revisiting it.
|
||||
- name: Enable and start the re-steer convergence timer
|
||||
ansible.builtin.systemd_service:
|
||||
name: vpn-egress-resteer.timer
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
# Publish the live uplink immediately rather than waiting for the timer's first tick, so a
|
||||
# first-time stand-up (or a re-run after a tunnel restart) converges within seconds.
|
||||
# `started`, NOT `restarted`: the tunnel's up-script fires this same unit on connect, and
|
||||
# `restarted` SIGTERMs that in-flight run (logging a spurious unit failure) instead of
|
||||
# letting it finish. The unit is a oneshot without RemainAfterExit, so it is `inactive`
|
||||
# once it has run and `started` genuinely re-runs it.
|
||||
- name: Re-steer Cilium onto the live tun0 now
|
||||
ansible.builtin.systemd_service:
|
||||
name: vpn-egress-resteer.service
|
||||
state: started
|
||||
changed_when: false
|
||||
|
||||
# --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ----
|
||||
- name: Read the current egress-gateway node label
|
||||
ansible.builtin.command:
|
||||
|
||||
@@ -8,15 +8,20 @@
|
||||
# then (2) remove the tunnel default from the dedicated table. The permanent
|
||||
# `blackhole default` in that table is LEFT in place as one fail-closed fallback.
|
||||
#
|
||||
# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED
|
||||
# here (we do NOT flush it). Keeping the last live tun0 address in
|
||||
# FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED here (we do
|
||||
# NOT flush it). Keeping every tunnel address this node has had in
|
||||
# `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule
|
||||
# `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the
|
||||
# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped
|
||||
# — defense-in-depth that does not depend on the stale SNAT source being non-routable or
|
||||
# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then
|
||||
# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects.
|
||||
# The set is scoped to vpn-class only; node and internet-class traffic are never in it.
|
||||
# tunnel is down. The up-script is ADDITIVE (2026-09-12 — it used to flush first, which
|
||||
# briefly left a previous tunnel address matched by nothing), so the set only ever grows
|
||||
# and only ever tightens the rule.
|
||||
#
|
||||
# SCOPE, HONESTLY: that rule covers NODE-ORIGINATED traffic sourced from a tunnel address.
|
||||
# It does NOT cover the egress-class=vpn POD traffic — Cilium's eBPF egress-gateway
|
||||
# datapath never presents those packets to netfilter (measured; see the kill-switch .nft
|
||||
# header). vpn-class pods fail closed because Cilium drops them when it cannot reach the
|
||||
# cached egress interface — verified with tcpdump on both the WAN and tun0 during a tunnel
|
||||
# stop and a tunnel restart: zero packets left either device.
|
||||
# The node's main-table WAN default is never touched here.
|
||||
set -uo pipefail
|
||||
|
||||
@@ -30,8 +35,8 @@ SRC="${ifconfig_local:-}"
|
||||
|
||||
# 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule.
|
||||
# NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the
|
||||
# last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit,
|
||||
# edge-independent fail-closed backstop until the up-script re-registers on reconnect.
|
||||
# tun0 addresses keeps the `oifname != tun0 drop` rule armed for node-originated
|
||||
# tunnel-sourced traffic until the up-script re-registers on reconnect.
|
||||
if [ -n "${SRC}" ]; then
|
||||
while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
|
||||
fi
|
||||
|
||||
@@ -1,40 +1,93 @@
|
||||
#!/usr/sbin/nft -f
|
||||
# {{ ansible_managed }}
|
||||
# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED.
|
||||
# ADR-0015 kill-switch — SCOPED TO tunnel-sourced traffic ONLY, fails CLOSED.
|
||||
#
|
||||
# =============================================================================
|
||||
# READ THIS BEFORE TRUSTING THIS FILE (measured 2026-09-12 on k3s-dev-3)
|
||||
# =============================================================================
|
||||
# This table does NOT, and CANNOT, guard the `egress-class=vpn` POD traffic that
|
||||
# ADR-0015 asks it to guard. It was measured, not reasoned about:
|
||||
#
|
||||
# A diagnostic nft table (counters only) was installed on the gateway node
|
||||
# alongside this one, counting every packet in `forward` and `postrouting` by
|
||||
# (oifname, saddr). While a vpn-class pod egressed through the gateway:
|
||||
# forward saddr=pod CIDR ................ 0 packets
|
||||
# forward saddr=PIA tunnel range ........ 0 packets
|
||||
# postrouting oif tun0 saddr=tunnel ...... 0 packets
|
||||
# postrouting oif WAN saddr=pod CIDR .... 0 packets
|
||||
# postrouting oif WAN (any) ............. 45166 packets <- hook IS live
|
||||
# and, as a positive control, a curl from the NODE itself out tun0 counted
|
||||
# postrouting oif tun0 saddr=tunnel ...... 16 packets
|
||||
#
|
||||
# So the hook is traversed and the matching works — but Cilium's egress-gateway
|
||||
# datapath never presents the steered pod packets to netfilter at all. With
|
||||
# `bpf.masquerade: true` the SNAT and the egress happen in eBPF, which
|
||||
# bpf_redirect()s the skb straight to the egress device: POSTROUTING is not
|
||||
# invoked for those packets, in either direction.
|
||||
#
|
||||
# WHAT ACTUALLY MAKES vpn-class FAIL CLOSED (also measured):
|
||||
# Cilium caches the gateway's (egress IP, egress ifindex) when it resolves this
|
||||
# policy. If the cached ifindex no longer exists — which is exactly what a tunnel
|
||||
# restart produces, tun0 being re-created with a NEW ifindex — the eBPF redirect
|
||||
# fails and the packet is DROPPED. Verified with tcpdump on BOTH the WAN and tun0
|
||||
# during a tunnel restart and during a full stop: 0 packets left either device.
|
||||
# There is no leak, but the mechanism is Cilium's, not this table's.
|
||||
#
|
||||
# WHAT THIS TABLE IS STILL GOOD FOR:
|
||||
# NODE-ORIGINATED traffic sourced from a tunnel address (the 16-packet control
|
||||
# above) — e.g. a host-side helper bound to tun0 — which DOES traverse netfilter.
|
||||
# That is a real, if narrow, guarantee, and it is the only one claimed here.
|
||||
# Every rule carries a `counter` so the claim is falsifiable at any moment:
|
||||
# nft list table inet {{ vpn_egress_gateway_nft_table }}
|
||||
# If `killswitch_drop` ever becomes non-zero, something (a Cilium datapath
|
||||
# change, bpf.masquerade turned off) has put this traffic back into netfilter —
|
||||
# read that as new information, not as reassurance.
|
||||
#
|
||||
# See platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-12.
|
||||
# =============================================================================
|
||||
#
|
||||
# Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules.
|
||||
# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class
|
||||
# traffic matched below; node traffic and internet-class traffic are untouched and keep
|
||||
# egressing the WAN (fail OPEN — invariant 4). It matches on:
|
||||
# * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs
|
||||
# vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script).
|
||||
# This IS the egress-class=vpn traffic and nothing else.
|
||||
# * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent
|
||||
# backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark.
|
||||
# The base chain is policy ACCEPT: this table can ONLY ever drop traffic whose source is
|
||||
# in the set below; node traffic and internet-class traffic are untouched and keep
|
||||
# egressing the WAN (fail OPEN — invariant 4).
|
||||
|
||||
# Idempotent reload. `table` + `flush table` clears the RULES only — a named set and its
|
||||
# elements are not chains, so the live tunnel address SURVIVES a reload and the guard is
|
||||
# not disarmed by an Ansible run. (A plain `nft -f` without this would append a second
|
||||
# copy of every rule on each load.)
|
||||
table inet {{ vpn_egress_gateway_nft_table }} { }
|
||||
flush table inet {{ vpn_egress_gateway_nft_table }}
|
||||
|
||||
table inet {{ vpn_egress_gateway_nft_table }} {
|
||||
# Live tunnel source address(es). Empty whenever the tunnel is down => the two
|
||||
# saddr rules below match nothing => vpn-class has no path at all (fail closed).
|
||||
# Every tunnel source address this gateway has had. The OpenVPN up-script ADDS the
|
||||
# live address on each connect and never flushes (see vpn-egress-up.sh): an element
|
||||
# can only ever tighten the guard, and retaining previous addresses means a packet
|
||||
# still sourced from an older tunnel address is dropped rather than unmatched.
|
||||
# Empty before the very first successful connect => the rules below match nothing,
|
||||
# and vpn-class has no path at all (fail closed, by Cilium's drop — see above).
|
||||
set {{ vpn_egress_gateway_nft_src_set }} {
|
||||
type ipv4_addr
|
||||
}
|
||||
|
||||
chain postrouting {
|
||||
# oifname is resolved at postrouting; SNAT/masquerade has already applied here,
|
||||
# so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address.
|
||||
# oifname is resolved at postrouting, after any netfilter SNAT.
|
||||
type filter hook postrouting priority 100; policy accept;
|
||||
|
||||
# Tunnel UP: let the steered vpn-class traffic out tun0.
|
||||
oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept
|
||||
# Tunnel UP: let tunnel-sourced traffic out tun0.
|
||||
oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} counter accept comment "killswitch_accept"
|
||||
|
||||
# FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other
|
||||
# interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it
|
||||
# can never match node or internet-class traffic.
|
||||
ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
|
||||
# FAIL CLOSED: tunnel-sourced traffic trying to leave ANY other interface (i.e.
|
||||
# the WAN) is dropped — never leaks. Scoped by the set, so it can never match
|
||||
# node or internet-class traffic. The counter is the proof-of-life for the whole
|
||||
# table: see the header for why it reads 0 for POD traffic by construction.
|
||||
ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" counter drop comment "killswitch_drop"
|
||||
|
||||
# Tunnel-down-independent backstop (contract hook, see role README): if vpn-class
|
||||
# packets carry the agreed fwmark, drop any not leaving via tun0 even when the set
|
||||
# is empty (tunnel fully down). Harmless if the Cilium side does not set the mark.
|
||||
meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
|
||||
# REMOVED 2026-09-12 — the `meta mark 0x1c8 oifname != tun0 drop` rule. It was a
|
||||
# "contract hook" for a fwmark that nothing stamps: Cilium OSS does not mark
|
||||
# egress-gateway traffic with 0x1c8 (the only Cilium mark visible on the gateway
|
||||
# node is 0x200/0xf00, which is the L7-proxy rule -> table 2004, unrelated), and
|
||||
# the traffic does not reach netfilter anyway. A dead rule matching an arbitrary
|
||||
# mark is a hazard, not a backstop: any future component that happens to use
|
||||
# 0x1c8 would have had its egress silently dropped.
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# {{ ansible_managed }}
|
||||
# Publishes this node's CURRENT tun0 (address + ifindex) as a node label so Cilium
|
||||
# re-resolves the CiliumEgressGatewayPolicy's `interface: tun0` (ADR-0015). Without this,
|
||||
# Cilium keeps a cached (egress IP, egress ifindex) from the FIRST resolution and vpn-class
|
||||
# egress is silently blackholed after every tunnel restart or reboot — measured, see the
|
||||
# script header and the DECISIONS-LOG entry of 2026-09-12.
|
||||
#
|
||||
# Triggered by the OpenVPN up-script (`systemctl start --no-block`) on every connect, and
|
||||
# by vpn-egress-resteer.timer as the convergence backstop. Deliberately NOT ordered
|
||||
# Before/After the tunnel: it must never be able to block or fail the tunnel.
|
||||
[Unit]
|
||||
Description=Re-steer Cilium Egress Gateway onto the live tun0 — ADR-0015
|
||||
Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
|
||||
After=network-online.target
|
||||
# The kill-switch owns the nft table this script re-registers the tunnel source into.
|
||||
After=vpn-egress-killswitch.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart={{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
|
||||
# Bounded: {{ vpn_egress_gateway_resteer_retries }} attempts x {{ vpn_egress_gateway_resteer_retry_seconds }}s waiting for the kube-API, plus slack.
|
||||
TimeoutStartSec={{ (vpn_egress_gateway_resteer_retries | int * vpn_egress_gateway_resteer_retry_seconds | int) + 60 }}
|
||||
# A failure here means "the kube-API was unreachable"; the timer retries. Do not let a
|
||||
# failed re-steer mark the whole gateway degraded.
|
||||
SuccessExitStatus=0 1
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# {{ ansible_managed }}
|
||||
# Re-steer the Cilium Egress Gateway onto THIS node's CURRENT tun0 (ADR-0015).
|
||||
#
|
||||
# WHY THIS EXISTS (measured on k3s-dev-3, 2026-09-12 — see DECISIONS-LOG):
|
||||
# Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` once, caching an
|
||||
# (egress IP, egress ifindex) pair in its BPF egress map, and does NOT re-resolve when
|
||||
# tun0 changes. Observed directly: after `systemctl restart vpn-egress-tunnel` the node
|
||||
# had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986, and
|
||||
# vpn-class pod egress was blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — no
|
||||
# leak, but no service). `egress-gateway-reconciliation-trigger-interval: 1s` does NOT
|
||||
# cover netlink link events; it only coalesces reconciliations that k8s object events
|
||||
# already triggered.
|
||||
#
|
||||
# Upstream remedy: "You can force a fresh selection by re-applying the Egress Gateway
|
||||
# policy." Any k8s event that makes the gateway node re-process the policy does it. We
|
||||
# use a NODE LABEL rather than touching the policy object, because:
|
||||
# * the policy is Flux-managed GitOps — writing to it from a node invites drift
|
||||
# fights and ownership churn; the node's labels are already this role's territory;
|
||||
# * it was verified to work: setting an unrelated label on k3s-dev-3 moved the cached
|
||||
# ifindex 1986 -> 1987 within 10s and vpn-class egress recovered (PIA IP returned);
|
||||
# * the label VALUE is the uplink identity, so `kubectl get node -L ...` shows at a
|
||||
# glance which tun0 Cilium was last told about. Diagnosis needs no exec into a pod.
|
||||
#
|
||||
# Idempotent: if the label already records the live uplink, this exits without touching
|
||||
# anything, so the timer can run as often as it likes.
|
||||
set -uo pipefail
|
||||
|
||||
IFACE="{{ vpn_egress_gateway_tun_iface }}"
|
||||
TABLE_ID="{{ vpn_egress_gateway_route_table_id }}"
|
||||
RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}"
|
||||
TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}"
|
||||
NFT_TABLE="{{ vpn_egress_gateway_nft_table }}"
|
||||
NFT_SET="{{ vpn_egress_gateway_nft_src_set }}"
|
||||
NODE="{{ inventory_hostname }}"
|
||||
UPLINK_LABEL="{{ vpn_egress_gateway_uplink_label_key }}"
|
||||
RETRIES="{{ vpn_egress_gateway_resteer_retries }}"
|
||||
RETRY_SLEEP="{{ vpn_egress_gateway_resteer_retry_seconds }}"
|
||||
export KUBECONFIG="{{ vpn_egress_gateway_kubeconfig }}"
|
||||
|
||||
log() { echo "vpn-egress-resteer: $*"; }
|
||||
|
||||
# --- 1) what is the live uplink? -------------------------------------------------
|
||||
addr="$(ip -4 -o addr show dev "${IFACE}" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)"
|
||||
idx="$(cat "/sys/class/net/${IFACE}/ifindex" 2>/dev/null || true)"
|
||||
if [ -z "${addr}" ] || [ -z "${idx}" ]; then
|
||||
# Tunnel down. Nothing to publish — and deliberately NOTHING is torn down here either:
|
||||
# vpn-class is already fail-closed (Cilium drops when the cached ifindex is gone), and
|
||||
# the kill-switch set is retained on purpose. The next successful connect re-runs this.
|
||||
log "${IFACE} has no address yet (tunnel down) — nothing to re-steer"
|
||||
exit 0
|
||||
fi
|
||||
# A k8s label value must be <=63 chars of [A-Za-z0-9] plus - _ . in the middle. An IPv4
|
||||
# address and an ifindex satisfy that; the separator is a dash.
|
||||
want="${addr}-${idx}"
|
||||
|
||||
# --- 2) re-assert the host side (idempotent; repairs a kill-switch reload) --------
|
||||
# `flush table` in the .nft file keeps set elements, but a full ExecStop/ExecStart cycle
|
||||
# (or a hand-run `nft delete table`) does not, so re-register unconditionally.
|
||||
nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${addr} }" 2>/dev/null || \
|
||||
log "WARNING could not add ${addr} to ${NFT_TABLE}/${NFT_SET} (is the kill-switch loaded?)"
|
||||
if ! grep -q "from ${addr} lookup" <<<"$(ip rule show)"; then
|
||||
ip rule add from "${addr}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null || true
|
||||
fi
|
||||
ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" 2>/dev/null || true
|
||||
|
||||
# --- 3) has Cilium already been told about THIS uplink? ---------------------------
|
||||
# The label is the record of what we last published. Retried because at boot the kube-API
|
||||
# is routinely not up yet when OpenVPN connects.
|
||||
i=0
|
||||
while :; do
|
||||
i=$((i + 1))
|
||||
# -o json + an exact-string match rather than jsonpath: the label key contains dots and
|
||||
# a slash, which jsonpath needs escaped differently across kubectl versions, and getting
|
||||
# that wrong would silently read "" and re-label on every timer tick.
|
||||
node_json="$(k3s kubectl get node "${NODE}" -o json 2>/dev/null)"
|
||||
if [ -n "${node_json}" ]; then
|
||||
# Here-strings, NOT pipes: with `set -o pipefail`, `grep -q` exiting early on a match
|
||||
# SIGPIPEs the writer and the pipeline reports 141 — i.e. a successful match would read
|
||||
# as a failure and this script would re-label on every single run.
|
||||
if grep -qF "\"${UPLINK_LABEL}\": \"${want}\"" <<<"${node_json}"; then
|
||||
log "uplink unchanged (${want}) — Cilium already re-steered, nothing to do"
|
||||
exit 0
|
||||
fi
|
||||
have="$(grep -F "\"${UPLINK_LABEL}\": " <<<"${node_json}" | head -1 | cut -d'"' -f4)"
|
||||
log "uplink changed: label='${have:-<unset>}' live='${want}' — re-steering"
|
||||
if k3s kubectl label node "${NODE}" "${UPLINK_LABEL}=${want}" --overwrite >/dev/null 2>&1; then
|
||||
log "re-steered: ${NODE} ${UPLINK_LABEL}=${want}"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
if [ "${i}" -ge "${RETRIES}" ]; then
|
||||
log "ERROR kube-API unreachable or label write failed after ${i} attempts — the timer will retry"
|
||||
exit 1
|
||||
fi
|
||||
sleep "${RETRY_SLEEP}"
|
||||
done
|
||||
@@ -0,0 +1,19 @@
|
||||
# {{ ansible_managed }}
|
||||
# Convergence backstop for the event-driven re-steer. The OpenVPN up-script fires
|
||||
# vpn-egress-resteer.service on every connect, but that call can be lost (kube-API not up
|
||||
# yet at boot and all in-script retries exhausted, an OpenVPN soft restart whose up-script
|
||||
# ran before k3s, a hand-run `nft delete table`). This timer makes the gateway converge on
|
||||
# its own instead of needing an operator. The script is a no-op when the published label
|
||||
# already matches the live tun0, so running it often is free.
|
||||
[Unit]
|
||||
Description=Periodic re-steer of the Cilium Egress Gateway onto the live tun0 — ADR-0015
|
||||
Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
|
||||
|
||||
[Timer]
|
||||
OnBootSec={{ vpn_egress_gateway_resteer_timer_boot }}
|
||||
OnUnitInactiveSec={{ vpn_egress_gateway_resteer_timer_interval }}
|
||||
AccuracySec=15s
|
||||
Unit=vpn-egress-resteer.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -35,9 +35,34 @@ ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}"
|
||||
while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
|
||||
ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}"
|
||||
|
||||
# Register the live tunnel source with the kill-switch set so the nft backstop can scope
|
||||
# to exactly this traffic. Flush any stale members first (address changes on reconnect).
|
||||
nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true
|
||||
# Register the live tunnel source with the kill-switch set. ADDITIVE ON PURPOSE — the set
|
||||
# is NEVER flushed here (it used to be, see the DECISIONS-LOG entry of 2026-09-12).
|
||||
# `nft add element` is idempotent, and every address that has ever been this tunnel's
|
||||
# source can only ever TIGHTEN the `oifname != tun0 drop` rule. Flushing first meant that
|
||||
# between a reconnect and Cilium re-resolving its cached egress IP there was a moment
|
||||
# where a packet still sourced from the PREVIOUS address matched neither the ip-rule nor
|
||||
# the drop rule. (Measurement since showed that window cannot leak for POD traffic,
|
||||
# because that traffic never enters netfilter at all — see the kill-switch .nft header —
|
||||
# but node-originated tunnel traffic DOES, and additive costs nothing.)
|
||||
nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }"
|
||||
|
||||
# THE LOAD-BEARING FIX (2026-09-12): tell Cilium that this gateway's uplink changed.
|
||||
#
|
||||
# Cilium resolves the CiliumEgressGatewayPolicy's `interface: tun0` ONCE, into a cached
|
||||
# (egress IP, egress ifindex) pair, and — measured on k3s-dev-3 — does NOT re-resolve it
|
||||
# when tun0 changes underneath, not even with `egress-gateway-reconciliation-trigger-
|
||||
# interval: 1s`. Upstream says as much: "it does not automatically respond to a change in
|
||||
# the gateway node's network configuration... You can force a fresh selection by
|
||||
# re-applying the Egress Gateway policy."
|
||||
#
|
||||
# Every tunnel restart re-creates tun0 with a NEW ifindex, so WITHOUT this the gateway is
|
||||
# silently dead after its first reconnect or reboot: vpn-class egress is blackholed (no
|
||||
# leak — the eBPF redirect to the dead ifindex just fails — but no service, and no alert).
|
||||
# Re-steering is done out-of-band by vpn-egress-resteer.service so this hook can never
|
||||
# block or fail the tunnel: OpenVPN's up-script must stay fast and must not depend on the
|
||||
# kube-API being reachable yet (at boot it usually is not — the resteer unit retries, and
|
||||
# its timer converges anyway).
|
||||
systemctl start --no-block vpn-egress-resteer.service || \
|
||||
echo "vpn-egress-up: WARNING could not trigger vpn-egress-resteer.service (timer will converge)" >&2
|
||||
|
||||
echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active"
|
||||
Reference in new issue
Block a user