From 6f698f59af2835edf412575500d2e7bb28a4cfdd Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:38:27 -0500 Subject: [PATCH 1/6] =?UTF-8?q?fix(vpn):=20Cilium=20never=20re-resolves=20?= =?UTF-8?q?tun0=20=E2=80=94=20the=20gateway=20dies=20on=20every=20reconnec?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "-" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 --- .../vpn_egress_gateway/defaults/main.yml | 40 ++++++-- .../roles/vpn_egress_gateway/tasks/main.yml | 44 +++++++++ .../templates/vpn-egress-down.sh.j2 | 23 +++-- .../templates/vpn-egress-killswitch.nft.j2 | 99 ++++++++++++++----- .../templates/vpn-egress-resteer.service.j2 | 25 +++++ .../templates/vpn-egress-resteer.sh.j2 | 97 ++++++++++++++++++ .../templates/vpn-egress-resteer.timer.j2 | 19 ++++ .../templates/vpn-egress-up.sh.j2 | 31 +++++- 8 files changed, 334 insertions(+), 44 deletions(-) create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 diff --git a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml index 8e82ef0..359d1ea 100644 --- a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml @@ -46,17 +46,39 @@ vpn_egress_gateway_blackhole_metric: 4000 # (Cilium VXLAN 1450 → needed 1300) no shrink is required by default. vpn_egress_gateway_tun_mtu: "" -# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ---------- +# --- nftables kill-switch ------------------------------------------------------- +# SCOPE, HONESTLY (measured 2026-09-12, see the .nft template header): this table guards +# NODE-ORIGINATED traffic sourced from a tunnel address. It does NOT and cannot guard the +# egress-class=vpn POD traffic, because Cilium's eBPF egress-gateway datapath +# (bpf.masquerade) redirects those packets to the egress device without ever invoking +# netfilter POSTROUTING. vpn-class fails closed via Cilium's own drop, not via this table. vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch -# Named set holding the LIVE tun0 source address (maintained by the up/down scripts). -# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway -# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never -# in this set and is never matched (fails OPEN toward the WAN). +# Named set of tunnel source addresses, ADDITIVE (the up-script adds, never flushes; the +# down-script retains). Every element only tightens the `oifname != tun0 drop` rule. vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src -# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets -# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an -# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation". -vpn_egress_gateway_fwmark: "0x1c8" +# REMOVED 2026-09-12: vpn_egress_gateway_fwmark ("0x1c8"). It backed a kill-switch rule +# for a mark that nothing stamps — Cilium OSS does not mark egress-gateway traffic with +# it (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule), and the +# traffic never reaches netfilter anyway. A dead rule matching an arbitrary mark is a +# hazard, not a backstop, so both the rule and the variable are gone. + +# --- re-steer: keep Cilium's cached egress interface in sync with tun0 ----------- +# Cilium resolves `interface: tun0` ONCE into a cached (egress IP, egress ifindex) and does +# not re-resolve when tun0 changes; every tunnel restart gives tun0 a NEW ifindex, which +# silently blackholes vpn-class egress. The re-steer script publishes the live uplink as a +# node label; that node update makes the gateway node re-process the policy. Verified on +# k3s-dev-3: cached ifindex moved 1986 -> 1987 within 10s and egress recovered. +# The label VALUE is "-", so `kubectl get nodes -L ` shows +# which uplink Cilium was last told about. +vpn_egress_gateway_uplink_label_key: homelab.keithjackson.dev/egress-uplink +# The kube-API is routinely not reachable yet when OpenVPN connects during boot, so the +# script retries. Bounded so a genuinely broken API does not hang a systemd job forever. +vpn_egress_gateway_resteer_retries: 30 +vpn_egress_gateway_resteer_retry_seconds: 10 +# Convergence backstop if the event-driven trigger is ever lost (no-op when already in +# sync, so it is cheap to run often). +vpn_egress_gateway_resteer_timer_boot: 2min +vpn_egress_gateway_resteer_timer_interval: 2min # --- kubernetes node label (steering contract) ---------------------------------- # The label the parallel CiliumEgressGatewayPolicy selects the gateway node by. diff --git a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml index 206786d..401a13f 100644 --- a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml @@ -69,6 +69,17 @@ - vpn-egress-down.sh notify: Restart vpn-egress tunnel +# The re-steer script is what keeps Cilium's cached egress interface in sync with tun0. +# It is NOT notified to the tunnel: it is triggered BY the tunnel's up-script and by its +# own timer, and re-deploying it must never bounce the tunnel. +- name: Deploy the Cilium egress-gateway re-steer script + ansible.builtin.template: + src: vpn-egress-resteer.sh.j2 + dest: "{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh" + owner: root + group: root + mode: "0750" + - name: Deploy the scoped nftables kill-switch ruleset ansible.builtin.template: src: vpn-egress-killswitch.nft.j2 @@ -100,6 +111,18 @@ - Reload systemd - Restart vpn-egress tunnel +- name: Deploy the egress-gateway re-steer unit and its convergence timer + ansible.builtin.template: + src: "{{ item }}.j2" + dest: "/etc/systemd/system/{{ item }}" + owner: root + group: root + mode: "0644" + loop: + - vpn-egress-resteer.service + - vpn-egress-resteer.timer + notify: Reload systemd + - name: Flush handlers so units/routing exist before enable+start ansible.builtin.meta: flush_handlers @@ -115,6 +138,27 @@ enabled: true state: started +# Enabled BEFORE the node label below: the label is what arms steering, and by then the +# re-steer path must already exist so Cilium is told about the live tun0 rather than +# caching whatever it resolves once and never revisiting it. +- name: Enable and start the re-steer convergence timer + ansible.builtin.systemd_service: + name: vpn-egress-resteer.timer + enabled: true + state: started + +# Publish the live uplink immediately rather than waiting for the timer's first tick, so a +# first-time stand-up (or a re-run after a tunnel restart) converges within seconds. +# `started`, NOT `restarted`: the tunnel's up-script fires this same unit on connect, and +# `restarted` SIGTERMs that in-flight run (logging a spurious unit failure) instead of +# letting it finish. The unit is a oneshot without RemainAfterExit, so it is `inactive` +# once it has run and `started` genuinely re-runs it. +- name: Re-steer Cilium onto the live tun0 now + ansible.builtin.systemd_service: + name: vpn-egress-resteer.service + state: started + changed_when: false + # --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ---- - name: Read the current egress-gateway node label ansible.builtin.command: diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 index 21ec2ba..14cd8ff 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 @@ -8,15 +8,20 @@ # then (2) remove the tunnel default from the dedicated table. The permanent # `blackhole default` in that table is LEFT in place as one fail-closed fallback. # -# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED -# here (we do NOT flush it). Keeping the last live tun0 address in +# FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED here (we do +# NOT flush it). Keeping every tunnel address this node has had in # `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule # `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the -# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped -# — defense-in-depth that does not depend on the stale SNAT source being non-routable or -# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then -# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects. -# The set is scoped to vpn-class only; node and internet-class traffic are never in it. +# tunnel is down. The up-script is ADDITIVE (2026-09-12 — it used to flush first, which +# briefly left a previous tunnel address matched by nothing), so the set only ever grows +# and only ever tightens the rule. +# +# SCOPE, HONESTLY: that rule covers NODE-ORIGINATED traffic sourced from a tunnel address. +# It does NOT cover the egress-class=vpn POD traffic — Cilium's eBPF egress-gateway +# datapath never presents those packets to netfilter (measured; see the kill-switch .nft +# header). vpn-class pods fail closed because Cilium drops them when it cannot reach the +# cached egress interface — verified with tcpdump on both the WAN and tun0 during a tunnel +# stop and a tunnel restart: zero packets left either device. # The node's main-table WAN default is never touched here. set -uo pipefail @@ -30,8 +35,8 @@ SRC="${ifconfig_local:-}" # 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule. # NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the -# last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit, -# edge-independent fail-closed backstop until the up-script re-registers on reconnect. +# tun0 addresses keeps the `oifname != tun0 drop` rule armed for node-originated +# tunnel-sourced traffic until the up-script re-registers on reconnect. if [ -n "${SRC}" ]; then while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done fi diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 index 134c6ad..d3faa48 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 @@ -1,40 +1,93 @@ #!/usr/sbin/nft -f # {{ ansible_managed }} -# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED. +# ADR-0015 kill-switch — SCOPED TO tunnel-sourced traffic ONLY, fails CLOSED. +# +# ============================================================================= +# READ THIS BEFORE TRUSTING THIS FILE (measured 2026-09-12 on k3s-dev-3) +# ============================================================================= +# This table does NOT, and CANNOT, guard the `egress-class=vpn` POD traffic that +# ADR-0015 asks it to guard. It was measured, not reasoned about: +# +# A diagnostic nft table (counters only) was installed on the gateway node +# alongside this one, counting every packet in `forward` and `postrouting` by +# (oifname, saddr). While a vpn-class pod egressed through the gateway: +# forward saddr=pod CIDR ................ 0 packets +# forward saddr=PIA tunnel range ........ 0 packets +# postrouting oif tun0 saddr=tunnel ...... 0 packets +# postrouting oif WAN saddr=pod CIDR .... 0 packets +# postrouting oif WAN (any) ............. 45166 packets <- hook IS live +# and, as a positive control, a curl from the NODE itself out tun0 counted +# postrouting oif tun0 saddr=tunnel ...... 16 packets +# +# So the hook is traversed and the matching works — but Cilium's egress-gateway +# datapath never presents the steered pod packets to netfilter at all. With +# `bpf.masquerade: true` the SNAT and the egress happen in eBPF, which +# bpf_redirect()s the skb straight to the egress device: POSTROUTING is not +# invoked for those packets, in either direction. +# +# WHAT ACTUALLY MAKES vpn-class FAIL CLOSED (also measured): +# Cilium caches the gateway's (egress IP, egress ifindex) when it resolves this +# policy. If the cached ifindex no longer exists — which is exactly what a tunnel +# restart produces, tun0 being re-created with a NEW ifindex — the eBPF redirect +# fails and the packet is DROPPED. Verified with tcpdump on BOTH the WAN and tun0 +# during a tunnel restart and during a full stop: 0 packets left either device. +# There is no leak, but the mechanism is Cilium's, not this table's. +# +# WHAT THIS TABLE IS STILL GOOD FOR: +# NODE-ORIGINATED traffic sourced from a tunnel address (the 16-packet control +# above) — e.g. a host-side helper bound to tun0 — which DOES traverse netfilter. +# That is a real, if narrow, guarantee, and it is the only one claimed here. +# Every rule carries a `counter` so the claim is falsifiable at any moment: +# nft list table inet {{ vpn_egress_gateway_nft_table }} +# If `killswitch_drop` ever becomes non-zero, something (a Cilium datapath +# change, bpf.masquerade turned off) has put this traffic back into netfilter — +# read that as new information, not as reassurance. +# +# See platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-12. +# ============================================================================= # # Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules. -# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class -# traffic matched below; node traffic and internet-class traffic are untouched and keep -# egressing the WAN (fail OPEN — invariant 4). It matches on: -# * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs -# vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script). -# This IS the egress-class=vpn traffic and nothing else. -# * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent -# backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark. +# The base chain is policy ACCEPT: this table can ONLY ever drop traffic whose source is +# in the set below; node traffic and internet-class traffic are untouched and keep +# egressing the WAN (fail OPEN — invariant 4). + +# Idempotent reload. `table` + `flush table` clears the RULES only — a named set and its +# elements are not chains, so the live tunnel address SURVIVES a reload and the guard is +# not disarmed by an Ansible run. (A plain `nft -f` without this would append a second +# copy of every rule on each load.) +table inet {{ vpn_egress_gateway_nft_table }} { } +flush table inet {{ vpn_egress_gateway_nft_table }} table inet {{ vpn_egress_gateway_nft_table }} { - # Live tunnel source address(es). Empty whenever the tunnel is down => the two - # saddr rules below match nothing => vpn-class has no path at all (fail closed). + # Every tunnel source address this gateway has had. The OpenVPN up-script ADDS the + # live address on each connect and never flushes (see vpn-egress-up.sh): an element + # can only ever tighten the guard, and retaining previous addresses means a packet + # still sourced from an older tunnel address is dropped rather than unmatched. + # Empty before the very first successful connect => the rules below match nothing, + # and vpn-class has no path at all (fail closed, by Cilium's drop — see above). set {{ vpn_egress_gateway_nft_src_set }} { type ipv4_addr } chain postrouting { - # oifname is resolved at postrouting; SNAT/masquerade has already applied here, - # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address. + # oifname is resolved at postrouting, after any netfilter SNAT. type filter hook postrouting priority 100; policy accept; - # Tunnel UP: let the steered vpn-class traffic out tun0. - oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept + # Tunnel UP: let tunnel-sourced traffic out tun0. + oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} counter accept comment "killswitch_accept" - # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other - # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it - # can never match node or internet-class traffic. - ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # FAIL CLOSED: tunnel-sourced traffic trying to leave ANY other interface (i.e. + # the WAN) is dropped — never leaks. Scoped by the set, so it can never match + # node or internet-class traffic. The counter is the proof-of-life for the whole + # table: see the header for why it reads 0 for POD traffic by construction. + ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" counter drop comment "killswitch_drop" - # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class - # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set - # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark. - meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + # REMOVED 2026-09-12 — the `meta mark 0x1c8 oifname != tun0 drop` rule. It was a + # "contract hook" for a fwmark that nothing stamps: Cilium OSS does not mark + # egress-gateway traffic with 0x1c8 (the only Cilium mark visible on the gateway + # node is 0x200/0xf00, which is the L7-proxy rule -> table 2004, unrelated), and + # the traffic does not reach netfilter anyway. A dead rule matching an arbitrary + # mark is a hazard, not a backstop: any future component that happens to use + # 0x1c8 would have had its egress silently dropped. } } diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 new file mode 100644 index 0000000..179cc54 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.service.j2 @@ -0,0 +1,25 @@ +# {{ ansible_managed }} +# Publishes this node's CURRENT tun0 (address + ifindex) as a node label so Cilium +# re-resolves the CiliumEgressGatewayPolicy's `interface: tun0` (ADR-0015). Without this, +# Cilium keeps a cached (egress IP, egress ifindex) from the FIRST resolution and vpn-class +# egress is silently blackholed after every tunnel restart or reboot — measured, see the +# script header and the DECISIONS-LOG entry of 2026-09-12. +# +# Triggered by the OpenVPN up-script (`systemctl start --no-block`) on every connect, and +# by vpn-egress-resteer.timer as the convergence backstop. Deliberately NOT ordered +# Before/After the tunnel: it must never be able to block or fail the tunnel. +[Unit] +Description=Re-steer Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +After=network-online.target +# The kill-switch owns the nft table this script re-registers the tunnel source into. +After=vpn-egress-killswitch.service + +[Service] +Type=oneshot +ExecStart={{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh +# Bounded: {{ vpn_egress_gateway_resteer_retries }} attempts x {{ vpn_egress_gateway_resteer_retry_seconds }}s waiting for the kube-API, plus slack. +TimeoutStartSec={{ (vpn_egress_gateway_resteer_retries | int * vpn_egress_gateway_resteer_retry_seconds | int) + 60 }} +# A failure here means "the kube-API was unreachable"; the timer retries. Do not let a +# failed re-steer mark the whole gateway degraded. +SuccessExitStatus=0 1 diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 new file mode 100644 index 0000000..bdd62dc --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.sh.j2 @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# {{ ansible_managed }} +# Re-steer the Cilium Egress Gateway onto THIS node's CURRENT tun0 (ADR-0015). +# +# WHY THIS EXISTS (measured on k3s-dev-3, 2026-09-12 — see DECISIONS-LOG): +# Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` once, caching an +# (egress IP, egress ifindex) pair in its BPF egress map, and does NOT re-resolve when +# tun0 changes. Observed directly: after `systemctl restart vpn-egress-tunnel` the node +# had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986, and +# vpn-class pod egress was blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — no +# leak, but no service). `egress-gateway-reconciliation-trigger-interval: 1s` does NOT +# cover netlink link events; it only coalesces reconciliations that k8s object events +# already triggered. +# +# Upstream remedy: "You can force a fresh selection by re-applying the Egress Gateway +# policy." Any k8s event that makes the gateway node re-process the policy does it. We +# use a NODE LABEL rather than touching the policy object, because: +# * the policy is Flux-managed GitOps — writing to it from a node invites drift +# fights and ownership churn; the node's labels are already this role's territory; +# * it was verified to work: setting an unrelated label on k3s-dev-3 moved the cached +# ifindex 1986 -> 1987 within 10s and vpn-class egress recovered (PIA IP returned); +# * the label VALUE is the uplink identity, so `kubectl get node -L ...` shows at a +# glance which tun0 Cilium was last told about. Diagnosis needs no exec into a pod. +# +# Idempotent: if the label already records the live uplink, this exits without touching +# anything, so the timer can run as often as it likes. +set -uo pipefail + +IFACE="{{ vpn_egress_gateway_tun_iface }}" +TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" +RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" +TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}" +NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" +NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +NODE="{{ inventory_hostname }}" +UPLINK_LABEL="{{ vpn_egress_gateway_uplink_label_key }}" +RETRIES="{{ vpn_egress_gateway_resteer_retries }}" +RETRY_SLEEP="{{ vpn_egress_gateway_resteer_retry_seconds }}" +export KUBECONFIG="{{ vpn_egress_gateway_kubeconfig }}" + +log() { echo "vpn-egress-resteer: $*"; } + +# --- 1) what is the live uplink? ------------------------------------------------- +addr="$(ip -4 -o addr show dev "${IFACE}" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)" +idx="$(cat "/sys/class/net/${IFACE}/ifindex" 2>/dev/null || true)" +if [ -z "${addr}" ] || [ -z "${idx}" ]; then + # Tunnel down. Nothing to publish — and deliberately NOTHING is torn down here either: + # vpn-class is already fail-closed (Cilium drops when the cached ifindex is gone), and + # the kill-switch set is retained on purpose. The next successful connect re-runs this. + log "${IFACE} has no address yet (tunnel down) — nothing to re-steer" + exit 0 +fi +# A k8s label value must be <=63 chars of [A-Za-z0-9] plus - _ . in the middle. An IPv4 +# address and an ifindex satisfy that; the separator is a dash. +want="${addr}-${idx}" + +# --- 2) re-assert the host side (idempotent; repairs a kill-switch reload) -------- +# `flush table` in the .nft file keeps set elements, but a full ExecStop/ExecStart cycle +# (or a hand-run `nft delete table`) does not, so re-register unconditionally. +nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${addr} }" 2>/dev/null || \ + log "WARNING could not add ${addr} to ${NFT_TABLE}/${NFT_SET} (is the kill-switch loaded?)" +if ! grep -q "from ${addr} lookup" <<<"$(ip rule show)"; then + ip rule add from "${addr}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null || true +fi +ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" 2>/dev/null || true + +# --- 3) has Cilium already been told about THIS uplink? --------------------------- +# The label is the record of what we last published. Retried because at boot the kube-API +# is routinely not up yet when OpenVPN connects. +i=0 +while :; do + i=$((i + 1)) + # -o json + an exact-string match rather than jsonpath: the label key contains dots and + # a slash, which jsonpath needs escaped differently across kubectl versions, and getting + # that wrong would silently read "" and re-label on every timer tick. + node_json="$(k3s kubectl get node "${NODE}" -o json 2>/dev/null)" + if [ -n "${node_json}" ]; then + # Here-strings, NOT pipes: with `set -o pipefail`, `grep -q` exiting early on a match + # SIGPIPEs the writer and the pipeline reports 141 — i.e. a successful match would read + # as a failure and this script would re-label on every single run. + if grep -qF "\"${UPLINK_LABEL}\": \"${want}\"" <<<"${node_json}"; then + log "uplink unchanged (${want}) — Cilium already re-steered, nothing to do" + exit 0 + fi + have="$(grep -F "\"${UPLINK_LABEL}\": " <<<"${node_json}" | head -1 | cut -d'"' -f4)" + log "uplink changed: label='${have:-}' live='${want}' — re-steering" + if k3s kubectl label node "${NODE}" "${UPLINK_LABEL}=${want}" --overwrite >/dev/null 2>&1; then + log "re-steered: ${NODE} ${UPLINK_LABEL}=${want}" + exit 0 + fi + fi + if [ "${i}" -ge "${RETRIES}" ]; then + log "ERROR kube-API unreachable or label write failed after ${i} attempts — the timer will retry" + exit 1 + fi + sleep "${RETRY_SLEEP}" +done diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 new file mode 100644 index 0000000..f2fb55a --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-resteer.timer.j2 @@ -0,0 +1,19 @@ +# {{ ansible_managed }} +# Convergence backstop for the event-driven re-steer. The OpenVPN up-script fires +# vpn-egress-resteer.service on every connect, but that call can be lost (kube-API not up +# yet at boot and all in-script retries exhausted, an OpenVPN soft restart whose up-script +# ran before k3s, a hand-run `nft delete table`). This timer makes the gateway converge on +# its own instead of needing an operator. The script is a no-op when the published label +# already matches the live tun0, so running it often is free. +[Unit] +Description=Periodic re-steer of the Cilium Egress Gateway onto the live tun0 — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh + +[Timer] +OnBootSec={{ vpn_egress_gateway_resteer_timer_boot }} +OnUnitInactiveSec={{ vpn_egress_gateway_resteer_timer_interval }} +AccuracySec=15s +Unit=vpn-egress-resteer.service + +[Install] +WantedBy=timers.target diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 index d108dbc..36a5f87 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 @@ -35,9 +35,34 @@ ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}" while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" -# Register the live tunnel source with the kill-switch set so the nft backstop can scope -# to exactly this traffic. Flush any stale members first (address changes on reconnect). -nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +# Register the live tunnel source with the kill-switch set. ADDITIVE ON PURPOSE — the set +# is NEVER flushed here (it used to be, see the DECISIONS-LOG entry of 2026-09-12). +# `nft add element` is idempotent, and every address that has ever been this tunnel's +# source can only ever TIGHTEN the `oifname != tun0 drop` rule. Flushing first meant that +# between a reconnect and Cilium re-resolving its cached egress IP there was a moment +# where a packet still sourced from the PREVIOUS address matched neither the ip-rule nor +# the drop rule. (Measurement since showed that window cannot leak for POD traffic, +# because that traffic never enters netfilter at all — see the kill-switch .nft header — +# but node-originated tunnel traffic DOES, and additive costs nothing.) nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }" +# THE LOAD-BEARING FIX (2026-09-12): tell Cilium that this gateway's uplink changed. +# +# Cilium resolves the CiliumEgressGatewayPolicy's `interface: tun0` ONCE, into a cached +# (egress IP, egress ifindex) pair, and — measured on k3s-dev-3 — does NOT re-resolve it +# when tun0 changes underneath, not even with `egress-gateway-reconciliation-trigger- +# interval: 1s`. Upstream says as much: "it does not automatically respond to a change in +# the gateway node's network configuration... You can force a fresh selection by +# re-applying the Egress Gateway policy." +# +# Every tunnel restart re-creates tun0 with a NEW ifindex, so WITHOUT this the gateway is +# silently dead after its first reconnect or reboot: vpn-class egress is blackholed (no +# leak — the eBPF redirect to the dead ifindex just fails — but no service, and no alert). +# Re-steering is done out-of-band by vpn-egress-resteer.service so this hook can never +# block or fail the tunnel: OpenVPN's up-script must stay fast and must not depend on the +# kube-API being reachable yet (at boot it usually is not — the resteer unit retries, and +# its timer converges anyway). +systemctl start --no-block vpn-egress-resteer.service || \ + echo "vpn-egress-up: WARNING could not trigger vpn-egress-resteer.service (timer will converge)" >&2 + echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active" From 60e3f09c677127a7bb1c520d21e258e50d6f355d Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:38:46 -0500 Subject: [PATCH 2/6] =?UTF-8?q?feat(vpn):=20prod=20egress-gateway=20group?= =?UTF-8?q?=20=E2=80=94=20k3s-prod-3,=20singleton,=20opt-in=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 --- .../vpn_egress_gateway_prod/README.md | 49 +++++++ .../vpn_egress_gateway_prod/vault.yml | 131 ++++++++++++++++++ infra/ansible/inventory/hosts.ini | 37 ++++- .../ansible/playbooks/vpn.egress.gateway.yml | 26 +++- 4 files changed, 234 insertions(+), 9 deletions(-) create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md new file mode 100644 index 0000000..f111110 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md @@ -0,0 +1,49 @@ +# group_vars for the `vpn_egress_gateway_prod` node — ADR-0015 host tunnel secrets (PROD) + +Prod's egress gateway is `k3s-prod-3`, in its **own** inventory group +(`[vpn_egress_gateway_prod]`) so that the flagless +`ansible-playbook playbooks/vpn.egress.gateway.yml` — which defaults to the **dev** group — +can never touch a prod node. Prod is reached only by asking for it: + +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# then drop --list-hosts +``` + +A separate group needs its own `group_vars`, because the dev group's vault is attached to the +dev group and is not visible to a prod host. The required keys are **identical** to dev's — +see `../vpn_egress_gateway/README.md` for what each one is: + +| Vault key | +|---| +| `vault_vpn_egress_gateway_pia_username` | +| `vault_vpn_egress_gateway_pia_password` | +| `vault_vpn_egress_gateway_pia_ovpn` | + +## How this file was created (2026-09-12) + +`vault.yml` here is a **verbatim copy of the dev group's encrypted blob**. That is deliberate: + +- the operator confirmed the **same PIA account** applies to prod and is well inside its + connection limit, so the values are the same values; +- copying the *ciphertext* means the plaintext was never decrypted to disk, never passed + through a shell, and never appeared in an agent transcript. The repo-wide vault passphrase + (`ansible.cfg` → `vault_password_file`) decrypts both files identically. + +The two files are now **independent**: rotating the PIA credentials means re-encrypting +*both*, and that is the intended behaviour — prod and dev are separate environments and a +credential change should be a deliberate act per environment, not an accident of sharing a +file. If prod is ever moved to its own PIA account (recommended if the connection limit ever +bites, or to keep the public egress IPs of the two environments independent), re-create this +file alone: + +```bash +cd infra/ansible +ansible-vault create inventory/group_vars/vpn_egress_gateway_prod/vault.yml +``` + +## Notes +- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`). +- **Never** run `ansible-inventory --list`: it prints DECRYPTED vault values. diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml new file mode 100644 index 0000000..1948dd7 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml @@ -0,0 +1,131 @@ +$ANSIBLE_VAULT;1.1;AES256 +38306636313935333030303664653739316233663139616563663233643262373637306331653264 +6538363539383464353964663632633561323562383038340a396233633232356137313435336231 +36333663363130643532613566643339373365626462313537653033306363336564343139373537 +3164323463616632630a373732333536613936366463313439343664363530383664636535616261 +31306332666666643637346364386537343062383033663530346163373536323639373366333661 +36303831323662333232633034393362333561643637623033656565303461336465333664366332 +36336466656437343431363735646661643837666335633464666130643866313830366465323166 +65616537343630303431623537303766383762646533616539613235343938656237613934623565 +35373835633539653333343232626336326638303961373230663561313330313632393066323130 +31383838386262393730666231376362383132646230383964313439663935613861323137646331 +36626365323566316436386662383434366664356166636431663332393665323734623735303239 +62383031613237333136613837356166343436306137353935613661613264326466346337346530 +65356630373235313131656131383965313366613965386662343961653864373964346337363562 +66626635363438656430306231323764346564666132633966353861643262353131623134656137 +39306134626163346336613732326264396438623630656534396565343763303761333538316661 +30623466366232323531633262623434643366613833353564386163303966316537623934336163 +66323933653036366535663266666531666236646431383066616262353733316165666434616534 +35386633313464363233326663363634626239616364383866653135386435636130316335666430 +38323363386462373862366230393764386338643466363161333733653065383966663935303964 +64333461333437653433373131343736333831313236306262373631353461356237386163623166 +30663761306534316439356365396663336135333537663366353866336536663934646133333332 +65353236366435326630653830346631643236326337626435333236386634333137623138316131 +61323365303461353932626336323534633533646233653135396665373765666337333934356662 +36646339376564366439646137306364623165626263383634326337333566663236346366366334 +37316166393763663139336538663266333736623661383062383732303133326233313232323836 +34356130396131346533633336373736663366366231343162633064343565613937366434613664 +31656365303439643837323135613936646133306163663265373933633231616135353335376435 +61633965356538613636663564346566313261306235313438316639626131303566363365663764 +64316562383361373663653034393333653831666239363037343636666364666536343534613066 +61373335353464633963623161353866333038643062303931653134626334393864663461653531 +65373766336437376435616138666563623337373038623566666136626637633463323965666139 +38336139613635383066336164373061663266326235363731663565613336623936373531383966 +62323330636263306665343462666639643530336633346531636263616436356631343763343432 +61636264316566663933346163373163653734623031393962343635653765653964633137346433 +62616138333536336438363564356636316531346566663066376234356230626165386165336165 +38656166633839656438323966376561306338623237303530376231393663663661343362653664 +37613433653634363534306432383061373165613266386562323632636537373736656137633639 +33666634316465383066333737616636363866626330313232326434666164316439333433633461 +35336161313039646264613835613336326332326464313339636532653837653430393939633462 +66656563626163376134623437633332633535336233656237303962623063363134616431623235 +62323265666136323366343565656236646430393237356333613836343438353164313535616138 +34613134663465333734663337303663316161313439626332326334363835376361366463363631 +30396336303761383465373439313666656237356433303532373162306166356165316164656334 +37326163393230306161653332323462333238313166373134343839323461363936353737613662 +61643938333464366433306363633531383264613030316631303034376131323135313136316461 +31666632393631376133373533616639396361633762636663323463363433626537656333653837 +38633632623139356630616637663331616461383361383733613563383165363139343035383333 +32653563303038336137336333653561363037633634323662616666356534323138353438616463 +64383639353733333537616632346230383330356166663433373734386435376563333038333934 +37616536333132303166366235613265336432616239393730353036616337386463363330306532 +65613066363434346562633630336135313736393137656230306163383730356133326561333431 +35393466323936636562616336353934626262336363306335636331666633623432343134396334 +37343765646134306638313636326531616665323136643264353666323236343239633639356537 +61343037666232386135303363363938353864396461363830373633363730323165303165633430 +32656532633761316132613463616365353066393165353331646665613239333738653861363134 +30343939333633366532656430303133643337356336336638653266653766353637356561663636 +61626632333438313436323465643434666334393034366166343930323661386664623663396435 +61633334656464623330623333346165363563356363656663373335646433343762663964323735 +34316239663164623431353164306235386661666632383730376139626666313935663462326433 +36313039326635396435646466306366343438656334346231393066326638313337373933393231 +35373731336164396165303935316366343565363631353137393333613363656564343063373933 +61643033373839656532373330666436396632626663353436356231323166326365323832346461 +31366630626635323465333238613734313265356231313338356431393338353166633939383334 +64333634373638383937663933336563663132366534643434373032333934383533383063643036 +39323534326139633931623831656565333930323362366539346239646464353536346639343436 +37663337396565363065343532616437333666316266633632653661396535636461653335363538 +37303263383030646233313239643536326434323933616135373835393965346462373739383230 +33383863353761346232643835633139363231663162353937343363653339393838616535393436 +38303737633231626236316336353832306232633032653032633365386563626434646139633063 +31313436343063643766663465626330643366313663633733663762653631633966383736336138 +36353738383865613537366364666665326464626365643032666630643039326538663531353964 +33646561303435666237303763663634613136633662633536303066363764383039373236666135 +30626639313531353134353730323334623631396463323035313733333965623665333466366164 +62646539393966626337333637646438313635373666306538356631396237323739613032353832 +31383439343531373561353038363535353266333265336434303934343239633334383865333436 +31616537666332373332623932623162333138373563363336303735396431396464613364643162 +61323737643134653763373232383134643931343366353437633632326138353237336333616238 +32613239316666343761363863376563613638366432313461643535353731383631633233366234 +39663632613062656337616561636533633264373930376437373132303830313863303734346632 +33313431336661663634663861643364363834626638666565383635343433323961623765633265 +34386631653833653865383865636364653265363162323462376364313964626633313635313765 +38633131653836396463393136333932346532333631626461373739346139306465653532366632 +65363333633964616632613830386566623865356131646338653236373861346430363164393634 +64373434663531643436616664393936396564666561313862636133306537373137636234626232 +61313861626235636235353431353738303161373939373966383139323833346662373731646434 +31356133623331323661666234613466366130336662653764363738626531346531333339396266 +32636564653732396339383533343266633730666434303161626362366134353361643362626533 +32333132656636333736616661323664316234393937313666653762316463306437633163383030 +65383962346233663061356338306164343363323736646165613730373166333533373635373039 +39343262393031626637623030373535663539613361613366333863336531663637323035376564 +30626232623661623065643136353237316333323764663633643132653264303162366134616564 +31366665313830323263373835613964386132333538396165663538653865356236646234373465 +30383533316532363133323465363063373962353763346366383563613862336465366565653637 +62346365373030383135323933366138656463323864663363626330653036623739343035336135 +62643262336436336462386430363430363139396566343963373664373334366430643830313439 +66613362396137313661666635643962346662376137393066653765636330366462633565346663 +38633864366666316534653665303664393863633464373638393463343537386437383039373530 +66623063653536666364303531373931613433383139376664633934376662336231643762353830 +31653065303430393864383532333766613965383831616563653233313338343861663334373465 +39363361313532333034613034666265386430333466396163303263636133316263313538656634 +65313464316463623134356562366261663635323530393137313233316638643364646639343935 +38313131633132636463323231393332363766653935323965373134653336376131636163376566 +33623934346431313433386430656264316566383436663535343532303763316437303436336465 +39626231303931303130643237383663666338313662663231313935643334393362323130356530 +39656330616138373962633732313436666662616165316236323930613134643161613464636536 +36613735636233616164396630333634333935393938623061313963636461316564313662353034 +33666434326165643964303561303062323061633766353761313464376262333361323532616361 +39313832396134643966353239333066346239626534303830396165396530623135333834313439 +37373661643731623935666437313465633565303232366638393837303462613035393031333061 +39666433373662633730363665313666346165366138306636623263633537346631666132306238 +34393363373335663631393135646231313064383837396164613536643533383464313536663561 +38633131303966356534636539333733336464393236623238373766616537613437343232303437 +64323636613862663135323034636239333433333165313239336239666665616631623934316637 +61303237336334316265643964663830336539353164343063636664633430326139356534313036 +65393464653330333365336335363531303536313439363066376438666133376664346336633937 +66346165666337666134333963613662303534656339373731663534656533316338353763643930 +38313239313637326435323463333863663433623366316532333065396635663966343835636630 +34643631613431333431636330623564663662383063353138656436336333333262306633666163 +38303032316133666533393663343366333162626436333762306263616362643936383637653330 +31643334613766636231333461643863333838343961383762316162636634323134313461313962 +35363062663934336632383931333262643534336563376537343265633434336439623131613337 +32626461643263623664353639613361393735663266373930366134633534343466346461313032 +34363162343464353037616637316636393139343563303331376362653238346631333137393164 +30306636366636366562656361346637356364666136623131316138393262633963356138383232 +64383932623831376632336331343333393739613663613763303035643136653965386435366663 +35313161383236356563653931313933633737646363363339336639613961343934373637643034 +63333036343532323166366132656538643331303131393737656338323939396461393764393134 +33663539633930636265393363326638636130313036333536306362663633616563316465616330 +35353539303335336565373962393433333539613439353765643061383132386133396339616137 +3265326634643062346666356263326639373130333362333064 diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 1d0f330..4b419ce 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -45,6 +45,34 @@ localhost ansible_connection=local [vpn_egress_gateway] k3s-dev-3 +# PROD's egress-gateway node — a SEPARATE group on purpose (2026-09-12). +# +# It is NOT a member of [vpn_egress_gateway] above, and k3s-prod-3 must never be added +# there: that group is the DEFAULT target of playbooks/vpn.egress.gateway.yml, so a +# flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` would then silently +# reconfigure a prod node's kill-switch datapath. Prod is reached only by asking for it: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 +# +# (`--list-hosts` first, every time. The play's target var is +# `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`.) +# +# WHY NODE 3, AND WHY EXACTLY ONE: +# * node 1's SSD was replaced 2026-09-11 and it has the least operational history; +# * node 2 carries the §0b memattr history; +# * node 3 has a single uplink (eno1 192.168.2.12 — its wireless is gone: no wl* netdev +# exists, the /etc/network/interfaces wlp1s0 stanza is commented out, and the +# 192.168.2.195 address the earlier note below claimed is NOT present. Verified +# 2026-09-12) which is what a kill-switch datapath wants. +# * LEXICAL ORDERING HAZARD: Cilium OSS picks "the first node in lexical ordering" when +# several match the policy's nodeSelector, so labelling k3s-prod-1 would SILENTLY take +# the gateway role from k3s-prod-3. Exactly one host here, forever. +# Its PIA credentials are its own vaulted group_vars/vpn_egress_gateway_prod/vault.yml — +# the dev group's vault does not apply to a prod host. +[vpn_egress_gateway_prod] +k3s-prod-3 + # The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml, # which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY # over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so @@ -116,8 +144,13 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 # needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA # SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now # three-way, not a node-1 outlier. -# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately -# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2. +# NOTE (CORRECTED 2026-09-12): this used to read "node 3 also has wlp1s0 up on +# 192.168.2.195 (wifi)". It does not — re-verified before making node 3 the VPN egress +# gateway, because a second live uplink in a kill-switch datapath is not a detail: there is +# NO wl* netdev on the node at all, no phy80211 device, the wlp1s0 stanza in +# /etc/network/interfaces is commented out, 192.168.2.195 is assigned nowhere, and the only +# IPv4 addresses are lo and eno1 192.168.2.12. `ansible_host` is that WIRED eno1 address — +# the same directly-routable 192.168.2.x LAN as nodes 1 and 2. k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] diff --git a/infra/ansible/playbooks/vpn.egress.gateway.yml b/infra/ansible/playbooks/vpn.egress.gateway.yml index 92d4a79..751e0c6 100644 --- a/infra/ansible/playbooks/vpn.egress.gateway.yml +++ b/infra/ansible/playbooks/vpn.egress.gateway.yml @@ -11,13 +11,25 @@ # vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn # Run: ansible-playbook playbooks/vpn.egress.gateway.yml # -# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod -# has NOT decided whether it runs an egress gateway at all, nor on which node -# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress -# group in the inventory yet. When there is, point this at it with -# -e vpn_egress_gateway_hosts= -# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to -# the dev group. +# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. PROD has +# its own group as of 2026-09-12 — `vpn_egress_gateway_prod` (k3s-prod-3), with its own +# vaulted credentials in group_vars/vpn_egress_gateway_prod/: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# +# The var is `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`. Prod is +# deliberately NOT a member of the default group, so a flagless run cannot reach it — and +# `--list-hosts` before every run is the habit that catches a mistyped flag, which in this +# repo has silently targeted all three dev nodes before. +# +# THE LABEL IS WHAT ARMS THE DATAPATH. The CiliumEgressGatewayPolicy is already reconciled +# on both clusters, so the instant a node carries +# `homelab.keithjackson.dev/egress-gateway=true` steering goes live. The role sequences +# kill-switch -> tunnel -> re-steer -> label, with the label LAST, on purpose. Never apply +# that label by hand, and never run this play with tags that skip the tunnel or kill-switch +# tasks. Exactly ONE node per cluster may carry it (Cilium OSS picks the lexically first +# match, so a labelled k3s-prod-1 would silently steal the role from k3s-prod-3). - name: Configure the node-level VPN egress gateway hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}" become: true From 04606d48b2bdab7996ecc0e605f75a0ec1704c34 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:57:52 -0500 Subject: [PATCH 3/6] docs(vpn): stop crediting the nftables kill-switch for a drop it never makes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 197 ++++++++++++++++++ .../vpn-killswitch/chainsaw-test.yaml | 27 ++- .../roles/vpn_egress_gateway/README.md | 152 ++++++++++---- platform/vpn-egress/README.md | 106 +++++++++- platform/vpn-egress/egress-steering.yaml | 50 ++++- 5 files changed, 479 insertions(+), 53 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 30f11df..c2df2c5 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2089,3 +2089,200 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP** (`placeholder-guard-coupling`, pre-existing), 0 FAIL. + +## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`) + +The operator asked for VPN egress on prod. An earlier assessment this session had recommended +holding off over three suspected defects; the operator decided, so the defects were fixed first +and then the gateway was stood up. All three were settled by **measurement on the dev gateway +node**, not by reading code — and two of the three turned out to be different from what was +suspected, in one case much worse. + +### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes + +`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on +that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex +1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and +the ledger still recorded the gateway as conformance-green. + +**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an +(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit: +*"it does not automatically respond to a change in the gateway node's network configuration… You +can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart +re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.** +`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that +setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that +k8s object events already triggered, and netlink link events are not among them. + +Measured, in order: + +| step | observation | +|---|---| +| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out | +| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately | +| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again | +| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later | +| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers | + +**Fix:** `roles/vpn_egress_gateway` now publishes +`homelab.keithjackson.dev/egress-uplink=-` on the gateway node — from the +OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or +fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence +timer for the boot case. A **node label, not an annotation on the policy**: the policy is +Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are +already this role's territory — and the label's value doubles as the diagnostic. After the fix, +stop->start recovers unaided (map 1989 -> 1990, PIA IP returns). + +**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the +tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified +twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and +remains sound. + +### Defect as reported vs. as measured + +**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps +SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match +neither the ip-rule nor the drop rule and **leave via the WAN**. + +**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing +lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in +place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the +old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on +the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out +tun0 — the redirect to a dead ifindex simply fails and the packet is dropped. + +The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the +down-script already retained it). It cannot leak pod traffic — see the next section for why — but +node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch +reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which +clears rules but leaves a named set's elements intact. + +### The nftables kill-switch cannot see the traffic it was written to guard + +Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table +on the gateway node, while a vpn-class pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any — control, the hook is live | 45166 | +| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 | + +With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF, +which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The +hook works and the matching works — the steered pod packets are simply never presented to it. + +**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a +tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes +because of that, not because of the nft rules. Every comment in the tree that credited the nft +table has been corrected rather than left to mislead the next reader. + +**The table is kept, not deleted**, because it does cover something real — node-originated traffic +sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`: + +- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms, + `killswitch_accept` advances; +- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop` + 0 -> 3 packets / 252 bytes**, 100% loss. + +Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever +counts in normal operation, the datapath has changed. + +Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps +that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule +to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary +mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the +role README's standing escalation ("add the shared fwmark to the interface contract so the +CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there +is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in. + +### Prod stand-up + +`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its +own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of +`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing +the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run +as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` +first (confirmed: exactly `k3s-prod-3`). + +Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries +the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical +ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway +role, and the public egress IP, away from `k3s-prod-3`. + +The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and +copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The +two files are independent from here on; rotating means re-encrypting both, which is correct for two +environments. + +**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said +node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink +in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211` +device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is +assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory +comment is corrected. + +**Result on prod:** + +| | before | after | +|---|---|---| +| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) | +| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged | +| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok | + +Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from +the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going +out. + +### The NFS media mounts are unaffected, verified rather than assumed + +sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and +`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was +checked before and after arming: + +- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node + kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches + it. No `excludedCIDRs` entry is needed for that. +- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing + `192.168.0.0/16` exclusion anyway. +- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway + node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own + `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS + media mount in this repo — only plex, sonarr, radarr and nextcloud do. + +### Left for the operator / not settled here + +1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can + only run where a node carries the gateway label, and its recovery step is only passable with + this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE + cluster, so it was not fired at prod unasked. +2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside + the connection limit, but the two environments are indistinguishable from outside. A separate + prod PIA account is the clean fix if that ever matters. +3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the + ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the + single-gateway choice is now recorded in the inventory itself, not just in prose. +4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but + nothing watches the invariant "the gateway node's `egress-uplink` label matches + `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one, + and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small + node-exporter textfile from the re-steer script would close it. +5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a + commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert + (wireless is gone) and not printed here. Worth scrubbing on all three nodes. +6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class + fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect: + > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress + > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be + > used; this was verified with packet capture on both the WAN and tunnel devices. The host + > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel + > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod + > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark + > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.* + > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a + > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every + > interface change. This is a standing operational requirement of the ADR-0015 design, not an + > implementation detail.* diff --git a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml index e89bc3b..7b317af 100644 --- a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml +++ b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml @@ -10,8 +10,31 @@ # egress-gateway node is STOPPED (fault injection) — # THEN the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS # and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS). -# The host nftables kill-switch blackholes egress-class=vpn traffic while tun0 -# is absent (ADR-0015 invariant 3/4: vpn fails CLOSED). +# vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4: +# vpn fails CLOSED). +# +# MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables +# kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway +# datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device +# without invoking netfilter POSTROUTING, measured with counters on the gateway +# node (0 packets for the pod CIDR and the PIA range in forward and postrouting, +# against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's +# own drop when it cannot use the cached egress interface; tcpdump on both the WAN +# and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and +# still the right one — only the explanation was wrong. +# +# PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node +# carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to +# it by nodeSelector, and with no gateway there is no tunnel to stop). +# +# RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives +# tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own, +# so before 2026-09-12 the recovery step could not pass: measured, the map stayed +# on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway +# now publishes the live uplink as a node label on every connect, which makes +# Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP +# returns). If this test's recovery step ever fails again, check +# `journalctl -u vpn-egress-resteer` on the gateway node first. # AND the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN). # WHEN the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP). # diff --git a/infra/ansible/roles/vpn_egress_gateway/README.md b/infra/ansible/roles/vpn_egress_gateway/README.md index 59992c8..eb34984 100644 --- a/infra/ansible/roles/vpn_egress_gateway/README.md +++ b/infra/ansible/roles/vpn_egress_gateway/README.md @@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below. > once the tunnel is live in dev/test. ## Interface contract (shared with the Cilium-policy agent) -- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`). +- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod` + (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the + flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the + first node in lexical ordering", so a second labelled node does not add failover — it silently + steals the role. - **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the - `CiliumEgressGatewayPolicy` selects the gateway node by it. + `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the + datapath**, which is why the role applies it LAST. - **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it. -- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side - MAY stamp on vpn-class packets for a tunnel-down-independent host backstop. +- **k8s node label** `homelab.keithjackson.dev/egress-uplink=-` — set by this + role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what + makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it + on its own, so without this the gateway is dead after its first reconnect. See "Re-steering". ## How the four ADR-0015 invariants are met @@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff 2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the narrow matches, never break anything). In `postrouting`: - - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out. - - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet - trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even - while the tunnel is down** because the source set is retained on tunnel-down (see below). - - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop. + - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let + tunnel-sourced traffic out. + - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a + tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed + even while the tunnel is down, because the source set is retained on tunnel-down (below). -`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then -re-adds** the current tunnel source on every connect; the down-script **intentionally retains -it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are -never in the set, never carry the mark, and are never targeted — so they always hit + **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a + source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and + moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet + would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252 + bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next + paragraph for what it can and cannot reach. + + **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic + table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward` + and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on + that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0. + Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs + straight to the egress device, so netfilter POSTROUTING is never invoked for them. What + makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is + unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving + either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic + sourced from a tunnel address. + +`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds** +the current tunnel source on every connect and never flushes (it used to flush first, which left +a brief moment where a packet still sourced from the previous address matched neither the +ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and +`internet`-class traffic are never in the set and are never targeted — so they always hit `policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental. +A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush +table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot +empty the set mid-flight. + ### Explicit, edge-independent fail-closed backstop (tunnel-down) Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was @@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in the host nft table — defense-in-depth, independent of edge-router behavior. -This is safe precisely because the **up-script flushes the set and re-adds the (possibly new) -tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained -stale address self-corrects the instant the tunnel comes back. The set remains scoped to -vpn-class only; node and internet-class traffic are never members, so **no invariant is -weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in -table `51820` remains as the second, independent fail-closed layer. +The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live +source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect +and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address +matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just +been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra +element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and +internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4 +upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed +layer. ## Why the node default route is provably unaffected (design-level verification) - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero** @@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer. ip-rules, never `table main` / `table default`. - The ip-rule matches **`from `** — an address the node itself never sources traffic from — so node-originated and internet-class packets never enter the tunnel table. -- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark. +- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`. - Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down. + Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up, + node curl = the WAN IP, k3s `/readyz` = ok. -## Kill-switch layering & ESCALATION (honest limit) -- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the - tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps. -- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when - its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no - valid egress path rather than falling back to the node's default route) **plus** the routing - blackhole. That is the ADR's expected behavior. -- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when - the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class - packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the - *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium - Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for - R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy - marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop. - This is flagged rather than worked around — no spec was weakened to reach done. +## Re-steering — Cilium does not follow tun0, and that is the whole ball game + +Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex) +pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it +does not automatically respond to a change in the gateway node's network configuration... You can +force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`: + +| what | observed | +|---|---| +| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 | +| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) | +| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them | +| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) | +| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately | + +So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix +is `templates/vpn-egress-resteer.sh.j2`, which publishes `-` as the +`egress-uplink` node label: + +- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and + writing to it from a node invites drift/ownership churn; node labels are already this role's + territory, and the label value doubles as the diagnostic (`kubectl get nodes -L + homelab.keithjackson.dev/egress-uplink`); +- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can + never block or fail the tunnel and does not need the kube-API to be up at that instant; +- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where + the API is not up yet and every in-script retry is exhausted; +- **idempotent** — if the label already records the live uplink it exits without writing, so the + timer is free to run often. + +Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map +1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely — +which also means the **recovery half of the `vpn-killswitch` conformance test could not have +passed** on the pre-fix implementation. ## What requires a LIVE tunnel to validate (cannot be tested without operator creds) 1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface. 2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP. 3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak) while internet-class is unaffected — the R-NET-06 fault-injection gate. -4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped - rules actually see the packets (eBPF datapath interaction — see escalation). +4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped + rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does + NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE, + HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended. 5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live). 6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role. ## Files - `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars. -- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label. +- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label + (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering). - `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart. -- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units. -- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`. +- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and + tunnel units, and `vpn-egress-resteer.{sh,service,timer}`. +- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: + `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and + `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod). + +## Running it against prod +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# confirm it says exactly `k3s-prod-3`, then drop --list-hosts +``` +The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply +the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or +kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is +not ready to carry it. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index cbd3e58..907e29a 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015 by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster gateway pod** — the tunnel is host infrastructure, not a Flux workload. -> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class` -> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster. +> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`. +> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev. +> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the +> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale +> egress ifindex — see below), so the test's recovery step could not have passed on the +> implementation as it stood. Whatever produced the earlier green result, re-run it. ## How it works (ADR-0015) Three layers, two owners: @@ -24,9 +28,101 @@ Three layers, two owners: (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel. -- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered - `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so - `internet`-class and node traffic are unaffected. +- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails + closed because Cilium drops the traffic when it cannot use its cached egress interface. The host + nftables table is retained for a narrower job. See "The kill-switch does not do what this file + used to claim" below — that section is the measurement, not an opinion. + +## LIVE ON PROD since 2026-09-12 — `k3s-prod-3` + +| | dev | prod | +|---|---|---| +| gateway node | `k3s-dev-3` | `k3s-prod-3` | +| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | +| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | +| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | + +Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three +nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home +WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was +reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed +`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the +gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also +means nothing had been leaking in the meantime. + +Both environments currently share one PIA account and, as it happens, one public egress IP. +That is inside the account's connection limit, but it means the two environments are not +distinguishable from the outside; a separate prod PIA account is the clean fix if that ever +matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). + +## The kill-switch does not do what this file used to claim (measured 2026-09-12) + +This README, `egress-steering.yaml`, and the role all used to say the **host nftables +kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.** +With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in +eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is +never invoked for those packets. + +A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class +pod egressed: + +| counter | packets | +|---|---| +| `forward`, saddr = pod CIDR | **0** | +| `forward`, saddr = PIA tunnel range | **0** | +| `postrouting`, oif tun0, saddr = tunnel range | **0** | +| `postrouting`, oif WAN, saddr = pod CIDR | **0** | +| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 | +| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 | + +**What actually fails closed is Cilium itself.** When it cannot use its cached egress +interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a +tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the +guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance +test passes because of it, not because of the nft rules. + +The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely +does cover: **node-originated** traffic sourced from a tunnel address. That was proven on +`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 +replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out +`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back. +If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read +that as news, not reassurance. + +Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that +mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table +2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is +a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark` +variable is gone with it, and the "add the fwmark to the interface contract" escalation in the +role README is **closed as not-implementable** rather than left open. + +## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help + +Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not +revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes +stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been +dead for ~80 minutes when this work started, with no alert and nothing in the ledger. + +`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that +setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces +reconciliations that k8s object events already triggered — netlink link events are not among them. + +The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=-` on +the gateway node on every connect, plus a 2-minute convergence timer. That node update makes +Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 +within 10s, egress recovered immediately). A node label rather than an annotation on this +Flux-managed policy, so GitOps state is never written to from a node. + +**Debugging a dead gateway starts here:** + +```bash +kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink +kubectl -n kube-system exec -c cilium-agent -- \ + cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0` +``` + +If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the +gateway node is the next stop. ## Resilience — there is NO gateway failover (spec correction, escalated) diff --git a/platform/vpn-egress/egress-steering.yaml b/platform/vpn-egress/egress-steering.yaml index 1cabfa8..bd06f2b 100644 --- a/platform/vpn-egress/egress-steering.yaml +++ b/platform/vpn-egress/egress-steering.yaml @@ -59,16 +59,38 @@ spec: # Excluded flows fall through to Cilium's normal local datapath (invariant 1). destinationCIDRs: - "0.0.0.0/0" + # + # BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media + # mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both + # inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason + # is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted + # by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never + # matches it and no pod-scoped egress steering can touch it. Verified after arming the prod + # gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list + # AND write the export, and the node's own mounts are intact. No RFC1918 destination needs + # adding here for the media path. excludedCIDRs: - "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net - "172.16.0.0/12" # RFC1918 - - "192.168.0.0/16" # RFC1918 (home LAN / node network family) + - "192.168.0.0/16" # RFC1918 (home LAN / node network family / the bulk NFS server) # --- Where it egresses -------------------------------------------------- - # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When - # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible, - # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to - # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED, - # internet fails OPEN). + # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. + # + # CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the + # authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot + # be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the + # skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never + # see this traffic. Measured with a counters-only nft table on the gateway node while a + # vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and + # the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live) + # and 16 for a node-originated curl out tun0 (so the matching works). + # WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used + # it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a + # tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak); + # internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold). + # The host nft table is retained, with counters, for the narrower case it CAN see: + # node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's + # counter moved 0 -> 3 packets when such a packet was routed at the WAN). # # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby. # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD @@ -89,6 +111,22 @@ spec: # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)" # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06. + # + # `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes. + # Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the + # first resolution and does NOT re-resolve on a netlink change. `systemctl restart + # vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class + # egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but + # dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events. + # Upstream: "it does not automatically respond to a change in the gateway node's network + # configuration... You can force a fresh selection by re-applying the Egress Gateway policy." + # The Ansible role closes this: roles/vpn_egress_gateway publishes + # `homelab.keithjackson.dev/egress-uplink=-` on the gateway node on every + # connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this + # policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s. + # DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node + # writing to GitOps state is a drift fight. Check the label, not this file, when debugging: + # kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink egressGateway: nodeSelector: matchLabels: From 37a6c94168c276e44595a97b613ce17b914aba83 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 18:05:49 -0500 Subject: [PATCH 4/6] fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Standing up prod broke dev, and the mechanism is worth spelling out because "same PIA account, well inside its connection limit" is true and still not enough. The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod landed on the SAME server AND were handed the SAME tunnel address — both journals read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198` and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on that server; the loser then receives nothing, trips `--ping-restart` after 60s, reconnects and steals it back. Measured cycle: dev restarting at 22:57:05, 22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes apart, indefinitely. Neither environment had reliable vpn-class egress — prod's probes looked green only because they happened to land inside an up-window. Fix: `vpn_egress_gateway_pia_remote` (" ", empty = honour the .ovpn) rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239} — and every server in the pool takes the same credentials and the same inline CA, so only the address differs. Dev keeps the .ovpn's .112 untouched, which is why the override is per-group rather than in the role or a second vaulted .ovpn. Two details that matter for whoever touches this next: * The regex is anchored to a line STARTING with `remote` plus whitespace, so `remote-cert-tls` and `remote-random` are not collateral damage. The task keeps `no_log: true` — the transformation never widens what is logged. * A PIA server address is NOT a credential, so it lives in group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault rather than inside it. Keeping it out of the vault is what makes it reviewable. If prod ever gets its own PIA account, this override can be deleted again. Co-Authored-By: Claude Opus 5 --- .../vpn_egress_gateway_prod/vars.yml | 22 +++++++++++++++++++ .../vpn_egress_gateway/defaults/main.yml | 17 ++++++++++++++ .../roles/vpn_egress_gateway/tasks/main.yml | 11 +++++++++- 3 files changed, 49 insertions(+), 1 deletion(-) create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml new file mode 100644 index 0000000..19e2ed4 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vars.yml @@ -0,0 +1,22 @@ +--- +# Non-secret group_vars for the PROD egress gateway (k3s-prod-3). Credentials live in the +# vaulted vault.yml beside this file. + +# PROD DIALS ITS OWN PIA SERVER — and it has to. +# +# The vaulted .ovpn pins ONE server, 158.173.241.112. Measured 2026-09-12: with dev +# (k3s-dev-3) and prod (k3s-prod-3) both dialling it, PIA handed BOTH sessions the SAME +# tunnel address — `Server-10779-2a ... ifconfig 10.72.192.7` in both journals. Each +# reconnect steals that address from the other; the loser then sees no data for 60s, hits +# `--ping-restart`, reconnects and steals it back. Both tunnels flapped on a ~2-minute cycle +# (dev 22:57 / 22:59 / 23:01 UTC, prod interleaved a minute off) and neither environment had +# reliable vpn-class egress. This is NOT the account's connection limit — that has room. +# +# 158.173.241.135 is another member of the same PIA Stockholm LB pool: `sweden.privacy.network` +# resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239}. Any server in the pool accepts +# the same credentials and the same inline CA, so only the address differs. Dev keeps the +# .ovpn's pinned .112 untouched, which is why the override lives HERE and not in the role. +# +# A public server address is not a credential, so this is plain group_vars, not vault. +# If prod is ever moved to its own PIA account, this can go away again. +vpn_egress_gateway_pia_remote: "158.173.241.135 1198" diff --git a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml index 359d1ea..817ad04 100644 --- a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml @@ -20,6 +20,23 @@ vpn_egress_gateway_pia_password: "{{ vault_vpn_egress_gateway_pia_password }}" # The FULL text of the PIA .ovpn (inline /, cipher/auth, remote line). vpn_egress_gateway_pia_ovpn: "{{ vault_vpn_egress_gateway_pia_ovpn }}" +# OVERRIDE THE .ovpn's PINNED `remote` — " ", empty to honour the .ovpn. +# +# WHY THIS EXISTS (measured 2026-09-12): the operator's .ovpn pins ONE PIA server IP. Point +# two gateways at the same vaulted .ovpn and both dial the SAME server, and PIA hands BOTH +# THE SAME tunnel address — dev and prod were both `Server-10779-2a` +# (158.173.241.112:1198) with `ifconfig 10.72.192.7`. Each reconnect then steals that +# address from the other, the loser sees no data for 60s, hits `--ping-restart`, reconnects +# and steals it back: both tunnels flapped on a ~2-minute cycle and neither was reliably +# up. It is NOT a connection-LIMIT problem (the account has room) — it is one pinned server. +# +# So each environment's gateway needs its own server. The PIA Stockholm LB pool +# (`sweden.privacy.network`) resolved to 158.173.241.{34,55,112,135,239}; any PIA server in +# the pool accepts the same credentials and CA, so this is a per-environment knob rather +# than a second vaulted .ovpn. NOT a secret — a public server address — so it belongs in +# group_vars//vars.yml, not the vault. +vpn_egress_gateway_pia_remote: "" + # --- on-disk layout ------------------------------------------------------------- vpn_egress_gateway_config_dir: /etc/openvpn/client vpn_egress_gateway_scripts_dir: /etc/openvpn/scripts diff --git a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml index 401a13f..f9a7761 100644 --- a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml +++ b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml @@ -28,9 +28,18 @@ - "{{ vpn_egress_gateway_config_dir }}" - "{{ vpn_egress_gateway_scripts_dir }}" +# `vpn_egress_gateway_pia_remote`, when set, rewrites ONLY the `remote ` line so +# this environment's gateway dials its own PIA server. Two gateways sharing the .ovpn's single +# pinned server are handed the SAME tunnel address and evict each other every ~2 minutes — see +# the variable's comment in defaults/main.yml. The regex is anchored to a line that STARTS with +# `remote` followed by whitespace, so `remote-cert-tls` / `remote-random` are untouched. - name: Deploy the operator's proven PIA .ovpn (from Ansible vault) ansible.builtin.copy: - content: "{{ vpn_egress_gateway_pia_ovpn }}" + content: >- + {{ vpn_egress_gateway_pia_ovpn if (vpn_egress_gateway_pia_remote | length == 0) + else (vpn_egress_gateway_pia_ovpn + | regex_replace('(?m)^[ \t]*remote[ \t]+\S+.*$', + 'remote ' ~ vpn_egress_gateway_pia_remote)) }} dest: "{{ vpn_egress_gateway_ovpn_path }}" owner: root group: root From 38e4bf1459ca4998c1328d74bd1adbc2f4f93387 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 18:09:31 -0500 Subject: [PATCH 5/6] docs(vpn): record the PIA server collision, and the accidental proof it handed us MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stand-up broke dev, so the "one gateway per PIA server" rule and the measurement behind it are now written down where the next person will look: platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on both gateways — and the flap timestamps, because "same account, inside the connection limit" is true and is exactly what makes this non-obvious. Also corrected: the dev/prod table now shows the two environments on different servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the open item about them sharing one public IP is replaced with the real remaining decision — whether prod gets its own PIA account, which would remove the hand-maintained per-environment server value entirely. The unplanned bonus, recorded in both places: moving prod to .135 gave it a genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 -> 10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved, and all four vpn-class workloads kept egressing with no manual step. Better evidence than the synthetic address swap used to characterise it, and it cost nothing to capture. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 46 ++++++++++++++++++++++++++++++++--- platform/vpn-egress/README.md | 39 +++++++++++++++++++++++++---- 2 files changed, 77 insertions(+), 8 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index c2df2c5..8e7fe31 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2253,15 +2253,55 @@ checked before and after arming: `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS media mount in this repo — only plex, sonarr, radarr and nextcloud do. +### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways + +Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account, +well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled +it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips +`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05, +22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute +mutual-eviction cycle with no end. Prod's first probes looked green only because they landed +inside an up-window; that is a good reminder that a single green probe is not a stability check. + +**Fix:** a new role var `vpn_egress_gateway_pia_remote` (" ", empty = honour the +.ovpn) rewrites only the `remote` line of the deployed .ovpn, and +`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another +member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to +`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and +the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is +why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server +address is not a credential, so it sits in plain group_vars next to the vault, reviewable. + +**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that +re-pins, must take an unused address from the pool. + +**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to +`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) -> +`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what +defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium +re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual +step, no CEGP edit, no operator. Before this branch that address change would have left prod's +vpn-class egress dead until someone noticed. + ### Left for the operator / not settled here 1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can only run where a node carries the gateway label, and its recovery step is only passable with this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE cluster, so it was not fired at prod unasked. -2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside - the connection limit, but the two environments are indistinguishable from outside. A separate - prod PIA account is the clean fix if that ever matters. +2. **Decide whether prod gets its own PIA account.** After the server split the two environments do + have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is + solved — but they still share one account, and the per-environment PIA server is now a + hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with + dev's. A separate prod account removes that coupling permanently. **Verify dev recovered**: dev + was still inside its flap window when this session ended (see the open item below). 3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the single-gateway choice is now recorded in the inventory itself, not just in prose. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index 907e29a..b1c9aa1 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -39,7 +39,8 @@ Three layers, two owners: |---|---|---| | gateway node | `k3s-dev-3` | `k3s-prod-3` | | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | -| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | +| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) | +| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` | | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three @@ -50,10 +51,38 @@ reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress lis gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime. -Both environments currently share one PIA account and, as it happens, one public egress IP. -That is inside the account's connection limit, but it means the two environments are not -distinguishable from the outside; a separate prod PIA account is the clean fix if that ever -matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). +### One PIA account is fine. One pinned PIA *server* is not. + +Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn` +pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same +tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips +`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute +off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's +first probes looked green only because they landed inside an up-window. + +**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` +rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the +same PIA Stockholm LB pool (`sweden.privacy.network` resolves to +`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev +keeps `.112`. + +**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must +pick an unused address from the pool. A separate prod PIA account would also solve it — see +`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`. + +That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's +tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real +new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink` +label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing, +now via `158.173.241.11`. No manual step, no CEGP edit. ## The kill-switch does not do what this file used to claim (measured 2026-09-12) From 53e09a2610bb3cdf94a24738a7e4c49396ab86ab Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 18:12:46 -0500 Subject: [PATCH 6/6] docs(vpn): dev recovered the minute prod left the shared PIA server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the loop on the collision entry with the recovery evidence rather than leaving it as an open question for the operator: dev's last Inactivity timeout is 23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same session was still the live one, seven minutes unbroken against a two-minute cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to 158.173.241.36 on three consecutive probes. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 9 +++++++-- platform/vpn-egress/README.md | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 8e7fe31..b20b43f 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2282,6 +2282,12 @@ address is not a credential, so it sits in plain group_vars next to the vault, r **Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that re-pins, must take an unused address from the pool. +**Dev recovered the moment prod left the shared server.** Dev's last `Inactivity timeout` is +23:05:06 UTC — the minute prod switched to `.135`. Re-checked at 23:12:03: the 23:05:07 session is +still the live one, seven minutes unbroken against a previous cycle of two, and a source-bound ping +through the tunnel returns 4/4 at 146 ms with 0% loss. vpn-class pod egress from dev is back to +`158.173.241.36` on three consecutive probes. + **And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to `.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) -> `10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what @@ -2300,8 +2306,7 @@ vpn-class egress dead until someone noticed. have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is solved — but they still share one account, and the per-environment PIA server is now a hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with - dev's. A separate prod account removes that coupling permanently. **Verify dev recovered**: dev - was still inside its flap window when this session ended (see the open item below). + dev's. A separate prod account removes that coupling permanently. 3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the single-gateway choice is now recorded in the inventory itself, not just in prose. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index b1c9aa1..6883332 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -68,6 +68,10 @@ Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's first probes looked green only because they landed inside an up-window. +Dev recovered the minute prod left `.112`: its last `Inactivity timeout` is 23:05:06 UTC, and at +23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with +a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to `158.173.241.36`. + **Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the same PIA Stockholm LB pool (`sweden.privacy.network` resolves to