# role: vpn_egress_gateway (ADR-0015)

Stands up the **node-level VPN tunnel** on the egress-gateway node(s): a host **OpenVPN
systemd service** (PIA over OpenVPN-UDP), **split-tunnel** via policy routing, and a
**scoped nftables kill-switch**. This is the host half of ADR-0015; the
`CiliumEgressGatewayPolicy` that steers `egress-class=vpn` pods to this node's `tun0` is
separate GitOps authored against the shared **interface contract** below.

> **CONFORMANCE IS GATED ON A LIVE TUNNEL.** This role cannot be proven green without the
> operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass
> in CI; the `egress-class` / `vpn-killswitch` conformance tests (R-NET-05/06/07) go green only
> once the tunnel is live in dev/test.

## Interface contract (shared with the Cilium-policy agent)
- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod`
  (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the
  flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the
  first node in lexical ordering", so a second labelled node does not add failover — it silently
  steals the role.
- **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
  `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the
  datapath**, which is why the role applies it LAST.
- **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
- **k8s node label** `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` — set by this
  role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what
  makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it
  on its own, so without this the gateway is dead after its first reconnect. See "Re-steering".

## How the four ADR-0015 invariants are met

### 1 & 2 — split-tunnel; tun0 is never a default route
`templates/vpn-egress.conf.j2` includes the operator's proven `.ovpn` and layers on
`route-noexec` + `pull-filter ignore "redirect-gateway"`, so **OpenVPN installs no routes and
never touches the main routing table**. The node's own default route stays on the WAN whether
the tunnel is up or down — k3s/etcd/kubelet are unaffected.

Steering is done with **policy routing** in a dedicated table (`51820`, named `vpn_egress`):
- The OpenVPN **up-script** adds `default dev tun0` to table `51820` and an ip-rule
  **`from <tun0-addr> lookup 51820`** (priority 5000). Only packets whose source is the tunnel
  address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing
  else does, so nothing else is re-routed.
- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole), and
  **retains** the nft source set as the explicit fail-closed backstop (see kill-switch section).

### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions
Two independent layers, both scoped so they can only ever affect vpn-class traffic:

1. **Routing blackhole (fail-closed fallback).** Table `51820` always carries a permanent
   `blackhole default` (installed by `vpn-egress-killswitch.service`, metric 4000). When the
   tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets
   hit the blackhole instead of falling through to the WAN.
2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
   vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
   narrow matches, never break anything). In `postrouting`:
   - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let
     tunnel-sourced traffic out.
   - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a
     tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed
     even while the tunnel is down, because the source set is retained on tunnel-down (below).

   **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a
   source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and
   moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet
   would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252
   bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next
   paragraph for what it can and cannot reach.

   **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic
   table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward`
   and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on
   that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0.
   Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs
   straight to the egress device, so netfilter POSTROUTING is never invoked for them. What
   makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is
   unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving
   either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic
   sourced from a tunnel address.

`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds**
the current tunnel source on every connect and never flushes (it used to flush first, which left
a brief moment where a packet still sourced from the previous address matched neither the
ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and
`internet`-class traffic are never in the set and are never targeted — so they always hit
`policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
directions are structural, not incidental.

A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush
table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot
empty the set mid-flight.

### Explicit, edge-independent fail-closed backstop (tunnel-down)
Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
`ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit
guarantee that could leak if the edge router NATed that stale source. The down-script now
**does NOT flush the set**: retaining the last tun0 address keeps the drop rule armed, so any
vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
the host nft table — defense-in-depth, independent of edge-router behavior.

The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live
source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect
and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address
matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just
been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra
element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and
internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4
upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed
layer.

## Why the node default route is provably unaffected (design-level verification)
- OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
  routes and cannot install a `0.0.0.0/0` via `tun0`. (This is exactly the failure mode that made
  `hostNetwork` gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.)
- All tunnel routing lives in **table 51820 only**; the up/down scripts touch that table and
  ip-rules, never `table main` / `table default`.
- The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
  from — so node-originated and internet-class packets never enter the tunnel table.
- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`.
- Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
  `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.
  Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up,
  node curl = the WAN IP, k3s `/readyz` = ok.

## Re-steering — Cilium does not follow tun0, and that is the whole ball game

Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex)
pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it
does not automatically respond to a change in the gateway node's network configuration... You can
force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`:

| what | observed |
|---|---|
| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 |
| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) |
| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them |
| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) |
| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately |

So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix
is `templates/vpn-egress-resteer.sh.j2`, which publishes `<tun0 addr>-<ifindex>` as the
`egress-uplink` node label:

- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and
  writing to it from a node invites drift/ownership churn; node labels are already this role's
  territory, and the label value doubles as the diagnostic (`kubectl get nodes -L
  homelab.keithjackson.dev/egress-uplink`);
- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can
  never block or fail the tunnel and does not need the kube-API to be up at that instant;
- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where
  the API is not up yet and every in-script retry is exhausted;
- **idempotent** — if the label already records the live uplink it exits without writing, so the
  timer is free to run often.

Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map
1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely —
which also means the **recovery half of the `vpn-killswitch` conformance test could not have
passed** on the pre-fix implementation.

## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
   while internet-class is unaffected — the R-NET-06 fault-injection gate.
4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
   rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does
   NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE,
   HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended.
5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
   the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.

## Files
- `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label
  (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering).
- `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and
  tunnel units, and `vpn-egress-resteer.{sh,service,timer}`.
- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys:
  `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and
  `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod).

## Running it against prod
```bash
cd infra/ansible
ansible-playbook playbooks/vpn.egress.gateway.yml \
  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
# confirm it says exactly `k3s-prod-3`, then drop --list-hosts
```
The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply
the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or
kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is
not ready to carry it.