# role: vpn_egress_gateway (ADR-0015)

Stands up the **node-level VPN tunnel** on the egress-gateway node(s): a host **OpenVPN
systemd service** (PIA over OpenVPN-UDP), **split-tunnel** via policy routing, and a
**scoped nftables kill-switch**. This is the host half of ADR-0015; the
`CiliumEgressGatewayPolicy` that steers `egress-class=vpn` pods to this node's `tun0` is
separate GitOps authored against the shared **interface contract** below.

> **CONFORMANCE IS GATED ON A LIVE TUNNEL.** This role cannot be proven green without the
> operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass
> in CI; the `egress-class` / `vpn-killswitch` conformance tests (R-NET-05/06/07) go green only
> once the tunnel is live in dev/test.

## Interface contract (shared with the Cilium-policy agent)
- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`).
- **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
  `CiliumEgressGatewayPolicy` selects the gateway node by it.
- **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side
  MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.

## How the four ADR-0015 invariants are met

### 1 & 2 — split-tunnel; tun0 is never a default route
`templates/vpn-egress.conf.j2` includes the operator's proven `.ovpn` and layers on
`route-noexec` + `pull-filter ignore "redirect-gateway"`, so **OpenVPN installs no routes and
never touches the main routing table**. The node's own default route stays on the WAN whether
the tunnel is up or down — k3s/etcd/kubelet are unaffected.

Steering is done with **policy routing** in a dedicated table (`51820`, named `vpn_egress`):
- The OpenVPN **up-script** adds `default dev tun0` to table `51820` and an ip-rule
  **`from <tun0-addr> lookup 51820`** (priority 5000). Only packets whose source is the tunnel
  address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing
  else does, so nothing else is re-routed.
- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole), and
  **retains** the nft source set as the explicit fail-closed backstop (see kill-switch section).

### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions
Two independent layers, both scoped so they can only ever affect vpn-class traffic:

1. **Routing blackhole (fail-closed fallback).** Table `51820` always carries a permanent
   `blackhole default` (installed by `vpn-egress-killswitch.service`, metric 4000). When the
   tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets
   hit the blackhole instead of falling through to the WAN.
2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
   vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
   narrow matches, never break anything). In `postrouting`:
   - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
   - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
     trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even
     while the tunnel is down** because the source set is retained on tunnel-down (see below).
   - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.

`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then
re-adds** the current tunnel source on every connect; the down-script **intentionally retains
it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are
never in the set, never carry the mark, and are never targeted — so they always hit
`policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
directions are structural, not incidental.

### Explicit, edge-independent fail-closed backstop (tunnel-down)
Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
`ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit
guarantee that could leak if the edge router NATed that stale source. The down-script now
**does NOT flush the set**: retaining the last tun0 address keeps the drop rule armed, so any
vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
the host nft table — defense-in-depth, independent of edge-router behavior.

This is safe precisely because the **up-script flushes the set and re-adds the (possibly new)
tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained
stale address self-corrects the instant the tunnel comes back. The set remains scoped to
vpn-class only; node and internet-class traffic are never members, so **no invariant is
weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in
table `51820` remains as the second, independent fail-closed layer.

## Why the node default route is provably unaffected (design-level verification)
- OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
  routes and cannot install a `0.0.0.0/0` via `tun0`. (This is exactly the failure mode that made
  `hostNetwork` gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.)
- All tunnel routing lives in **table 51820 only**; the up/down scripts touch that table and
  ip-rules, never `table main` / `table default`.
- The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
  from — so node-originated and internet-class packets never enter the tunnel table.
- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark.
- Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
  `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.

## Kill-switch layering & ESCALATION (honest limit)
- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the
  tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when
  its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no
  valid egress path rather than falling back to the node's default route) **plus** the routing
  blackhole. That is the ADR's expected behavior.
- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when
  the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class
  packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the
  *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium
  Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for
  R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy
  marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop.
  This is flagged rather than worked around — no spec was weakened to reach done.

## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
   while internet-class is unaffected — the R-NET-06 fault-injection gate.
4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
   rules actually see the packets (eBPF datapath interaction — see escalation).
5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
   the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.

## Files
- `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label.
- `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units.
- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`.