Resolve OQ-1/2/3, refine OQ-4: single ExpressVPN gateway, single WAN, external media

Operator answers folded into the specs:
- OQ-3 VPN: ExpressVPN, cluster-controlled SINGLE shared in-cluster tunnel
  (OpenVPN UDP / non-TCP), rules enforced outside the service. New ADR-0012
  supersedes ADR-0008 (Gluetun-sidecar default dropped). Central kill-switch.
- OQ-1 WAN/edge: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
  VLAN/VPN/services). Multi-WAN out of scope; segmentation + VPN in-cluster;
  edge = NAT + manual port-forward. Gateway VIP resolved to Cilium L2
  announcement. T-3.3 marked won't-do.
- OQ-2 storage: bulk media lives outside the cluster (backing TBD, non-blocking)
  via a 'bulk' external-mount indirection; Plex metadata DB stays on Longhorn.
  Split out a separate OQ-BKP for the off-cluster DR backup target (Phase 4).
- OQ-4 refined: wildcard record covers reachability but NOT ACME DNS-01 (needs
  TXT-write access) and does not match *.home.* depth. Still OPEN: Cloudflare
  API token (Option A) vs acme-dns delegation (Option B). T-2.4/T-2.5 blocked.

Updated: docs 00,01,03,04,08,09; ADR-0008/0012 + index; task board.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

docs/03-networking-isolation-and-egress.md

@@ -60,46 +60,51 @@ seerr  →  plex      tcp/32400  (Seerr talks to Plex)
 Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
 CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
 
-## Multi-WAN & egress routing (R-NET-04/05)
+## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)
 
-The platform recognizes **egress profiles**. Initial profiles:
+The environment has **one physical commercial internet WAN** and **no plans for additional ISP
+uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
+as multi-ISP routing but as **two egress paths selected by routing rules**:
 
 | Profile | Meaning | Mechanism |
 |---|---|---|
-| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
-| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
+| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
+| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) |
 
-**Where multi-WAN policy routing lives (OQ-1):**
-- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
-  **edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
-  Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
-  available WANs.
-- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
-  Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
-  *physical WAN* for the internet class.
-- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
-  is answered. The **service-facing contract does not change**: a service still just declares
-  `egress: internet|vpn`.
+- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
+  route across; the `internet` path is simply the default route out.
+- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
+  **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
+  the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
+  ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
+- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
 
-## VPN egress model (R-NET-05/06 — ADR-0008)
+## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008)
 
-VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
-via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
+VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
+shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
+the service**:
 
-Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
+- **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN
+  tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is
+  viable)** — the cluster provisions and controls the tunnel (nothing on the edge router).
+- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
+  **no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
+  namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
+  and cannot leak on its own.
+- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
+  pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
+  services are unaffected.
+- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
+  for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
+  ([10](10-observability-and-alerting.md)).
+- **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain
+  an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a
+  SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end
+  (tunnel + kill-switch) in dev/test.
 
-- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
-  tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
-  **firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
-  Simple, self-contained, per-service isolation; more sidecars to run.
-- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
-  deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
-  Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
-  fallback to the internet path.
-
-**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
-down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
-assumed.
+**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
+closed (no packets to the plain internet). Tested, not assumed.
 
 ## Ingress (north-south) — summary
 
@@ -120,7 +125,13 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
 - Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
 - Changing a service's `egress` class in git flips its path with no workload rearchitecting.
 
+## Resolved
+- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
+  segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
+- **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced
+  outside the service (ADR-0012).
+
 ## Open questions
-- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
-- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
-  egress IPs (affects Pattern A vs B and HA of the gateway).
+- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
+  WS-NET implementation detail; the kill-switch acceptance test is the gate.
+- Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism).