# 03 — Networking, Isolation & Egress Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md), [ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md). This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience. ## Principles 1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly permits it, at the port/protocol level (R-NET-01/02). 2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes it accordingly (R-NET-05/07). 3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including during tunnel failure (R-NET-06, kill-switch). 4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03). ## CNI: Cilium Cilium replaces the default k3s CNI and provides: - **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement. - **kube-proxy replacement** (eBPF) for efficient service routing. - **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the building block for WAN/VPN path selection. - **Hubble** for flow observability — the audit and troubleshooting tool for isolation. - **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)). ## Segmentation model - **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own namespace. Namespace = isolation boundary. - **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network policy and egress routing. - **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application namespaces; only the specific flows they need (e.g. metrics scraping) are opened. ### Default-deny baseline (R-NET-01) Every namespace gets a baseline policy set applied by GitOps at namespace creation: - **Deny all ingress** and **deny all egress** by default. - Then explicitly allow only: - DNS to the in-cluster resolver (required for anything to function). - Declared intra-cluster dependencies (source → destination → port). - The declared egress class path (internet or VPN). - Observability scraping from the monitoring namespace (metrics/logs). > A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended > starting posture for every onboarded service. ### Explicit allow rules (R-NET-02) Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not final manifest): ``` seerr → sonarr tcp/8989 (Seerr requests to Sonarr) seerr → radarr tcp/7878 (Seerr requests to Radarr) seerr → plex tcp/32400 (Seerr talks to Plex) ``` Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML. ## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1) The environment has **one physical commercial internet WAN** and **no plans for additional ISP uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not as multi-ISP routing but as **two egress paths selected by routing rules**: | Profile | Meaning | Mechanism | |---|---|---| | `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet | | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) | - **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy- route across; the `internet` path is simply the default route out. - **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)). - The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`. ## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008) VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single, shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside the service**: - **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is viable)** — the cluster provisions and controls the tunnel (nothing on the edge router). - **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have **no independent egress path**. The kill-switch and allow rules are enforced at the **gateway + namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn` and cannot leak on its own. - **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn` pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class services are unaffected. - **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state ([10](10-observability-and-alerting.md)). - **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end (tunnel + kill-switch) in dev/test. **Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail closed (no packets to the plain internet). Tested, not assumed. ## Ingress (north-south) — summary North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md). Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge). ## Observability of the network (supports R-SEC-03, R-OBS-04) - **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules. - Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md). ## Acceptance gates for this spec - A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via Hubble + a connectivity probe). - Declared dependencies work; **undeclared** ones are blocked. - A `vpn` service's public egress IP is the VPN's, not the home IP (verified). - Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected. - Changing a service's `egress` class in git flips its path with no workload rearchitecting. ## Resolved - **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope. - **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced outside the service (ADR-0012). ## Open questions - Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) — WS-NET implementation detail; the kill-switch acceptance test is the gate. - Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism).