# 03 — Networking, Isolation & Egress

Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
else is convenience.

## Principles

1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
   permits it, at the port/protocol level (R-NET-01/02).
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
   it accordingly (R-NET-05/07).
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
   during tunnel failure (R-NET-06, kill-switch).
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).

## CNI: Cilium

Cilium replaces the default k3s CNI and provides:
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
- **kube-proxy replacement** (eBPF) for efficient service routing.
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
  building block for WAN/VPN path selection.
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).

## Segmentation model

- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
  namespace. Namespace = isolation boundary.
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
  policy and egress routing.
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
  namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

### Default-deny baseline (R-NET-01)
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
- **Deny all ingress** and **deny all egress** by default.
- Then explicitly allow only:
  - DNS to the in-cluster resolver (required for anything to function).
  - Declared intra-cluster dependencies (source → destination → port).
  - The declared egress class path (internet or VPN).
  - Observability scraping from the monitoring namespace (metrics/logs).

> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
> starting posture for every onboarded service.

### Explicit allow rules (R-NET-02)
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
final manifest):

```
seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)
```

Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

## Multi-WAN & egress routing (R-NET-04/05)

The platform recognizes **egress profiles**. Initial profiles:

| Profile | Meaning | Mechanism |
|---|---|---|
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |

**Where multi-WAN policy routing lives (OQ-1):**
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
  **edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
  Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
  available WANs.
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
  Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
  *physical WAN* for the internet class.
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
  is answered. The **service-facing contract does not change**: a service still just declares
  `egress: internet|vpn`.

## VPN egress model (R-NET-05/06 — ADR-0008)

VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.

Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):

- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
  tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
  **firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
  Simple, self-contained, per-service isolation; more sidecars to run.
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
  deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
  Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
  fallback to the internet path.

**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
assumed.

## Ingress (north-south) — summary

North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).

## Observability of the network (supports R-SEC-03, R-OBS-04)
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
  reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).

## Acceptance gates for this spec
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
  Hubble + a connectivity probe).
- Declared dependencies work; **undeclared** ones are blocked.
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.

## Open questions
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
  egress IPs (affects Pattern A vs B and HA of the gateway).