home-network-k3s/docs/03-networking-isolation-and-egress.md

# 03 — Networking, Isolation & Egress

Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
else is convenience.

## Principles

1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
   permits it, at the port/protocol level (R-NET-01/02).
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
   it accordingly (R-NET-05/07).
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
   during tunnel failure (R-NET-06, kill-switch).
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).

## CNI: Cilium

Cilium replaces the default k3s CNI and provides:
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
- **kube-proxy replacement** (eBPF) for efficient service routing.
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
  building block for WAN/VPN path selection.
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).

## Segmentation model

- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
  namespace. Namespace = isolation boundary.
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
  policy and egress routing.
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
  namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

### Default-deny baseline (R-NET-01)
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
- **Deny all ingress** and **deny all egress** by default.
- Then explicitly allow only:
  - DNS to the in-cluster resolver (required for anything to function).
  - Declared intra-cluster dependencies (source → destination → port).
  - The declared egress class path (internet or VPN).
  - Observability scraping from the monitoring namespace (metrics/logs).

> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
> starting posture for every onboarded service.

### Explicit allow rules (R-NET-02)
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
final manifest):

```
seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)
```

Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has **one physical commercial internet WAN** and **no plans for additional ISP
uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
as multi-ISP routing but as **two egress paths selected by routing rules**:

| Profile | Meaning | Mechanism |
|---|---|---|
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) |

- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
  route across; the `internet` path is simply the default route out.
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
  **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
  the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
  ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.

## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008)

VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**:

- **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN
  tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is
  viable)** — the cluster provisions and controls the tunnel (nothing on the edge router).
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
  **no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
  namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
  and cannot leak on its own.
- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
  pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
  services are unaffected.
- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
  for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
  ([10](10-observability-and-alerting.md)).
- **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain
  an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a
  SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end
  (tunnel + kill-switch) in dev/test.

**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
closed (no packets to the plain internet). Tested, not assumed.

## Ingress (north-south) — summary

North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).

## Observability of the network (supports R-SEC-03, R-OBS-04)
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
  reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).

## Acceptance gates for this spec
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
  Hubble + a connectivity probe).
- Declared dependencies work; **undeclared** ones are blocked.
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.

## Resolved
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
  segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
- **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced
  outside the service (ADR-0012).

## Open questions
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
  WS-NET implementation detail; the kill-switch acceptance test is the gate.
- Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism).