home-network-k3s/docs/03-networking-isolation-and-egress.md

03 — Networking, Isolation & Egress

Serves: R-NET-01..07, R-SEC-02. Related: ADR-0002 (Cilium), ADR-0008 (VPN egress).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience.

Principles

1. Deny by default, allow by exception. No service may talk to another unless a rule explicitly permits it, at the port/protocol level (R-NET-01/02).
2. Egress is a declared property. Each service declares an egress class; the platform routes it accordingly (R-NET-05/07).
3. VPN means VPN-only. A VPN-class service must never leak to the plain internet, including during tunnel failure (R-NET-06, kill-switch).
4. Isolation is auditable. An operator can prove what a service can and cannot reach (R-SEC-03).

CNI: Cilium

Cilium replaces the default k3s CNI and provides:

• NetworkPolicy / CiliumNetworkPolicy for L3/L4 (and optional L7) enforcement.
• kube-proxy replacement (eBPF) for efficient service routing.
• Egress Gateway to force selected pods' egress through a chosen node/identity/SNAT IP — the building block for WAN/VPN path selection.
• Hubble for flow observability — the audit and troubleshooting tool for isolation.
• Gateway API implementation for north-south ingress (see 09).

Segmentation model

• Namespaces as trust zones. Each service (or tightly-coupled service group) gets its own namespace. Namespace = isolation boundary.
• Egress classes as zones: internet and vpn are first-class labels that drive both network policy and egress routing.
• Platform namespace(s) (Flux, cert-manager, monitoring, storage) are isolated from application namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

Default-deny baseline (R-NET-01)

Every namespace gets a baseline policy set applied by GitOps at namespace creation:

• Deny all ingress and deny all egress by default.
• Then explicitly allow only:
  • DNS to the in-cluster resolver (required for anything to function).
  • Declared intra-cluster dependencies (source → destination → port).
  • The declared egress class path (internet or VPN).
  • Observability scraping from the monitoring namespace (metrics/logs).

A workload with no explicit allow rules can do nothing except resolve DNS. This is the intended starting posture for every onboarded service.

Explicit allow rules (R-NET-02)

Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not final manifest):

seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)

Rules are declared once in the service's catalog entry (its dependencies) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

Multi-WAN & egress routing (R-NET-04/05)

The platform recognizes egress profiles. Initial profiles:

Profile	Meaning	Mechanism
internet	Standard local network → default WAN(s)	Normal pod egress via node → edge router
vpn	Routed through a commercial VPN tunnel	Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008)

Where multi-WAN policy routing lives (OQ-1):

• Physical multi-WAN (two ISP uplinks, failover/balancing by routing rules) is primarily an edge-router/firewall responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster. Traffic leaving the cluster on the internet profile is policy-routed by the edge router across available WANs.
• In-cluster egress selection (which pods use VPN vs. plain internet) is handled by Cilium Egress Gateway + the VPN egress model. The cluster decides class; the edge router decides physical WAN for the internet class.
• Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware) is answered. The service-facing contract does not change: a service still just declares egress: internet|vpn.

VPN egress model (R-NET-05/06 — ADR-0008)

VPN-class services: sonarr, radarr, jackett, qbittorrent. Requirements: all their egress leaves via the VPN; if the tunnel drops, they lose connectivity rather than leaking.

Two supported patterns (ADR-0008 selects the default; both preserve the egress: vpn contract):

• Pattern A — Gluetun sidecar per pod. A Gluetun container in the pod holds the WireGuard/OpenVPN tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's firewall kill-switch blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically. Simple, self-contained, per-service isolation; more sidecars to run.
• Pattern B — Central VPN egress gateway + Cilium Egress Gateway. One (HA) VPN gateway deployment holds the tunnel; Cilium Egress Gateway steers vpn-labeled pods' egress through it. Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying fallback to the internet path.

Kill-switch is mandatory in both patterns and is an acceptance gate: with the tunnel forcibly down, a vpn service must fail closed (no packets to the plain internet). This must be tested, not assumed.

Ingress (north-south) — summary

North-south exposure is handled by Gateway API implemented by Cilium, with TLS from cert-manager and DNS from ExternalDNS. Detailed in 09-dns-and-ingress.md. Exposure is a declared property: internal (LAN-only) vs public (port-forwarded at the edge).

Observability of the network (supports R-SEC-03, R-OBS-04)

• Hubble provides flow visibility to verify default-deny and troubleshoot allow rules.
• Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy reconciliation errors. Wired into 10-observability-and-alerting.md.

Acceptance gates for this spec

• A freshly onboarded service with no rules can reach nothing but DNS (default-deny proven via Hubble + a connectivity probe).
• Declared dependencies work; undeclared ones are blocked.
• A vpn service's public egress IP is the VPN's, not the home IP (verified).
• Kill-switch: tunnel down ⇒ vpn service has no internet path; internet services unaffected.
• Changing a service's egress class in git flips its path with no workload rearchitecting.

Open questions

• OQ-1 — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
• OQ-3 — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct egress IPs (affects Pattern A vs B and HA of the gateway).