home-network-k3s/docs/03-networking-isolation-and-egress.md

03 — Networking, Isolation & Egress

Serves: R-NET-01..07, R-SEC-02. Related: ADR-0002 (Cilium), ADR-0008 (VPN egress).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience.

Principles

1. Deny by default, allow by exception. No service may talk to another unless a rule explicitly permits it, at the port/protocol level (R-NET-01/02).
2. Egress is a declared property. Each service declares an egress class; the platform routes it accordingly (R-NET-05/07).
3. VPN means VPN-only. A VPN-class service must never leak to the plain internet, including during tunnel failure (R-NET-06, kill-switch).
4. Isolation is auditable. An operator can prove what a service can and cannot reach (R-SEC-03).

CNI: Cilium

Cilium replaces the default k3s CNI and provides:

• NetworkPolicy / CiliumNetworkPolicy for L3/L4 (and optional L7) enforcement.
• kube-proxy replacement (eBPF) for efficient service routing.
• Egress Gateway to force selected pods' egress through a chosen node/identity/SNAT IP — the building block for WAN/VPN path selection.
• Hubble for flow observability — the audit and troubleshooting tool for isolation.
• Gateway API implementation for north-south ingress (see 09).

Segmentation model

• Namespaces as trust zones. Each service (or tightly-coupled service group) gets its own namespace. Namespace = isolation boundary.
• Egress classes as zones: internet and vpn are first-class labels that drive both network policy and egress routing.
• Platform namespace(s) (Flux, cert-manager, monitoring, storage) are isolated from application namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

Default-deny baseline (R-NET-01)

Every namespace gets a baseline policy set applied by GitOps at namespace creation:

• Deny all ingress and deny all egress by default.
• Then explicitly allow only:
  • DNS to the in-cluster resolver (required for anything to function).
  • Declared intra-cluster dependencies (source → destination → port).
  • The declared egress class path (internet or VPN).
  • Observability scraping from the monitoring namespace (metrics/logs).

A workload with no explicit allow rules can do nothing except resolve DNS. This is the intended starting posture for every onboarded service.

Explicit allow rules (R-NET-02)

Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not final manifest):

seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)

Rules are declared once in the service's catalog entry (its dependencies) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has one physical commercial internet WAN and no plans for additional ISP uplinks or physical WANs (OQ-1). "Multiple WANs" from the original intent is therefore realized not as multi-ISP routing but as two egress paths selected by routing rules:

Profile	Meaning	Mechanism
internet	Out the single commercial WAN	Normal pod egress via node → SOHO edge router → internet
vpn	Routed through the in-cluster VPN tunnel	Steered to the shared VPN egress gateway (ADR-0013)

• Physical multi-WAN (multi-ISP) is out of scope. With a single WAN there is nothing to policy- route across; the internet path is simply the default route out.
• The edge router is a dumb SOHO device — no VLANs, no VPN, no services, no API/IaC. Therefore all segmentation and all VPN handling live in-cluster (Cilium policy + the VPN gateway), not on the edge. The edge's only jobs are NAT out and port-forwarding 80/443 (and specific service ports) to the ingress VIP for public services — a documented manual step (09).
• The service-facing contract is unchanged: a service still just declares egress: internet|vpn.

VPN egress model: single shared PIA/Gluetun gateway (R-NET-05/06 — ADR-0013, supersedes ADR-0012/0008)

VPN-class services: sonarr, radarr, jackett, qbittorrent. Per OQ-3 the design is a single, shared VPN connection — not a per-service sidecar — so that network rules are enforced outside the service:

• One in-cluster VPN egress gateway (an HA-capable Deployment) runs Gluetun configured for PIA over WireGuard (UDP — "non-TCP or better"). The cluster provisions and controls the tunnel (nothing on the edge router). Gluetun provides the built-in kill-switch/firewall, health checks, and auto-reconnect.
• vpn-class pods are steered through the gateway (Cilium Egress Gateway / routing); they have no independent egress path. The kill-switch and allow rules are enforced at the gateway + namespace/policy level, i.e. outside each service — the service is just labeled egress: vpn and cannot leak on its own.
• Kill-switch (fail-closed) is mandatory and enforced centrally: if the tunnel is down, vpn pods have no route to the plain internet; only the tunnel path exists. internet-class services are unaffected.
• Port forwarding for qBittorrent: Gluetun native PF (VPN_PORT_FORWARDING=on) on a PF-enabled PIA region; a port-sync helper (e.g. qbittorrent-natmap) writes the assigned port into qBittorrent so it tracks reconnects — kept hands-off. Alert if PF acquisition fails.
• Trade-off (documented): one shared tunnel = one shared egress IP and a single failure domain for vpn services → run the gateway active/standby for resilience; alert on tunnel state (10).
• Prerequisite (simple): just the PIA username/password, stored SOPS-encrypted (07) — no manual .ovpn extraction. Validate end-to-end (tunnel + kill-switch + PF) in dev/test.

Kill-switch is an acceptance gate: with the tunnel forcibly down, every vpn service must fail closed (no packets to the plain internet). Tested, not assumed.

Ingress (north-south) — summary

North-south exposure is handled by Gateway API implemented by Cilium, with TLS from cert-manager and DNS from ExternalDNS. Detailed in 09-dns-and-ingress.md. Exposure is a declared property: internal (LAN-only) vs public (port-forwarded at the edge).

Observability of the network (supports R-SEC-03, R-OBS-04)

• Hubble provides flow visibility to verify default-deny and troubleshoot allow rules.
• Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy reconciliation errors. Wired into 10-observability-and-alerting.md.

Acceptance gates for this spec

• A freshly onboarded service with no rules can reach nothing but DNS (default-deny proven via Hubble + a connectivity probe).
• Declared dependencies work; undeclared ones are blocked.
• A vpn service's public egress IP is the VPN's, not the home IP (verified).
• Kill-switch: tunnel down ⇒ vpn service has no internet path; internet services unaffected.
• Changing a service's egress class in git flips its path with no workload rearchitecting.

Resolved

• OQ-1 — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
• OQ-3 — PIA via Gluetun, single shared in-cluster tunnel over WireGuard, native port forwarding, rules enforced outside the service (ADR-0013, supersedes ADR-0012's ExpressVPN choice).

Open questions

• Exact steering mechanism for vpn pods → gateway (Cilium Egress Gateway vs routing/SNAT) — WS-NET implementation detail; the kill-switch acceptance test is the gate.
• Which PF-enabled PIA region to use, and the port-sync helper wiring for qBittorrent.