home-network-k3s/docs/03-networking-isolation-and-egress.md

03 — Networking, Isolation & Egress

Serves: R-NET-01..07, R-SEC-02. Related: ADR-0002 (Cilium), ADR-0008 (VPN egress).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience.

Principles

1. Deny by default, allow by exception. No service may talk to another unless a rule explicitly permits it, at the port/protocol level (R-NET-01/02).
2. Egress is a declared property. Each service declares an egress class; the platform routes it accordingly (R-NET-05/07).
3. VPN means VPN-only. A VPN-class service must never leak to the plain internet, including during tunnel failure (R-NET-06, kill-switch).
4. Isolation is auditable. An operator can prove what a service can and cannot reach (R-SEC-03).

CNI: Cilium

Cilium replaces the default k3s CNI and provides:

• NetworkPolicy / CiliumNetworkPolicy for L3/L4 (and optional L7) enforcement.
• kube-proxy replacement (eBPF) for efficient service routing.
• Egress Gateway to force selected pods' egress through a chosen node/identity/SNAT IP — the building block for WAN/VPN path selection.
• Hubble for flow observability — the audit and troubleshooting tool for isolation.
• Gateway API implementation for north-south ingress (see 09).

Segmentation model

• Namespaces as trust zones. Each service (or tightly-coupled service group) gets its own namespace. Namespace = isolation boundary.
• Egress classes as zones: internet and vpn are first-class labels that drive both network policy and egress routing.
• Platform namespace(s) (Flux, cert-manager, monitoring, storage) are isolated from application namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

Platform label-key convention. The spec fixes label values (e.g. internet|vpn, oidc|forward-auth|native|none); the label keys the platform uses are: homelab.keithjackson.dev/egress-class, homelab.keithjackson.dev/auth-mode, and the supporting homelab.keithjackson.dev/{app-namespace,baseline}. These keys are the contract that network policies, Kyverno rules, and renderers key off (values per this doc, 13, and 15).

Default-deny baseline (R-NET-01)

Every namespace gets a baseline policy set applied by GitOps at namespace creation:

• Deny all ingress and deny all egress by default.
• Then explicitly allow only:
  • DNS to the in-cluster resolver (required for anything to function).
  • Declared intra-cluster dependencies (source → destination → port).
  • The declared egress class path (internet or VPN).
  • Observability scraping from the monitoring namespace (metrics/logs).

A workload with no explicit allow rules can do nothing except resolve DNS. This is the intended starting posture for every onboarded service.

Explicit allow rules (R-NET-02)

Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not final manifest):

seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)

Rules are declared once in the service's catalog entry (its dependencies) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has one physical commercial internet WAN and no plans for additional ISP uplinks or physical WANs (OQ-1). "Multiple WANs" from the original intent is therefore realized not as multi-ISP routing but as two egress paths selected by routing rules:

Profile	Meaning	Mechanism
internet	Out the single commercial WAN	Normal pod egress via node → SOHO edge router → internet
vpn	Routed through the in-cluster VPN tunnel	Steered to the shared VPN egress gateway (ADR-0013)

• Physical multi-WAN (multi-ISP) is out of scope. With a single WAN there is nothing to policy- route across; the internet path is simply the default route out.
• The edge router is a dumb SOHO device — no VLANs, no VPN, no services, no API/IaC. Therefore all segmentation and all VPN handling live in-cluster (Cilium policy + the VPN gateway), not on the edge. The edge's only jobs are NAT out and a few manual port-forwards that are not web ingress: WireGuard udp/51820 and Plex tcp/32400 (09). Public web ingress uses an outbound-only Cloudflare Tunnel from in-cluster cloudflared (ADR-0016), so the edge does not forward 80/443.
• The service-facing contract is unchanged: a service still just declares egress: internet|vpn.

VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)

VPN-class services: sonarr, radarr, jackett, qbittorrent. Per OQ-3 the design is a single, shared VPN connection — not a per-service sidecar — so that network rules are enforced outside the service. The steering mechanism was settled by ADR-0015 after the original in-cluster-Gluetun approach proved un-steerable (a Gluetun pod's tun0 is invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own default route).

Default is the WAN; the VPN is strictly opt-in. Every pod — and the nodes themselves — egress the local/private WAN by default and never the VPN. Only services labeled egress-class=vpn are steered to the tunnel, and the kill-switch blackholes only that class. The two classes fail in opposite directions: internet fails open (WAN, never blackholed); vpn fails closed (blackhole, never leaks to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole the wrong class.

The model:

• Tunnel on the node, not in a pod. One or two labeled egress-gateway nodes run a host OpenVPN systemd service — PIA over OpenVPN-UDP (provider/protocol per ADR-0013; "non-TCP or better"), Ansible-managed host infra (like k3s itself), NOT a Flux workload. It is split-tunnel: it creates tun0 but does not set the node's default route (route-noexec + policy routing), so the node's own traffic stays on the WAN — only what Cilium points at tun0 uses the VPN. The tunnel is on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
• vpn-class pods are steered by Cilium Egress Gateway. A CiliumEgressGatewayPolicy selects pods labeled egress-class=vpn and egresses them via the gateway node's tun0 (native eBPF SNAT to the tunnel IP). No client-pod changes, no sidecar, no overlay. They have no independent egress path; the service is just labeled egress: vpn and cannot leak on its own.
• Kill-switch (fail-closed) — host nftables. An nftables rule on the gateway node DROPS steered (egress-class=vpn) traffic whenever tun0 is down — a host backstop independent of any Cilium egress-gateway fail-open edge case. If the tunnel is down, vpn pods have no route to the plain internet; internet-class services are unaffected.
• Port forwarding for qBittorrent: PIA NAT-PMP, re-solved host-side (a small pf/natpmp helper on the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert if PF acquisition fails.
• Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) give the active/standby resilience; alert on tunnel state (10).
• Prerequisite: the PIA username/password + .ovpn on the gateway node(s) via Ansible vault (07) — the tunnel is host infra, so its creds are host-side (not a cluster SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test.

Kill-switch is an acceptance gate: with the tunnel forcibly down, every vpn service must fail closed (no packets to the plain internet), while internet-class is unaffected. Tested, not assumed.

Ingress (north-south) — summary

North-south exposure is handled by Gateway API implemented by Cilium, with TLS from cert-manager and DNS from ExternalDNS. Detailed in 09-dns-and-ingress.md. Exposure is a declared property: internal (LAN/VPN-only) vs public (published through the Cloudflare Tunnel, ADR-0016).

Observability of the network (supports R-SEC-03, R-OBS-04)

• Hubble provides flow visibility to verify default-deny and troubleshoot allow rules.
• Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy reconciliation errors. Wired into 10-observability-and-alerting.md.

Acceptance gates for this spec

• A freshly onboarded service with no rules can reach nothing but DNS (default-deny proven via Hubble + a connectivity probe).
• Declared dependencies work; undeclared ones are blocked.
• A vpn service's public egress IP is the VPN's, not the home IP (verified).
• Kill-switch: tunnel down ⇒ vpn service has no internet path; internet services unaffected.
• Changing a service's egress class in git flips its path with no workload rearchitecting.

Resolved

• OQ-1 — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope. Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare Tunnel.
• OQ-3 — PIA over OpenVPN-UDP (ADR-0013); egress is a node-level host tunnel steered by Cilium Egress Gateway with a host nftables kill-switch (ADR-0015), rules enforced outside the service.

Open questions

• Exact steering mechanism for vpn pods → gateway (Cilium Egress Gateway vs routing/SNAT) — WS-NET implementation detail; the kill-switch acceptance test is the gate.
• Which PF-enabled PIA region to use, and the port-sync helper wiring for qBittorrent.