home-network-k3s/docs/03-networking-isolation-and-egress.md

03 — Networking, Isolation & Egress

Serves: R-NET-01..07, R-SEC-02. Related: ADR-0002 (Cilium), ADR-0015 (VPN egress steering), ADR-0020 (single gateway, Cilium fail-closed), ADR-0021 (inbound remote access).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience.

Principles

1. Deny by default, allow by exception. No service may talk to another unless a rule explicitly permits it, at the port/protocol level (R-NET-01/02).
2. Egress is a declared property. Each service declares an egress class; the platform routes it accordingly (R-NET-05/07).
3. VPN means VPN-only. A VPN-class service must never leak to the plain internet, including during tunnel failure (R-NET-06, kill-switch).
4. Isolation is auditable. An operator can prove what a service can and cannot reach (R-SEC-03).

CNI: Cilium

Cilium replaces the default k3s CNI and provides:

• NetworkPolicy / CiliumNetworkPolicy for L3/L4 (and optional L7) enforcement.
• kube-proxy replacement (eBPF) for efficient service routing.
• Egress Gateway to force selected pods' egress through a chosen node/identity/SNAT IP — the building block for WAN/VPN path selection.
• Hubble for flow observability — the audit and troubleshooting tool for isolation.
• Gateway API implementation for north-south ingress (see 09).

Segmentation model

• Namespaces as trust zones. Each service (or tightly-coupled service group) gets its own namespace. Namespace = isolation boundary.
• Egress classes as zones: internet and vpn are first-class labels that drive both network policy and egress routing.
• Platform namespace(s) (Flux, cert-manager, monitoring, storage) are isolated from application namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

Platform label-key convention. The spec fixes label values (e.g. internet|vpn, oidc|forward-auth|native|none); the label keys the platform uses are: homelab.keithjackson.dev/egress-class, homelab.keithjackson.dev/auth-mode, and the supporting homelab.keithjackson.dev/{app-namespace,baseline}. These keys are the contract that network policies, Kyverno rules, and renderers key off (values per this doc, 13, and 15).

Default-deny baseline (R-NET-01)

Every namespace gets a baseline policy set applied by GitOps at namespace creation:

• Deny all ingress and deny all egress by default.
• Then explicitly allow only:
  • DNS to the in-cluster resolver (required for anything to function).
  • Declared intra-cluster dependencies (source → destination → port).
  • The declared egress class path (internet or VPN).
  • Observability scraping from the monitoring namespace (metrics/logs).

A workload with no explicit allow rules can do nothing except resolve DNS. This is the intended starting posture for every onboarded service.

Explicit allow rules (R-NET-02)

Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not final manifest):

seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)

Rules are declared once in the service's catalog entry (its dependencies) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has one physical commercial internet WAN and no plans for additional ISP uplinks or physical WANs (OQ-1). "Multiple WANs" from the original intent is therefore realized not as multi-ISP routing but as two egress paths selected by routing rules:

Profile	Meaning	Mechanism
internet	Out the single commercial WAN	Normal pod egress via node → SOHO edge router → internet
vpn	Routed through the in-cluster VPN tunnel	Steered to the shared VPN egress gateway (ADR-0015, ADR-0020)

• Physical multi-WAN (multi-ISP) is out of scope. With a single WAN there is nothing to policy- route across; the internet path is simply the default route out.
• The edge router is a dumb SOHO device — no VLANs, no VPN, no services, no API/IaC. Therefore all segmentation and all VPN handling live in-cluster (Cilium policy + the VPN gateway), not on the edge. The edge's only jobs are NAT out and a few manual port-forwards that are not web ingress: WireGuard udp/51820 and Plex tcp/32400 (09). Public web ingress uses an outbound-only Cloudflare Tunnel from in-cluster cloudflared (ADR-0016), so the edge does not forward 80/443.
• The service-facing contract is unchanged: a service still just declares egress: internet|vpn.

VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015, ADR-0020; provider ADR-0013)

VPN-class services: sonarr, radarr, jackett, qbittorrent. Per OQ-3 the design is a single, shared VPN connection — not a per-service sidecar — so that network rules are enforced outside the service. The steering mechanism was settled by ADR-0015 after the original in-cluster-Gluetun approach proved un-steerable (a Gluetun pod's tun0 is invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own default route).

Default is the WAN; the VPN is strictly opt-in. Every pod — and the nodes themselves — egress the local/private WAN by default and never the VPN. Only services labeled egress-class=vpn are steered to the tunnel, and the kill-switch blackholes only that class. The two classes fail in opposite directions: internet fails open (WAN, never blackholed); vpn fails closed (blackhole, never leaks to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole the wrong class.

The model:

• Tunnel on the node, not in a pod. Exactly one labeled egress-gateway node runs a host OpenVPN systemd service — PIA over OpenVPN-UDP (provider/protocol per ADR-0013; "non-TCP or better"), Ansible-managed host infra (like k3s itself), NOT a Flux workload. It is split-tunnel: it creates tun0 but does not set the node's default route (route-noexec + policy routing), so the node's own traffic stays on the WAN — only what Cilium points at tun0 uses the VPN. The tunnel is on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
• vpn-class pods are steered by Cilium Egress Gateway. A CiliumEgressGatewayPolicy selects pods labeled egress-class=vpn and egresses them via the gateway node's tun0 (native eBPF SNAT to the tunnel IP). No client-pod changes, no sidecar, no overlay. They have no independent egress path; the service is just labeled egress: vpn and cannot leak on its own. Only public destinations are tunnelled; private ranges (the cluster, the LAN, bulk storage) stay on the local datapath.
• Kill-switch (fail-closed) — Cilium's egress datapath (ADR-0020). Steered traffic is forwarded in eBPF, and Cilium drops it whenever the gateway's tunnel interface cannot be used. If the tunnel is down, vpn pods have no route to the plain internet; internet-class services are unaffected. Steered pod traffic never reaches netfilter, so a host nftables table on the gateway is a narrow backstop only: it covers node-originated traffic sourced from a tunnel address.
• Re-steering across reconnects. Cilium resolves the tunnel interface once and does not follow its re-creation, so every reconnect is signalled back to Cilium from the gateway node, without writing to GitOps state. A reconnect Cilium was not told about raises an alert (10).
• Port forwarding for qBittorrent: PIA NAT-PMP, re-solved host-side (a small pf/natpmp helper on the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert if PF acquisition fails (see known gaps).
• Resilience: one gateway, no failover (ADR-0020). Cilium OSS resolves the gateway selector to a single node, so a second labeled node is not a standby: it silently takes the role, and the public egress IP, by name order. Label exactly one node. Losing it is a vpn-class outage, fail-closed: no leak and no service. Alert on tunnel state (10).
• Prerequisite: the PIA username/password + .ovpn on the gateway node via Ansible vault (07) — the tunnel is host infra, so its creds are host-side (not a cluster SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) by conformance test (14).

Kill-switch is an acceptance gate: with the tunnel forcibly down, every vpn service must fail closed (no packets to the plain internet), while internet-class is unaffected. Tested, not assumed.

Remote access (inbound): WireGuard (ADR-0021)

The operator's devices reach the home network through a WireGuard listener on udp/51820. This is the opposite direction to vpn egress and shares nothing with it: vpn egress sends pods out through the provider tunnel; remote access lets clients in. It is the "VPN" in "LAN/VPN-only".

• Host network, every node. A Flux-managed listener runs in the host network namespace on all nodes. A pod-networked server would give every client that pod's Cilium identity, and every Gateway hostname would refuse it. A listener on each node makes loss of the forwarded node a one-line router change, with no new client profiles.
• A deliberate exception to default-deny. A host-network workload has no Cilium endpoint, so no network policy constrains it. The boundary around the port is WireGuard's peer authentication: a packet from an unknown key gets no reply. The exception applies to this platform component only; application namespaces may not use the host network.
• Authorization is reviewable. Peers are public keys in plain text in git, so granting or revoking a device is a reviewed change. The server's private key is SOPS-encrypted.
• Reach equals a LAN machine. An enrolled device can reach the whole LAN, including the nodes' control-plane ports, and egresses through the house WAN. A lost device must be revoked promptly. Clients stay internet class and are never steered to the provider tunnel.

Ingress (north-south) — summary

North-south exposure is handled by Gateway API implemented by Cilium, with TLS from cert-manager and DNS from ExternalDNS. Detailed in 09-dns-and-ingress.md. Exposure is a declared property: internal (LAN/VPN-only) vs public (published through the Cloudflare Tunnel, ADR-0016).

Observability of the network (supports R-SEC-03, R-OBS-04)

• Hubble provides flow visibility to verify default-deny and troubleshoot allow rules.
• Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy reconciliation errors. Wired into 10-observability-and-alerting.md.

Acceptance gates for this spec

• A freshly onboarded service with no rules can reach nothing but DNS (default-deny proven via Hubble + a connectivity probe).
• Declared dependencies work; undeclared ones are blocked.
• A vpn service's public egress IP is the VPN's, not the home IP (verified).
• Kill-switch: tunnel down ⇒ vpn service has no internet path; internet services unaffected.
• Changing a service's egress class in git flips its path with no workload rearchitecting.

Resolved

• OQ-1 — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope. Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare Tunnel.
• OQ-3 — PIA over OpenVPN-UDP (ADR-0013); egress is a node-level host tunnel steered by Cilium Egress Gateway and failing closed in Cilium's datapath (ADR-0015, ADR-0020), rules enforced outside the service.

Open questions

• Which PF-enabled PIA region to use, and the port-sync helper wiring for qBittorrent.