home-network-k3s/docs/03-networking-isolation-and-egress.md

# 03 — Networking, Isolation & Egress

Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
[ADR-0015 (VPN egress steering)](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md),
[ADR-0020 (single gateway, Cilium fail-closed)](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md),
[ADR-0021 (inbound remote access)](../decisions/adr-0021-inbound-remote-access-wireguard.md).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
else is convenience.

## Principles

1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
   permits it, at the port/protocol level (R-NET-01/02).
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
   it accordingly (R-NET-05/07).
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
   during tunnel failure (R-NET-06, kill-switch).
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).

## CNI: Cilium

Cilium replaces the default k3s CNI and provides:
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
- **kube-proxy replacement** (eBPF) for efficient service routing.
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
  building block for WAN/VPN path selection.
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).

## Segmentation model

- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
  namespace. Namespace = isolation boundary.
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
  policy and egress routing.
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
  namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

**Platform label-key convention.** The spec fixes label *values* (e.g. `internet|vpn`,
`oidc|forward-auth|native|none`); the label *keys* the platform uses are:
`homelab.keithjackson.dev/egress-class`, `homelab.keithjackson.dev/auth-mode`, and the supporting
`homelab.keithjackson.dev/{app-namespace,baseline}`. These keys are the contract that network
policies, Kyverno rules, and renderers key off (values per this doc, [13](13-service-catalog.md),
and [15](15-identity-and-access.md)).

### Default-deny baseline (R-NET-01)
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
- **Deny all ingress** and **deny all egress** by default.
- Then explicitly allow only:
  - DNS to the in-cluster resolver (required for anything to function).
  - Declared intra-cluster dependencies (source → destination → port).
  - The declared egress class path (internet or VPN).
  - Observability scraping from the monitoring namespace (metrics/logs).

> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
> starting posture for every onboarded service.

### Explicit allow rules (R-NET-02)
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
final manifest):

```
seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)
```

Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has **one physical commercial internet WAN** and **no plans for additional ISP
uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
as multi-ISP routing but as **two egress paths selected by routing rules**:

| Profile | Meaning | Mechanism |
|---|---|---|
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0015, ADR-0020) |

- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
  route across; the `internet` path is simply the default route out.
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
  **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
  the edge. The edge's only jobs are NAT out and a few **manual port-forwards that are not web
  ingress**: WireGuard udp/51820 and Plex tcp/32400 ([09](09-dns-and-ingress.md)). Public web ingress
  uses an outbound-only **Cloudflare Tunnel** from in-cluster cloudflared (ADR-0016), so the edge does
  not forward 80/443.
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.

## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015, ADR-0020; provider ADR-0013)

VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
`hostNetwork` Gluetun hijacks the node's own default route).

**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the
local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to
the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite**
directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks
to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole
the wrong class.

The model:

- **Tunnel on the node, not in a pod.** Exactly one labeled **egress-gateway node** runs a **host OpenVPN
  systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
  **Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it
  creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the
  node's own traffic stays on the WAN — only what Cilium points at `tun0` uses the VPN. The tunnel is
  on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
- **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods
  labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the
  tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**;
  the service is just labeled `egress: vpn` and cannot leak on its own. Only public destinations are
  tunnelled; private ranges (the cluster, the LAN, bulk storage) stay on the local datapath.
- **Kill-switch (fail-closed) — Cilium's egress datapath (ADR-0020).** Steered traffic is forwarded in
  eBPF, and **Cilium drops it** whenever the gateway's tunnel interface cannot be used. If the tunnel is
  down, `vpn` pods have **no route to the plain internet**; `internet`-class services are unaffected.
  Steered pod traffic never reaches netfilter, so a **host nftables table** on the gateway is a narrow
  backstop only: it covers node-originated traffic sourced from a tunnel address.
- **Re-steering across reconnects.** Cilium resolves the tunnel interface once and does not follow its
  re-creation, so every reconnect is signalled back to Cilium from the gateway node, without writing to
  GitOps state. A reconnect Cilium was not told about raises an alert
  ([10](10-observability-and-alerting.md)).
- **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on
  the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert
  if PF acquisition fails (see [known gaps](16-known-gaps.md)).
- **Resilience: one gateway, no failover (ADR-0020).** Cilium OSS resolves the gateway selector to a
  single node, so a second labeled node is not a standby: it silently takes the role, and the public
  egress IP, by name order. **Label exactly one node.** Losing it is a `vpn`-class outage, fail-closed:
  no leak and no service. Alert on tunnel state ([10](10-observability-and-alerting.md)).
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node via **Ansible vault**
  ([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster
  SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) by conformance test
  ([14](14-testing-and-conformance.md)).

**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed.

## Remote access (inbound): WireGuard (ADR-0021)

The operator's devices reach the home network through a **WireGuard** listener on udp/51820. This is
the opposite direction to `vpn` egress and shares nothing with it: `vpn` egress sends pods *out*
through the provider tunnel; remote access lets clients *in*. It is the "VPN" in "LAN/VPN-only".

- **Host network, every node.** A Flux-managed listener runs in the host network namespace on all
  nodes. A pod-networked server would give every client that pod's Cilium identity, and every Gateway
  hostname would refuse it. A listener on each node makes loss of the forwarded node a one-line router
  change, with no new client profiles.
- **A deliberate exception to default-deny.** A host-network workload has no Cilium endpoint, so no
  network policy constrains it. The boundary around the port is **WireGuard's peer authentication**: a
  packet from an unknown key gets no reply. The exception applies to this platform component only;
  application namespaces may not use the host network.
- **Authorization is reviewable.** Peers are public keys in plain text in git, so granting or revoking
  a device is a reviewed change. The server's private key is SOPS-encrypted.
- **Reach equals a LAN machine.** An enrolled device can reach the whole LAN, including the nodes'
  control-plane ports, and egresses through the house WAN. A lost device must be revoked promptly.
  Clients stay `internet` class and are never steered to the provider tunnel.

## Ingress (north-south) — summary

North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
Exposure is a declared property: `internal` (LAN/VPN-only) vs `public` (published through the
Cloudflare Tunnel, ADR-0016).

## Observability of the network (supports R-SEC-03, R-OBS-04)
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
  reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).

## Acceptance gates for this spec
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
  Hubble + a connectivity probe).
- Declared dependencies work; **undeclared** ones are blocked.
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.

## Resolved
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
  segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
  Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare
  Tunnel.
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
  Egress Gateway** and failing closed in Cilium's datapath (ADR-0015, ADR-0020), rules enforced outside
  the service.

## Open questions
- Which **PF-enabled PIA region** to use, and the port-sync helper wiring for qBittorrent.