home-network-k3s/docs/03-networking-isolation-and-egress.md

# 03 — Networking, Isolation & Egress

Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).

This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
else is convenience.

## Principles

1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
   permits it, at the port/protocol level (R-NET-01/02).
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
   it accordingly (R-NET-05/07).
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
   during tunnel failure (R-NET-06, kill-switch).
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).

## CNI: Cilium

Cilium replaces the default k3s CNI and provides:
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
- **kube-proxy replacement** (eBPF) for efficient service routing.
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
  building block for WAN/VPN path selection.
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).

## Segmentation model

- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
  namespace. Namespace = isolation boundary.
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
  policy and egress routing.
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
  namespaces; only the specific flows they need (e.g. metrics scraping) are opened.

**Platform label-key convention.** The spec fixes label *values* (e.g. `internet|vpn`,
`oidc|forward-auth|native|none`); the label *keys* the platform uses are:
`homelab.keithjackson.dev/egress-class`, `homelab.keithjackson.dev/auth-mode`, and the supporting
`homelab.keithjackson.dev/{app-namespace,baseline}`. These keys are the contract that network
policies, Kyverno rules, and renderers key off (values per this doc, [13](13-service-catalog.md),
and [15](15-identity-and-access.md)).

### Default-deny baseline (R-NET-01)
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
- **Deny all ingress** and **deny all egress** by default.
- Then explicitly allow only:
  - DNS to the in-cluster resolver (required for anything to function).
  - Declared intra-cluster dependencies (source → destination → port).
  - The declared egress class path (internet or VPN).
  - Observability scraping from the monitoring namespace (metrics/logs).

> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
> starting posture for every onboarded service.

### Explicit allow rules (R-NET-02)
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
final manifest):

```
seerr  →  sonarr    tcp/8989   (Seerr requests to Sonarr)
seerr  →  radarr    tcp/7878   (Seerr requests to Radarr)
seerr  →  plex      tcp/32400  (Seerr talks to Plex)
```

Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.

## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)

The environment has **one physical commercial internet WAN** and **no plans for additional ISP
uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
as multi-ISP routing but as **two egress paths selected by routing rules**:

| Profile | Meaning | Mechanism |
|---|---|---|
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0013) |

- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
  route across; the `internet` path is simply the default route out.
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
  **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
  the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
  ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.

## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)

VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
`hostNetwork` Gluetun hijacks the node's own default route).

**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the
local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to
the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite**
directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks
to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole
the wrong class.

The model:

- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
  systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
  **Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it
  creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the
  node's own traffic stays on the WAN — only what Cilium points at `tun0` uses the VPN. The tunnel is
  on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
- **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods
  labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the
  tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**;
  the service is just labeled `egress: vpn` and cannot leak on its own.
- **Kill-switch (fail-closed) — host nftables.** An **nftables rule on the gateway node** DROPS steered
  (`egress-class=vpn`) traffic whenever `tun0` is down — a host backstop independent of any Cilium
  egress-gateway fail-open edge case. If the tunnel is down, `vpn` pods have **no route to the plain
  internet**; `internet`-class services are unaffected.
- **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on
  the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert
  if PF acquisition fails.
- **Resilience:** two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) give the
  active/standby resilience; alert on tunnel state ([10](10-observability-and-alerting.md)).
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node(s) via **Ansible vault**
  ([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster
  SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test.

**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed.

## Ingress (north-south) — summary

North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).

## Observability of the network (supports R-SEC-03, R-OBS-04)
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
  reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).

## Acceptance gates for this spec
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
  Hubble + a connectivity probe).
- Declared dependencies work; **undeclared** ones are blocked.
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.

## Resolved
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
  segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
  Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service.

## Open questions
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
  WS-NET implementation detail; the kill-switch acceptance test is the gate.
- Which **PF-enabled PIA region** to use, and the port-sync helper wiring for qBittorrent.