jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
127 lines
6.8 KiB
Markdown
127 lines
6.8 KiB
Markdown
# 03 — Networking, Isolation & Egress
|
|
|
|
Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
|
|
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).
|
|
|
|
This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
|
|
else is convenience.
|
|
|
|
## Principles
|
|
|
|
1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
|
|
permits it, at the port/protocol level (R-NET-01/02).
|
|
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
|
|
it accordingly (R-NET-05/07).
|
|
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
|
|
during tunnel failure (R-NET-06, kill-switch).
|
|
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).
|
|
|
|
## CNI: Cilium
|
|
|
|
Cilium replaces the default k3s CNI and provides:
|
|
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
|
|
- **kube-proxy replacement** (eBPF) for efficient service routing.
|
|
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
|
|
building block for WAN/VPN path selection.
|
|
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
|
|
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).
|
|
|
|
## Segmentation model
|
|
|
|
- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
|
|
namespace. Namespace = isolation boundary.
|
|
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
|
|
policy and egress routing.
|
|
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
|
|
namespaces; only the specific flows they need (e.g. metrics scraping) are opened.
|
|
|
|
### Default-deny baseline (R-NET-01)
|
|
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
|
|
- **Deny all ingress** and **deny all egress** by default.
|
|
- Then explicitly allow only:
|
|
- DNS to the in-cluster resolver (required for anything to function).
|
|
- Declared intra-cluster dependencies (source → destination → port).
|
|
- The declared egress class path (internet or VPN).
|
|
- Observability scraping from the monitoring namespace (metrics/logs).
|
|
|
|
> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
|
|
> starting posture for every onboarded service.
|
|
|
|
### Explicit allow rules (R-NET-02)
|
|
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
|
|
final manifest):
|
|
|
|
```
|
|
seerr → sonarr tcp/8989 (Seerr requests to Sonarr)
|
|
seerr → radarr tcp/7878 (Seerr requests to Radarr)
|
|
seerr → plex tcp/32400 (Seerr talks to Plex)
|
|
```
|
|
|
|
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
|
|
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
|
|
|
|
## Multi-WAN & egress routing (R-NET-04/05)
|
|
|
|
The platform recognizes **egress profiles**. Initial profiles:
|
|
|
|
| Profile | Meaning | Mechanism |
|
|
|---|---|---|
|
|
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
|
|
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
|
|
|
|
**Where multi-WAN policy routing lives (OQ-1):**
|
|
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
|
|
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
|
|
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
|
|
available WANs.
|
|
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
|
|
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
|
|
*physical WAN* for the internet class.
|
|
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
|
|
is answered. The **service-facing contract does not change**: a service still just declares
|
|
`egress: internet|vpn`.
|
|
|
|
## VPN egress model (R-NET-05/06 — ADR-0008)
|
|
|
|
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
|
|
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
|
|
|
|
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
|
|
|
|
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
|
|
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
|
|
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
|
|
Simple, self-contained, per-service isolation; more sidecars to run.
|
|
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
|
|
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
|
|
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
|
|
fallback to the internet path.
|
|
|
|
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
|
|
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
|
|
assumed.
|
|
|
|
## Ingress (north-south) — summary
|
|
|
|
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
|
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
|
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
|
|
|
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
|
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
|
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
|
|
reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).
|
|
|
|
## Acceptance gates for this spec
|
|
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
|
|
Hubble + a connectivity probe).
|
|
- Declared dependencies work; **undeclared** ones are blocked.
|
|
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
|
|
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
|
|
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
|
|
|
|
## Open questions
|
|
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
|
|
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
|
|
egress IPs (affects Pattern A vs B and HA of the gateway).
|