Initial baseline: home platform (k3s + Longhorn) specification repository
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
37 files changed
+2830
No files matched your search
@@ -0,0 +1,126 @@
|
||||
# 03 — Networking, Isolation & Egress
|
||||
|
||||
Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).
|
||||
|
||||
This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
|
||||
else is convenience.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
|
||||
permits it, at the port/protocol level (R-NET-01/02).
|
||||
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
|
||||
it accordingly (R-NET-05/07).
|
||||
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
|
||||
during tunnel failure (R-NET-06, kill-switch).
|
||||
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).
|
||||
|
||||
## CNI: Cilium
|
||||
|
||||
Cilium replaces the default k3s CNI and provides:
|
||||
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
|
||||
- **kube-proxy replacement** (eBPF) for efficient service routing.
|
||||
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
|
||||
building block for WAN/VPN path selection.
|
||||
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
|
||||
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).
|
||||
|
||||
## Segmentation model
|
||||
|
||||
- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
|
||||
namespace. Namespace = isolation boundary.
|
||||
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
|
||||
policy and egress routing.
|
||||
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
|
||||
namespaces; only the specific flows they need (e.g. metrics scraping) are opened.
|
||||
|
||||
### Default-deny baseline (R-NET-01)
|
||||
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
|
||||
- **Deny all ingress** and **deny all egress** by default.
|
||||
- Then explicitly allow only:
|
||||
- DNS to the in-cluster resolver (required for anything to function).
|
||||
- Declared intra-cluster dependencies (source → destination → port).
|
||||
- The declared egress class path (internet or VPN).
|
||||
- Observability scraping from the monitoring namespace (metrics/logs).
|
||||
|
||||
> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
|
||||
> starting posture for every onboarded service.
|
||||
|
||||
### Explicit allow rules (R-NET-02)
|
||||
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
|
||||
final manifest):
|
||||
|
||||
```
|
||||
seerr → sonarr tcp/8989 (Seerr requests to Sonarr)
|
||||
seerr → radarr tcp/7878 (Seerr requests to Radarr)
|
||||
seerr → plex tcp/32400 (Seerr talks to Plex)
|
||||
```
|
||||
|
||||
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
|
||||
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
|
||||
|
||||
## Multi-WAN & egress routing (R-NET-04/05)
|
||||
|
||||
The platform recognizes **egress profiles**. Initial profiles:
|
||||
|
||||
| Profile | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
|
||||
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
|
||||
|
||||
**Where multi-WAN policy routing lives (OQ-1):**
|
||||
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
|
||||
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
|
||||
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
|
||||
available WANs.
|
||||
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
|
||||
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
|
||||
*physical WAN* for the internet class.
|
||||
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
|
||||
is answered. The **service-facing contract does not change**: a service still just declares
|
||||
`egress: internet|vpn`.
|
||||
|
||||
## VPN egress model (R-NET-05/06 — ADR-0008)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
|
||||
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
|
||||
|
||||
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
|
||||
|
||||
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
|
||||
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
|
||||
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
|
||||
Simple, self-contained, per-service isolation; more sidecars to run.
|
||||
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
|
||||
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
|
||||
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
|
||||
fallback to the internet path.
|
||||
|
||||
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
|
||||
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
|
||||
assumed.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
||||
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
||||
|
||||
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
||||
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
||||
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
|
||||
reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).
|
||||
|
||||
## Acceptance gates for this spec
|
||||
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
|
||||
Hubble + a connectivity probe).
|
||||
- Declared dependencies work; **undeclared** ones are blocked.
|
||||
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
|
||||
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
|
||||
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
|
||||
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
|
||||
egress IPs (affects Pattern A vs B and HA of the gateway).
|
||||
Reference in new issue
Block a user