From a2cbdf656d42cf14945c08866c6cfe733ac2ca56 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 15 Aug 2026 23:00:11 -0500 Subject: [PATCH] Resolve OQ-1/2/3, refine OQ-4: single ExpressVPN gateway, single WAN, external media Operator answers folded into the specs: - OQ-3 VPN: ExpressVPN, cluster-controlled SINGLE shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced outside the service. New ADR-0012 supersedes ADR-0008 (Gluetun-sidecar default dropped). Central kill-switch. - OQ-1 WAN/edge: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). Multi-WAN out of scope; segmentation + VPN in-cluster; edge = NAT + manual port-forward. Gateway VIP resolved to Cilium L2 announcement. T-3.3 marked won't-do. - OQ-2 storage: bulk media lives outside the cluster (backing TBD, non-blocking) via a 'bulk' external-mount indirection; Plex metadata DB stays on Longhorn. Split out a separate OQ-BKP for the off-cluster DR backup target (Phase 4). - OQ-4 refined: wildcard record covers reachability but NOT ACME DNS-01 (needs TXT-write access) and does not match *.home.* depth. Still OPEN: Cloudflare API token (Option A) vs acme-dns delegation (Option B). T-2.4/T-2.5 blocked. Updated: docs 00,01,03,04,08,09; ADR-0008/0012 + index; task board. Co-Authored-By: Claude Opus 4.8 (1M context) --- agents/task-board.md | 32 ++++---- decisions/README.md | 3 +- decisions/adr-0008-vpn-egress-model.md | 6 +- ...12-vpn-egress-single-expressvpn-gateway.md | 56 +++++++++++++ docs/00-vision-and-scope.md | 41 +++++++--- docs/01-architecture-overview.md | 2 +- docs/03-networking-isolation-and-egress.md | 79 +++++++++++-------- docs/04-storage-and-state.md | 31 +++++--- docs/08-tls-and-certificates.md | 25 +++++- docs/09-dns-and-ingress.md | 34 ++++++-- 10 files changed, 222 insertions(+), 87 deletions(-) create mode 100644 decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md diff --git a/agents/task-board.md b/agents/task-board.md index 803dffa..720224d 100644 --- a/agents/task-board.md +++ b/agents/task-board.md @@ -36,29 +36,30 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst ## Phase 2 — Platform Core (net/storage/TLS/DNS/ingress) | ID | Task | WS | Serves | Blocked-by | Owner | Status | |---|---|---|---|---|---|---| -| T-2.1 | Longhorn (3-replica StorageClass) + bulk-tier plan (OQ-2) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo | +| T-2.1 | Longhorn (3-replica StorageClass) + `bulk` external-mount indirection (media lives off-cluster, backing TBD — OQ-2 resolved) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo | | T-2.2 | Default-deny network baseline (per-namespace) + Hubble | WS-NET | R-NET-01/03 | T-1.3 | — | todo | | T-2.3 | Render `dependencies` → CiliumNetworkPolicy from ServiceDefinition | WS-NET/WS-GITOPS | R-NET-02 | T-0.3, T-2.2 | — | todo | -| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` | WS-SEC | R-TLS-01/02/03 | T-1.4 | — | todo | -| T-2.5 | ExternalDNS + Gateway API listeners + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo | +| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` (**blocked on OQ-4**: Cloudflare token vs acme-dns) | WS-SEC | R-TLS-01/02/03 | T-1.4, OQ-4 | — | blocked | +| T-2.5 | ExternalDNS + Gateway API listeners (VIP via **Cilium L2 announcement**) + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo | | T-2.6 | Prove: default-deny (DNS-only probe), PVC survives rebuild, trusted cert, auto-DNS | WS-NET/STORAGE/SEC | R-NET/STATE/TLS/DNS | T-2.1..2.5 | — | todo | | T-2.7 | Deploy **Kyverno in-cluster as admission enforcer** (same policies as the CI L0 gate) | WS-TEST/WS-SEC | R-SEC-02, R-NET-01, R-SEC-01 | T-1.4, T-0.5 | — | todo | -## Phase 3 — Egress, VPN & Multi-WAN +## Phase 3 — Egress & VPN (single WAN; no multi-WAN — OQ-1) | ID | Task | WS | Serves | Blocked-by | Owner | Status | |---|---|---|---|---|---|---| -| T-3.1 | Egress classes; Cilium Egress Gateway wiring | WS-NET | R-NET-04/05 | T-2.2 | — | todo | -| T-3.2 | VPN egress (Gluetun sidecar) + kill-switch (ADR-0008) | WS-NET | R-NET-05/06 | T-3.1, OQ-3 | — | todo | -| T-3.3 | Edge-router multi-WAN split for `internet` class (OQ-1) | WS-NET | R-NET-04 | OQ-1 | — | todo | | T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo | -| T-3.4 | Prove: VPN IP egress, kill-switch fail-closed, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo | +| T-3.1 | Egress classes; Cilium Egress Gateway wiring (steer `vpn` pods → gateway) | WS-NET | R-NET-04/05 | T-2.2 | — | todo | +| T-3.2p | **Prereq:** obtain ExpressVPN manual OpenVPN-UDP config + manual creds; store SOPS-encrypted | WS-SEC | R-NET-05 | — | — | todo | +| T-3.2 | **Single shared in-cluster ExpressVPN gateway** (OpenVPN UDP, active/standby) + centrally-enforced kill-switch (ADR-0012) | WS-NET | R-NET-05/06 | T-3.1, T-3.2p | — | todo | +| ~~T-3.3~~ | ~~Edge-router multi-WAN split~~ — **out of scope** (single WAN, dumb SOHO edge; OQ-1) | WS-NET | R-NET-04 | — | — | won't-do | +| T-3.4 | Prove: VPN IP egress, **kill-switch fail-closed (central)**, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo | ## Phase 4 — Observability, Backup/DR & Updates | ID | Task | WS | Serves | Blocked-by | Owner | Status | |---|---|---|---|---|---|---| | T-4.1 | kube-prometheus-stack + Loki + Alloy + Alertmanager(email) | WS-OBS | R-OBS-01/02/03 | T-2.1 | — | todo | | T-4.2 | Platform-health alerts (node/Longhorn/Flux/cert/VPN/backup) | WS-OBS | R-OBS-04 | T-4.1 | — | todo | -| T-4.3 | Velero + Longhorn off-cluster backup schedule | WS-STORAGE | R-STATE-03 | T-2.1, OQ-2 | — | todo | +| T-4.3 | Velero + Longhorn off-cluster backup schedule (**needs backup-target choice** — S3/NFS endpoint) | WS-STORAGE | R-STATE-03 | T-2.1, OQ-BKP | — | todo | | T-4.4 | DR drill: restore namespace + volumes from off-cluster backup | WS-STORAGE | R-STATE-03 | T-4.3 | — | todo | | T-4.5 | Flux image-automation + Renovate + per-service update policy | WS-UPDATE | R-UPD-01/02/03 | T-0.3, T-4.1 | — | todo | | T-4.6 | Prove: auto stable update on schedule, bad-update auto-rollback+alert, pin holds | WS-UPDATE | R-UPD-* | T-4.5 | — | todo | @@ -93,10 +94,11 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst --- -## Open questions (resolve to unblock) -| OQ | Question | Blocks | Owner | +## Open questions +| OQ | Question | Blocks | Status | |---|---|---|---| -| OQ-1 | WAN uplink count + edge-router product | T-3.3 | — | -| OQ-2 | Bulk-media home + backup posture | T-2.1, T-4.3 | — | -| OQ-3 | VPN provider/protocol + concurrent-tunnel limits | T-3.2 | — | -| OQ-4 | DNS provider API (Cloudflare) token for DNS-01 + ExternalDNS | T-2.4, T-2.5 | — | +| OQ-1 | WAN count + edge router | — | ✅ resolved: 1 WAN, dumb SOHO edge; multi-WAN out of scope; VPN + segmentation in-cluster | +| OQ-2 | Bulk-media home | — | ✅ resolved: media outside cluster, backing TBD later (not blocking) | +| OQ-3 | VPN provider/protocol | — | ✅ resolved: ExpressVPN, single shared in-cluster tunnel, OpenVPN UDP (ADR-0012) | +| **OQ-4** | **DNS-01 challenge access** | **T-2.4, T-2.5** | ⛔ **OPEN** — Cloudflare API token (Option A, recommended) vs acme-dns delegation (Option B) | +| **OQ-BKP** | **Off-cluster DR backup target** (S3/NFS endpoint) | T-4.3 | ⛔ **OPEN** — needed by Phase 4, not Phase 2 | diff --git a/decisions/README.md b/decisions/README.md index 353206d..eaeb02f 100644 --- a/decisions/README.md +++ b/decisions/README.md @@ -49,7 +49,8 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne | [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 | | [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 | | [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 | -| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | Accepted | R-NET-05/06 | +| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | ~~Superseded by 0012~~ | R-NET-05/06 | | [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 | | [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 | | [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 | +| [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | Accepted | R-NET-05/06/07 | diff --git a/decisions/adr-0008-vpn-egress-model.md b/decisions/adr-0008-vpn-egress-model.md index a37d215..22da948 100644 --- a/decisions/adr-0008-vpn-egress-model.md +++ b/decisions/adr-0008-vpn-egress-model.md @@ -1,9 +1,13 @@ # ADR-0008 — VPN egress model: Gluetun sidecar (default) + Cilium Egress Gateway -- Status: Accepted +- Status: **Superseded by [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md)** (2026-08-15) - Date: 2026-08-15 - Serves: R-NET-05, R-NET-06, R-NET-07 - Supersedes: — +> **Superseded.** OQ-3 resolved to ExpressVPN + a **single shared in-cluster tunnel** with rules +> enforced outside the service. The Gluetun-sidecar-per-pod default below is no longer the model; see +> [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md). Retained for history. + ## Context A subset of services (sonarr, radarr, jackett, qbittorrent) must egress **only** through a commercial VPN, with a **kill-switch** guaranteeing no leak to the plain internet if the tunnel drops. Egress diff --git a/decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md b/decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md new file mode 100644 index 0000000..d00a84d --- /dev/null +++ b/decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md @@ -0,0 +1,56 @@ +# ADR-0012 — VPN egress: single shared in-cluster ExpressVPN gateway +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-NET-05, R-NET-06, R-NET-07 +- Supersedes: [ADR-0008](adr-0008-vpn-egress-model.md) + +## Context +ADR-0008 offered two VPN-egress patterns (Gluetun sidecar per pod as default, central gateway as +alternative) and left the provider open (OQ-3). The operator has now specified (OQ-3): +- Provider is **ExpressVPN**; the **cluster must provision and control** the connection (nothing on + the edge router — the edge is a dumb SOHO device with no VPN, OQ-1). +- Protocol **must not be TCP**; UDP or better is required, data must be protected. +- A **single shared VPN connection** is preferred over per-service sidecars, **so network rules are + enforced outside the service**. + +This changes the decision, so ADR-0008 is superseded. + +## Decision +Adopt a **single, shared, in-cluster VPN egress gateway** as the mandated model (ADR-0008's +"Pattern B" becomes the only pattern; the per-service Gluetun sidecar default is dropped): + +- **One VPN egress gateway** (HA-capable Deployment, run **active/standby**) holds a **single + ExpressVPN tunnel** using a **non-TCP protocol**: **OpenVPN UDP** (via ExpressVPN's manual + OpenVPN-UDP config), or **Lightway-UDP** if a containerizable client proves viable. The cluster — + not the edge — provisions and controls the tunnel. +- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing) and have + **no independent egress path**. The **kill-switch (fail-closed)** and egress allow rules are + enforced at the **gateway + namespace/policy level — outside each service**. A service is merely + labeled `egress: vpn`; it cannot leak on its own even if misconfigured. +- Credentials (ExpressVPN manual OpenVPN config + manual-config username/password) are stored + **SOPS-encrypted** ([ADR-0007](adr-0007-secrets-sops-age.md)). +- The `egress: internet | vpn` service contract is unchanged (R-NET-07). + +## Alternatives considered +- **ADR-0008 Pattern A (Gluetun sidecar per pod)** — rejected per OQ-3: puts VPN inside each service + (rules enforced *in* the service), multiplies tunnels, and isn't the operator's preferred single- + connection model. +- **VPN on the edge router** — impossible: the SOHO edge has no VPN capability (OQ-1); the cluster + must own the tunnel. +- **WireGuard** — ExpressVPN does not offer WireGuard; OpenVPN UDP (or Lightway-UDP) is the non-TCP + path. +- **OpenVPN TCP** — rejected: operator requires non-TCP. + +## Consequences +- Rules enforced outside the service (operator's goal); one tunnel to manage; simpler per-service + workloads (no sidecars). +- **Single shared egress IP** for all `vpn` services and a **single failure domain** → mitigate with + **active/standby** gateway and tunnel-state alerting ([10](../docs/10-observability-and-alerting.md)). +- **ExpressVPN is not a first-class containerized provider**: expect a **custom OpenVPN-UDP** setup + (e.g. gluetun custom-provider mode or a purpose-built OpenVPN client), and the operator must obtain + the manual OpenVPN config + manual credentials. Tunnel + kill-switch **must be validated in + dev/test** (fault-injection test, R-TEST-05). +- Exact pod→gateway steering (Cilium Egress Gateway vs routing/SNAT) is a WS-NET implementation + detail; the **kill-switch acceptance test is the gate** ([14](../docs/14-testing-and-conformance.md)). +- If ExpressVPN containerization proves impractical, revisit provider choice via a superseding ADR + (escalation to the operator — agents may not change this decision themselves). diff --git a/docs/00-vision-and-scope.md b/docs/00-vision-and-scope.md index d317e7a..fcef8d3 100644 --- a/docs/00-vision-and-scope.md +++ b/docs/00-vision-and-scope.md @@ -74,10 +74,15 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. destination → port/protocol. - **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced boundaries). -- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing - rules. +- **R-NET-04** — The platform MUST support **multiple egress paths** selected by routing rules. Per + OQ-1 the environment has **one physical commercial WAN and no multi-ISP plans**, so this is + realized as two paths — the local `internet` path and the in-cluster `vpn` path — not multi-ISP + routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens + it. - **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local - network/internet) or `vpn` (routed through a custom VPN connection). + network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced + **outside the service** (a single shared in-cluster gateway), so a service cannot leak by + misconfiguration (ADR-0012, OQ-3). - **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the service MUST NOT leak traffic to the standard internet. - **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps @@ -176,14 +181,24 @@ demonstration. The testing approach is specified in implement against the spec and prove conformance with tests; they never edit the spec to fit the implementation. -## Open questions (tracked, not blocking the architecture) +## Open questions -- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do - the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See - [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md). -- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk - vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003. -- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many - concurrent tunnels/egress IPs are available. -- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an - API token scoped for ACME DNS-01 and ExternalDNS. +### Resolved (2026-08-15) +- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no + VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical + multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md). +- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the + cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's + metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app + config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md). +- **OQ-3 (VPN)** — **RESOLVED**: **ExpressVPN**, cluster-provisioned, **single shared in-cluster + tunnel**, **non-TCP** (OpenVPN UDP / Lightway-UDP), rules enforced outside the service + (ADR-0012). See [`03`](03-networking-isolation-and-egress.md). + +### Still open +- **OQ-4 (TLS/DNS challenge access)**: Cloudflare already has a wildcard `*.keithjackson.dev` → public + IP (covers reachability), but **ACME DNS-01 needs write access** to create `_acme-challenge` TXT + records, and `*.keithjackson.dev` does **not** match the two-label-deep `*.home.keithjackson.dev`. + Decide the challenge mechanism: **(a)** a scoped Cloudflare **API token** (Zone·DNS·Edit) → fully + automated DNS-01 + ExternalDNS, or **(b)** **acme-dns delegation** (one-time `_acme-challenge` + CNAME edit, then no standing Cloudflare credential). See [`08`](08-tls-and-certificates.md). diff --git a/docs/01-architecture-overview.md b/docs/01-architecture-overview.md index e5a19e5..9842163 100644 --- a/docs/01-architecture-overview.md +++ b/docs/01-architecture-overview.md @@ -15,7 +15,7 @@ choice here without a superseding ADR.** | VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) | | TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) | | Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) | -| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) | +| VPN egress | **Single shared in-cluster ExpressVPN gateway** (OpenVPN UDP) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0012](../decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md) (supersedes 0008) | [03](03-networking-isolation-and-egress.md) | | Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) | | DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) | | Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) | diff --git a/docs/03-networking-isolation-and-egress.md b/docs/03-networking-isolation-and-egress.md index efc0859..24f931c 100644 --- a/docs/03-networking-isolation-and-egress.md +++ b/docs/03-networking-isolation-and-egress.md @@ -60,46 +60,51 @@ seerr → plex tcp/32400 (Seerr talks to Plex) Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML. -## Multi-WAN & egress routing (R-NET-04/05) +## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1) -The platform recognizes **egress profiles**. Initial profiles: +The environment has **one physical commercial internet WAN** and **no plans for additional ISP +uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not +as multi-ISP routing but as **two egress paths selected by routing rules**: | Profile | Meaning | Mechanism | |---|---|---| -| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router | -| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) | +| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet | +| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) | -**Where multi-WAN policy routing lives (OQ-1):** -- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an - **edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster. - Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across - available WANs. -- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium - Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides - *physical WAN* for the internet class. -- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware) - is answered. The **service-facing contract does not change**: a service still just declares - `egress: internet|vpn`. +- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy- + route across; the `internet` path is simply the default route out. +- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore + **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on + the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service + ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)). +- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`. -## VPN egress model (R-NET-05/06 — ADR-0008) +## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008) -VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves -via the VPN; if the tunnel drops, they lose connectivity rather than leaking. +VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single, +shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside +the service**: -Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract): +- **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN + tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is + viable)** — the cluster provisions and controls the tunnel (nothing on the edge router). +- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have + **no independent egress path**. The kill-switch and allow rules are enforced at the **gateway + + namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn` + and cannot leak on its own. +- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn` + pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class + services are unaffected. +- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain + for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state + ([10](10-observability-and-alerting.md)). +- **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain + an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a + SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end + (tunnel + kill-switch) in dev/test. -- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN - tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's - **firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically. - Simple, self-contained, per-service isolation; more sidecars to run. -- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway - deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it. - Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying - fallback to the internet path. - -**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly -down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not -assumed. +**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail +closed (no packets to the plain internet). Tested, not assumed. ## Ingress (north-south) — summary @@ -120,7 +125,13 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward - Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected. - Changing a service's `egress` class in git flips its path with no workload rearchitecting. +## Resolved +- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All + segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope. +- **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced + outside the service (ADR-0012). + ## Open questions -- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split. -- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct - egress IPs (affects Pattern A vs B and HA of the gateway). +- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) — + WS-NET implementation detail; the kill-switch acceptance test is the gate. +- Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism). diff --git a/docs/04-storage-and-state.md b/docs/04-storage-and-state.md index 1d5c55a..3d21259 100644 --- a/docs/04-storage-and-state.md +++ b/docs/04-storage-and-state.md @@ -39,18 +39,19 @@ service comes back with its data. This is verified per service, not assumed. ## Bulk media strategy (R-STATE-04 — the Plex problem) Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB -media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a -**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2: +media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk +media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not +block the core platform. -- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps - bulk data off the replicated tier; simplest capacity story; introduces an external dependency. -- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but - node-bound; requires its own backup plan. -- **Object storage** for some content — generally poor fit for Plex direct play. +- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be + specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete + backing can be filled in later without touching service definitions. +- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack. -Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica -Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads -(qbittorrent) stage to the bulk tier and hand off to the arr stack. +**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica +Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk +media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise +when the backing is chosen. ## Backup & disaster recovery (R-STATE-03) @@ -86,6 +87,12 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads - Restore a namespace + volumes from off-cluster backup into a clean state (DR drill). - Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier. +## Resolved +- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an + external NFS/SMB mount via the `bulk` indirection). Does not block the core platform. + ## Open questions -- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture. -- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience. +- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS + host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs), + and its own resilience. Needed by **Phase 4**, not Phase 2. +- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen). diff --git a/docs/08-tls-and-certificates.md b/docs/08-tls-and-certificates.md index cbc38cb..f0fb252 100644 --- a/docs/08-tls-and-certificates.md +++ b/docs/08-tls-and-certificates.md @@ -28,6 +28,25 @@ services (R-TLS-01/02/03). needs no new cert), with per-service certs available where isolation warrants. - The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)). +## DNS-01 challenge access (OQ-4 — the one open item) + +Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record. **That covers +reachability but not certificate issuance**: ACME DNS-01 requires cert-manager to **write +`_acme-challenge` TXT records**, and the existing wildcard does not match two-label-deep names like +`plex.home.keithjackson.dev`. So a **challenge-access decision is still required**: + +- **Option A — scoped Cloudflare API token** (Zone·DNS·Edit for `keithjackson.dev`), stored + SOPS-encrypted ([07](07-security-and-secrets.md)). cert-manager writes TXT challenges and + ExternalDNS manages the `*.home.keithjackson.dev` records — **fully automated**. *Recommended + (simplest).* +- **Option B — acme-dns delegation**: a one-time `_acme-challenge` **CNAME** in Cloudflare points to + an in-cluster acme-dns server; cert-manager writes only to acme-dns, so **no standing Cloudflare + credential** lives in the cluster. More moving parts; public `*.home` records then need the + one-time wildcard record instead of ExternalDNS. + +Either option supports the wildcard `*.home.keithjackson.dev` cert and internal-only issuance +(R-TLS-03). This is the remaining blocker for the TLS/DNS workstream (Phase 2). + ## Renewal & revocation (R-TLS-02) - cert-manager renews well before expiry automatically; no human action. @@ -57,5 +76,7 @@ CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves - Onboarding a new service requires no manual certificate step. ## Open questions -- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01. -- Wildcard vs. per-service certs as the default (pinned in ADR-0006). +- **OQ-4** — DNS-01 challenge access: **Option A (scoped Cloudflare API token, recommended)** vs + **Option B (acme-dns delegation)**. Provider is Cloudflare (confirmed). This is the only item still + blocking Phase 2 TLS. +- Wildcard vs. per-service certs default — **resolved: wildcard** `*.home.keithjackson.dev` (ADR-0006). diff --git a/docs/09-dns-and-ingress.md b/docs/09-dns-and-ingress.md index e6a1b36..040b215 100644 --- a/docs/09-dns-and-ingress.md +++ b/docs/09-dns-and-ingress.md @@ -44,13 +44,27 @@ Internal and external answers differ: - `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the Gateway. -## Edge integration +## Edge integration (single SOHO router — OQ-1 resolved) -- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address - and applies multi-WAN inbound rules as needed (OQ-1). +- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only + ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400, + Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step + (the SOHO device has no automation) — the analogue of the old pfSense manual-config exception. + Confirm the router can port-forward (basic NAT; almost all can). +- No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)). - `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN. -- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium - L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up. +- **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not + available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice. + +## DNS record depth (note for `*.home.keithjackson.dev`) + +A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `home.keithjackson.dev` +but **not** `plex.home.keithjackson.dev`. Consequences: +- **`internal` services need no public record** — PiHole resolves `.home.keithjackson.dev` to + the internal Gateway VIP (split-horizon). This is the majority of services. +- **`public` services** need a public record for `.home.keithjackson.dev`. That is provided by + either a one-time `*.home.keithjackson.dev` → public-IP record, or by **ExternalDNS via the DNS + API** — both hinge on OQ-4 (see [08](08-tls-and-certificates.md)). ## Interaction with default-deny @@ -65,7 +79,11 @@ not let other in-cluster services reach it unless an explicit allow rule exists. - Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert. - The root `home.keithjackson.dev` serves the C2 UI. +## Resolved +- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**. + ## Open questions -- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and - inbound WAN rules). -- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up. +- **OQ-4** — DNS challenge/record access: Cloudflare **API token** (automated ExternalDNS + DNS-01) + vs **acme-dns delegation** (one-time CNAME, no standing credential). Determines whether public + `*.home.keithjackson.dev` records and ACME challenges are automated or partly manual. See + [08](08-tls-and-certificates.md).