Merge feature/oq-answers-vpn-wan-storage-dns into main
Fold operator answers to OQ-1/2/3 into the specs; refine OQ-4 to the one remaining decision (ACME DNS-01 access). ADR-0012 supersedes ADR-0008. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
10 files changed
+222
-87
No files matched your search
+17
-15
@@ -36,29 +36,30 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst
|
||||
## Phase 2 — Platform Core (net/storage/TLS/DNS/ingress)
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-2.1 | Longhorn (3-replica StorageClass) + bulk-tier plan (OQ-2) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo |
|
||||
| T-2.1 | Longhorn (3-replica StorageClass) + `bulk` external-mount indirection (media lives off-cluster, backing TBD — OQ-2 resolved) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo |
|
||||
| T-2.2 | Default-deny network baseline (per-namespace) + Hubble | WS-NET | R-NET-01/03 | T-1.3 | — | todo |
|
||||
| T-2.3 | Render `dependencies` → CiliumNetworkPolicy from ServiceDefinition | WS-NET/WS-GITOPS | R-NET-02 | T-0.3, T-2.2 | — | todo |
|
||||
| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` | WS-SEC | R-TLS-01/02/03 | T-1.4 | — | todo |
|
||||
| T-2.5 | ExternalDNS + Gateway API listeners + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo |
|
||||
| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` (**blocked on OQ-4**: Cloudflare token vs acme-dns) | WS-SEC | R-TLS-01/02/03 | T-1.4, OQ-4 | — | blocked |
|
||||
| T-2.5 | ExternalDNS + Gateway API listeners (VIP via **Cilium L2 announcement**) + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo |
|
||||
| T-2.6 | Prove: default-deny (DNS-only probe), PVC survives rebuild, trusted cert, auto-DNS | WS-NET/STORAGE/SEC | R-NET/STATE/TLS/DNS | T-2.1..2.5 | — | todo |
|
||||
| T-2.7 | Deploy **Kyverno in-cluster as admission enforcer** (same policies as the CI L0 gate) | WS-TEST/WS-SEC | R-SEC-02, R-NET-01, R-SEC-01 | T-1.4, T-0.5 | — | todo |
|
||||
|
||||
## Phase 3 — Egress, VPN & Multi-WAN
|
||||
## Phase 3 — Egress & VPN (single WAN; no multi-WAN — OQ-1)
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-3.1 | Egress classes; Cilium Egress Gateway wiring | WS-NET | R-NET-04/05 | T-2.2 | — | todo |
|
||||
| T-3.2 | VPN egress (Gluetun sidecar) + kill-switch (ADR-0008) | WS-NET | R-NET-05/06 | T-3.1, OQ-3 | — | todo |
|
||||
| T-3.3 | Edge-router multi-WAN split for `internet` class (OQ-1) | WS-NET | R-NET-04 | OQ-1 | — | todo |
|
||||
| T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo |
|
||||
| T-3.4 | Prove: VPN IP egress, kill-switch fail-closed, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo |
|
||||
| T-3.1 | Egress classes; Cilium Egress Gateway wiring (steer `vpn` pods → gateway) | WS-NET | R-NET-04/05 | T-2.2 | — | todo |
|
||||
| T-3.2p | **Prereq:** obtain ExpressVPN manual OpenVPN-UDP config + manual creds; store SOPS-encrypted | WS-SEC | R-NET-05 | — | — | todo |
|
||||
| T-3.2 | **Single shared in-cluster ExpressVPN gateway** (OpenVPN UDP, active/standby) + centrally-enforced kill-switch (ADR-0012) | WS-NET | R-NET-05/06 | T-3.1, T-3.2p | — | todo |
|
||||
| ~~T-3.3~~ | ~~Edge-router multi-WAN split~~ — **out of scope** (single WAN, dumb SOHO edge; OQ-1) | WS-NET | R-NET-04 | — | — | won't-do |
|
||||
| T-3.4 | Prove: VPN IP egress, **kill-switch fail-closed (central)**, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo |
|
||||
|
||||
## Phase 4 — Observability, Backup/DR & Updates
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-4.1 | kube-prometheus-stack + Loki + Alloy + Alertmanager(email) | WS-OBS | R-OBS-01/02/03 | T-2.1 | — | todo |
|
||||
| T-4.2 | Platform-health alerts (node/Longhorn/Flux/cert/VPN/backup) | WS-OBS | R-OBS-04 | T-4.1 | — | todo |
|
||||
| T-4.3 | Velero + Longhorn off-cluster backup schedule | WS-STORAGE | R-STATE-03 | T-2.1, OQ-2 | — | todo |
|
||||
| T-4.3 | Velero + Longhorn off-cluster backup schedule (**needs backup-target choice** — S3/NFS endpoint) | WS-STORAGE | R-STATE-03 | T-2.1, OQ-BKP | — | todo |
|
||||
| T-4.4 | DR drill: restore namespace + volumes from off-cluster backup | WS-STORAGE | R-STATE-03 | T-4.3 | — | todo |
|
||||
| T-4.5 | Flux image-automation + Renovate + per-service update policy | WS-UPDATE | R-UPD-01/02/03 | T-0.3, T-4.1 | — | todo |
|
||||
| T-4.6 | Prove: auto stable update on schedule, bad-update auto-rollback+alert, pin holds | WS-UPDATE | R-UPD-* | T-4.5 | — | todo |
|
||||
@@ -93,10 +94,11 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst
|
||||
|
||||
---
|
||||
|
||||
## Open questions (resolve to unblock)
|
||||
| OQ | Question | Blocks | Owner |
|
||||
## Open questions
|
||||
| OQ | Question | Blocks | Status |
|
||||
|---|---|---|---|
|
||||
| OQ-1 | WAN uplink count + edge-router product | T-3.3 | — |
|
||||
| OQ-2 | Bulk-media home + backup posture | T-2.1, T-4.3 | — |
|
||||
| OQ-3 | VPN provider/protocol + concurrent-tunnel limits | T-3.2 | — |
|
||||
| OQ-4 | DNS provider API (Cloudflare) token for DNS-01 + ExternalDNS | T-2.4, T-2.5 | — |
|
||||
| OQ-1 | WAN count + edge router | — | ✅ resolved: 1 WAN, dumb SOHO edge; multi-WAN out of scope; VPN + segmentation in-cluster |
|
||||
| OQ-2 | Bulk-media home | — | ✅ resolved: media outside cluster, backing TBD later (not blocking) |
|
||||
| OQ-3 | VPN provider/protocol | — | ✅ resolved: ExpressVPN, single shared in-cluster tunnel, OpenVPN UDP (ADR-0012) |
|
||||
| **OQ-4** | **DNS-01 challenge access** | **T-2.4, T-2.5** | ⛔ **OPEN** — Cloudflare API token (Option A, recommended) vs acme-dns delegation (Option B) |
|
||||
| **OQ-BKP** | **Off-cluster DR backup target** (S3/NFS endpoint) | T-4.3 | ⛔ **OPEN** — needed by Phase 4, not Phase 2 |
|
||||
+2
-1
@@ -49,7 +49,8 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
|
||||
| [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 |
|
||||
| [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 |
|
||||
| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | Accepted | R-NET-05/06 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | ~~Superseded by 0012~~ | R-NET-05/06 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
| [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | Accepted | R-NET-05/06/07 |
|
||||
@@ -1,9 +1,13 @@
|
||||
# ADR-0008 — VPN egress model: Gluetun sidecar (default) + Cilium Egress Gateway
|
||||
- Status: Accepted
|
||||
- Status: **Superseded by [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md)** (2026-08-15)
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes: —
|
||||
|
||||
> **Superseded.** OQ-3 resolved to ExpressVPN + a **single shared in-cluster tunnel** with rules
|
||||
> enforced outside the service. The Gluetun-sidecar-per-pod default below is no longer the model; see
|
||||
> [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md). Retained for history.
|
||||
|
||||
## Context
|
||||
A subset of services (sonarr, radarr, jackett, qbittorrent) must egress **only** through a commercial
|
||||
VPN, with a **kill-switch** guaranteeing no leak to the plain internet if the tunnel drops. Egress
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# ADR-0012 — VPN egress: single shared in-cluster ExpressVPN gateway
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes: [ADR-0008](adr-0008-vpn-egress-model.md)
|
||||
|
||||
## Context
|
||||
ADR-0008 offered two VPN-egress patterns (Gluetun sidecar per pod as default, central gateway as
|
||||
alternative) and left the provider open (OQ-3). The operator has now specified (OQ-3):
|
||||
- Provider is **ExpressVPN**; the **cluster must provision and control** the connection (nothing on
|
||||
the edge router — the edge is a dumb SOHO device with no VPN, OQ-1).
|
||||
- Protocol **must not be TCP**; UDP or better is required, data must be protected.
|
||||
- A **single shared VPN connection** is preferred over per-service sidecars, **so network rules are
|
||||
enforced outside the service**.
|
||||
|
||||
This changes the decision, so ADR-0008 is superseded.
|
||||
|
||||
## Decision
|
||||
Adopt a **single, shared, in-cluster VPN egress gateway** as the mandated model (ADR-0008's
|
||||
"Pattern B" becomes the only pattern; the per-service Gluetun sidecar default is dropped):
|
||||
|
||||
- **One VPN egress gateway** (HA-capable Deployment, run **active/standby**) holds a **single
|
||||
ExpressVPN tunnel** using a **non-TCP protocol**: **OpenVPN UDP** (via ExpressVPN's manual
|
||||
OpenVPN-UDP config), or **Lightway-UDP** if a containerizable client proves viable. The cluster —
|
||||
not the edge — provisions and controls the tunnel.
|
||||
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing) and have
|
||||
**no independent egress path**. The **kill-switch (fail-closed)** and egress allow rules are
|
||||
enforced at the **gateway + namespace/policy level — outside each service**. A service is merely
|
||||
labeled `egress: vpn`; it cannot leak on its own even if misconfigured.
|
||||
- Credentials (ExpressVPN manual OpenVPN config + manual-config username/password) are stored
|
||||
**SOPS-encrypted** ([ADR-0007](adr-0007-secrets-sops-age.md)).
|
||||
- The `egress: internet | vpn` service contract is unchanged (R-NET-07).
|
||||
|
||||
## Alternatives considered
|
||||
- **ADR-0008 Pattern A (Gluetun sidecar per pod)** — rejected per OQ-3: puts VPN inside each service
|
||||
(rules enforced *in* the service), multiplies tunnels, and isn't the operator's preferred single-
|
||||
connection model.
|
||||
- **VPN on the edge router** — impossible: the SOHO edge has no VPN capability (OQ-1); the cluster
|
||||
must own the tunnel.
|
||||
- **WireGuard** — ExpressVPN does not offer WireGuard; OpenVPN UDP (or Lightway-UDP) is the non-TCP
|
||||
path.
|
||||
- **OpenVPN TCP** — rejected: operator requires non-TCP.
|
||||
|
||||
## Consequences
|
||||
- Rules enforced outside the service (operator's goal); one tunnel to manage; simpler per-service
|
||||
workloads (no sidecars).
|
||||
- **Single shared egress IP** for all `vpn` services and a **single failure domain** → mitigate with
|
||||
**active/standby** gateway and tunnel-state alerting ([10](../docs/10-observability-and-alerting.md)).
|
||||
- **ExpressVPN is not a first-class containerized provider**: expect a **custom OpenVPN-UDP** setup
|
||||
(e.g. gluetun custom-provider mode or a purpose-built OpenVPN client), and the operator must obtain
|
||||
the manual OpenVPN config + manual credentials. Tunnel + kill-switch **must be validated in
|
||||
dev/test** (fault-injection test, R-TEST-05).
|
||||
- Exact pod→gateway steering (Cilium Egress Gateway vs routing/SNAT) is a WS-NET implementation
|
||||
detail; the **kill-switch acceptance test is the gate** ([14](../docs/14-testing-and-conformance.md)).
|
||||
- If ExpressVPN containerization proves impractical, revisit provider choice via a superseding ADR
|
||||
(escalation to the operator — agents may not change this decision themselves).
|
||||
+28
-13
@@ -74,10 +74,15 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
destination → port/protocol.
|
||||
- **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced
|
||||
boundaries).
|
||||
- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing
|
||||
rules.
|
||||
- **R-NET-04** — The platform MUST support **multiple egress paths** selected by routing rules. Per
|
||||
OQ-1 the environment has **one physical commercial WAN and no multi-ISP plans**, so this is
|
||||
realized as two paths — the local `internet` path and the in-cluster `vpn` path — not multi-ISP
|
||||
routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens
|
||||
it.
|
||||
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
|
||||
network/internet) or `vpn` (routed through a custom VPN connection).
|
||||
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced
|
||||
**outside the service** (a single shared in-cluster gateway), so a service cannot leak by
|
||||
misconfiguration (ADR-0012, OQ-3).
|
||||
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
|
||||
service MUST NOT leak traffic to the standard internet.
|
||||
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
|
||||
@@ -176,14 +181,24 @@ demonstration. The testing approach is specified in
|
||||
implement against the spec and prove conformance with tests; they never edit the spec to fit the
|
||||
implementation.
|
||||
|
||||
## Open questions (tracked, not blocking the architecture)
|
||||
## Open questions
|
||||
|
||||
- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do
|
||||
the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md).
|
||||
- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk
|
||||
vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003.
|
||||
- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many
|
||||
concurrent tunnels/egress IPs are available.
|
||||
- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an
|
||||
API token scoped for ACME DNS-01 and ExternalDNS.
|
||||
### Resolved (2026-08-15)
|
||||
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
|
||||
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
|
||||
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
|
||||
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
|
||||
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
|
||||
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-3 (VPN)** — **RESOLVED**: **ExpressVPN**, cluster-provisioned, **single shared in-cluster
|
||||
tunnel**, **non-TCP** (OpenVPN UDP / Lightway-UDP), rules enforced outside the service
|
||||
(ADR-0012). See [`03`](03-networking-isolation-and-egress.md).
|
||||
|
||||
### Still open
|
||||
- **OQ-4 (TLS/DNS challenge access)**: Cloudflare already has a wildcard `*.keithjackson.dev` → public
|
||||
IP (covers reachability), but **ACME DNS-01 needs write access** to create `_acme-challenge` TXT
|
||||
records, and `*.keithjackson.dev` does **not** match the two-label-deep `*.home.keithjackson.dev`.
|
||||
Decide the challenge mechanism: **(a)** a scoped Cloudflare **API token** (Zone·DNS·Edit) → fully
|
||||
automated DNS-01 + ExternalDNS, or **(b)** **acme-dns delegation** (one-time `_acme-challenge`
|
||||
CNAME edit, then no standing Cloudflare credential). See [`08`](08-tls-and-certificates.md).
|
||||
@@ -15,7 +15,7 @@ choice here without a superseding ADR.**
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| VPN egress | **Single shared in-cluster ExpressVPN gateway** (OpenVPN UDP) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0012](../decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md) (supersedes 0008) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
|
||||
@@ -60,46 +60,51 @@ seerr → plex tcp/32400 (Seerr talks to Plex)
|
||||
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
|
||||
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
|
||||
|
||||
## Multi-WAN & egress routing (R-NET-04/05)
|
||||
## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)
|
||||
|
||||
The platform recognizes **egress profiles**. Initial profiles:
|
||||
The environment has **one physical commercial internet WAN** and **no plans for additional ISP
|
||||
uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
|
||||
as multi-ISP routing but as **two egress paths selected by routing rules**:
|
||||
|
||||
| Profile | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
|
||||
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
|
||||
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
|
||||
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) |
|
||||
|
||||
**Where multi-WAN policy routing lives (OQ-1):**
|
||||
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
|
||||
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
|
||||
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
|
||||
available WANs.
|
||||
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
|
||||
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
|
||||
*physical WAN* for the internet class.
|
||||
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
|
||||
is answered. The **service-facing contract does not change**: a service still just declares
|
||||
`egress: internet|vpn`.
|
||||
- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
|
||||
route across; the `internet` path is simply the default route out.
|
||||
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
|
||||
**all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
|
||||
the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
|
||||
ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
|
||||
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
|
||||
|
||||
## VPN egress model (R-NET-05/06 — ADR-0008)
|
||||
## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
|
||||
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
|
||||
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
|
||||
the service**:
|
||||
|
||||
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
|
||||
- **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN
|
||||
tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is
|
||||
viable)** — the cluster provisions and controls the tunnel (nothing on the edge router).
|
||||
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
|
||||
**no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
|
||||
namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
|
||||
and cannot leak on its own.
|
||||
- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
|
||||
pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
|
||||
services are unaffected.
|
||||
- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
|
||||
for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
|
||||
([10](10-observability-and-alerting.md)).
|
||||
- **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain
|
||||
an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a
|
||||
SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end
|
||||
(tunnel + kill-switch) in dev/test.
|
||||
|
||||
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
|
||||
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
|
||||
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
|
||||
Simple, self-contained, per-service isolation; more sidecars to run.
|
||||
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
|
||||
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
|
||||
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
|
||||
fallback to the internet path.
|
||||
|
||||
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
|
||||
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
|
||||
assumed.
|
||||
**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
|
||||
closed (no packets to the plain internet). Tested, not assumed.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
@@ -120,7 +125,13 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
|
||||
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
|
||||
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
|
||||
|
||||
## Resolved
|
||||
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
|
||||
segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
|
||||
- **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced
|
||||
outside the service (ADR-0012).
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
|
||||
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
|
||||
egress IPs (affects Pattern A vs B and HA of the gateway).
|
||||
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
|
||||
WS-NET implementation detail; the kill-switch acceptance test is the gate.
|
||||
- Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism).
|
||||
@@ -39,18 +39,19 @@ service comes back with its data. This is verified per service, not assumed.
|
||||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||||
|
||||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||||
media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
|
||||
**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
|
||||
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
|
||||
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
|
||||
block the core platform.
|
||||
|
||||
- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
|
||||
bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
|
||||
- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
|
||||
node-bound; requires its own backup plan.
|
||||
- **Object storage** for some content — generally poor fit for Plex direct play.
|
||||
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
|
||||
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
|
||||
backing can be filled in later without touching service definitions.
|
||||
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
|
||||
|
||||
Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
|
||||
(qbittorrent) stage to the bulk tier and hand off to the arr stack.
|
||||
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
|
||||
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
|
||||
when the backing is chosen.
|
||||
|
||||
## Backup & disaster recovery (R-STATE-03)
|
||||
|
||||
@@ -86,6 +87,12 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads
|
||||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||||
|
||||
## Resolved
|
||||
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
|
||||
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
|
||||
|
||||
## Open questions
|
||||
- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
|
||||
- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.
|
||||
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
|
||||
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
|
||||
and its own resilience. Needed by **Phase 4**, not Phase 2.
|
||||
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
|
||||
@@ -28,6 +28,25 @@ services (R-TLS-01/02/03).
|
||||
needs no new cert), with per-service certs available where isolation warrants.
|
||||
- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)).
|
||||
|
||||
## DNS-01 challenge access (OQ-4 — the one open item)
|
||||
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record. **That covers
|
||||
reachability but not certificate issuance**: ACME DNS-01 requires cert-manager to **write
|
||||
`_acme-challenge` TXT records**, and the existing wildcard does not match two-label-deep names like
|
||||
`plex.home.keithjackson.dev`. So a **challenge-access decision is still required**:
|
||||
|
||||
- **Option A — scoped Cloudflare API token** (Zone·DNS·Edit for `keithjackson.dev`), stored
|
||||
SOPS-encrypted ([07](07-security-and-secrets.md)). cert-manager writes TXT challenges and
|
||||
ExternalDNS manages the `*.home.keithjackson.dev` records — **fully automated**. *Recommended
|
||||
(simplest).*
|
||||
- **Option B — acme-dns delegation**: a one-time `_acme-challenge` **CNAME** in Cloudflare points to
|
||||
an in-cluster acme-dns server; cert-manager writes only to acme-dns, so **no standing Cloudflare
|
||||
credential** lives in the cluster. More moving parts; public `*.home` records then need the
|
||||
one-time wildcard record instead of ExternalDNS.
|
||||
|
||||
Either option supports the wildcard `*.home.keithjackson.dev` cert and internal-only issuance
|
||||
(R-TLS-03). This is the remaining blocker for the TLS/DNS workstream (Phase 2).
|
||||
|
||||
## Renewal & revocation (R-TLS-02)
|
||||
|
||||
- cert-manager renews well before expiry automatically; no human action.
|
||||
@@ -57,5 +76,7 @@ CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
|
||||
- Onboarding a new service requires no manual certificate step.
|
||||
|
||||
## Open questions
|
||||
- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01.
|
||||
- Wildcard vs. per-service certs as the default (pinned in ADR-0006).
|
||||
- **OQ-4** — DNS-01 challenge access: **Option A (scoped Cloudflare API token, recommended)** vs
|
||||
**Option B (acme-dns delegation)**. Provider is Cloudflare (confirmed). This is the only item still
|
||||
blocking Phase 2 TLS.
|
||||
- Wildcard vs. per-service certs default — **resolved: wildcard** `*.home.keithjackson.dev` (ADR-0006).
|
||||
@@ -44,13 +44,27 @@ Internal and external answers differ:
|
||||
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
|
||||
Gateway.
|
||||
|
||||
## Edge integration
|
||||
## Edge integration (single SOHO router — OQ-1 resolved)
|
||||
|
||||
- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address
|
||||
and applies multi-WAN inbound rules as needed (OQ-1).
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only
|
||||
ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400,
|
||||
Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step
|
||||
(the SOHO device has no automation) — the analogue of the old pfSense manual-config exception.
|
||||
Confirm the router can port-forward (basic NAT; almost all can).
|
||||
- No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)).
|
||||
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
|
||||
- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium
|
||||
L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up.
|
||||
- **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not
|
||||
available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice.
|
||||
|
||||
## DNS record depth (note for `*.home.keithjackson.dev`)
|
||||
|
||||
A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `home.keithjackson.dev`
|
||||
but **not** `plex.home.keithjackson.dev`. Consequences:
|
||||
- **`internal` services need no public record** — PiHole resolves `<svc>.home.keithjackson.dev` to
|
||||
the internal Gateway VIP (split-horizon). This is the majority of services.
|
||||
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`. That is provided by
|
||||
either a one-time `*.home.keithjackson.dev` → public-IP record, or by **ExternalDNS via the DNS
|
||||
API** — both hinge on OQ-4 (see [08](08-tls-and-certificates.md)).
|
||||
|
||||
## Interaction with default-deny
|
||||
|
||||
@@ -65,7 +79,11 @@ not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
## Resolved
|
||||
- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and
|
||||
inbound WAN rules).
|
||||
- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up.
|
||||
- **OQ-4** — DNS challenge/record access: Cloudflare **API token** (automated ExternalDNS + DNS-01)
|
||||
vs **acme-dns delegation** (one-time CNAME, no standing credential). Determines whether public
|
||||
`*.home.keithjackson.dev` records and ACME challenges are automated or partly manual. See
|
||||
[08](08-tls-and-certificates.md).
|
||||
Reference in new issue
Block a user