Resolve OQ-1/2/3, refine OQ-4: single ExpressVPN gateway, single WAN, external media

Operator answers folded into the specs:
- OQ-3 VPN: ExpressVPN, cluster-controlled SINGLE shared in-cluster tunnel
  (OpenVPN UDP / non-TCP), rules enforced outside the service. New ADR-0012
  supersedes ADR-0008 (Gluetun-sidecar default dropped). Central kill-switch.
- OQ-1 WAN/edge: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
  VLAN/VPN/services). Multi-WAN out of scope; segmentation + VPN in-cluster;
  edge = NAT + manual port-forward. Gateway VIP resolved to Cilium L2
  announcement. T-3.3 marked won't-do.
- OQ-2 storage: bulk media lives outside the cluster (backing TBD, non-blocking)
  via a 'bulk' external-mount indirection; Plex metadata DB stays on Longhorn.
  Split out a separate OQ-BKP for the off-cluster DR backup target (Phase 4).
- OQ-4 refined: wildcard record covers reachability but NOT ACME DNS-01 (needs
  TXT-write access) and does not match *.home.* depth. Still OPEN: Cloudflare
  API token (Option A) vs acme-dns delegation (Option B). T-2.4/T-2.5 blocked.

Updated: docs 00,01,03,04,08,09; ADR-0008/0012 + index; task board.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-15 23:00:11 -05:00
1 parent 1a67ab5cfa
commit a2cbdf656d
10 files changed
+222 -87

No files matched your search

+28 -13
View File
@@ -74,10 +74,15 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
destination → port/protocol.
- **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced
boundaries).
- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing
rules.
- **R-NET-04** — The platform MUST support **multiple egress paths** selected by routing rules. Per
OQ-1 the environment has **one physical commercial WAN and no multi-ISP plans**, so this is
realized as two paths — the local `internet` path and the in-cluster `vpn` path — not multi-ISP
routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens
it.
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
network/internet) or `vpn` (routed through a custom VPN connection).
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced
**outside the service** (a single shared in-cluster gateway), so a service cannot leak by
misconfiguration (ADR-0012, OQ-3).
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
service MUST NOT leak traffic to the standard internet.
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
@@ -176,14 +181,24 @@ demonstration. The testing approach is specified in
implement against the spec and prove conformance with tests; they never edit the spec to fit the
implementation.
## Open questions (tracked, not blocking the architecture)
## Open questions
- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do
the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md).
- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk
vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003.
- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many
concurrent tunnels/egress IPs are available.
- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an
API token scoped for ACME DNS-01 and ExternalDNS.
### Resolved (2026-08-15)
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
- **OQ-3 (VPN)** — **RESOLVED**: **ExpressVPN**, cluster-provisioned, **single shared in-cluster
tunnel**, **non-TCP** (OpenVPN UDP / Lightway-UDP), rules enforced outside the service
(ADR-0012). See [`03`](03-networking-isolation-and-egress.md).
### Still open
- **OQ-4 (TLS/DNS challenge access)**: Cloudflare already has a wildcard `*.keithjackson.dev` → public
IP (covers reachability), but **ACME DNS-01 needs write access** to create `_acme-challenge` TXT
records, and `*.keithjackson.dev` does **not** match the two-label-deep `*.home.keithjackson.dev`.
Decide the challenge mechanism: **(a)** a scoped Cloudflare **API token** (Zone·DNS·Edit) → fully
automated DNS-01 + ExternalDNS, or **(b)** **acme-dns delegation** (one-time `_acme-challenge`
CNAME edit, then no standing Cloudflare credential). See [`08`](08-tls-and-certificates.md).
+1 -1
View File
@@ -15,7 +15,7 @@ choice here without a superseding ADR.**
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) |
| VPN egress | **Single shared in-cluster ExpressVPN gateway** (OpenVPN UDP) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0012](../decisions/adr-0012-vpn-egress-single-expressvpn-gateway.md) (supersedes 0008) | [03](03-networking-isolation-and-egress.md) |
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
+45 -34
View File
@@ -60,46 +60,51 @@ seerr → plex tcp/32400 (Seerr talks to Plex)
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
## Multi-WAN & egress routing (R-NET-04/05)
## Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)
The platform recognizes **egress profiles**. Initial profiles:
The environment has **one physical commercial internet WAN** and **no plans for additional ISP
uplinks or physical WANs** (OQ-1). "Multiple WANs" from the original intent is therefore realized not
as multi-ISP routing but as **two egress paths selected by routing rules**:
| Profile | Meaning | Mechanism |
|---|---|---|
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0012) |
**Where multi-WAN policy routing lives (OQ-1):**
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
available WANs.
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
*physical WAN* for the internet class.
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
is answered. The **service-facing contract does not change**: a service still just declares
`egress: internet|vpn`.
- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
route across; the `internet` path is simply the default route out.
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
**all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
## VPN egress model (R-NET-05/06 — ADR-0008)
## VPN egress model: single shared ExpressVPN gateway (R-NET-05/06 — ADR-0012, supersedes ADR-0008)
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**:
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
- **One in-cluster VPN egress gateway** (an HA-capable Deployment) holds a **single ExpressVPN
tunnel** using a **non-TCP protocol (OpenVPN UDP; Lightway-UDP if a containerizable client is
viable)** — the cluster provisions and controls the tunnel (nothing on the edge router).
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
**no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
and cannot leak on its own.
- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
services are unaffected.
- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
([10](10-observability-and-alerting.md)).
- **Prerequisite:** ExpressVPN is not a first-class containerized provider — the operator must obtain
an **ExpressVPN manual OpenVPN-UDP config + manual credentials**; the gateway consumes them as a
SOPS-encrypted secret ([07](07-security-and-secrets.md)). This must be validated end-to-end
(tunnel + kill-switch) in dev/test.
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
Simple, self-contained, per-service isolation; more sidecars to run.
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
fallback to the internet path.
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
assumed.
**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
closed (no packets to the plain internet). Tested, not assumed.
## Ingress (north-south) — summary
@@ -120,7 +125,13 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
## Resolved
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
- **OQ-3** — ExpressVPN, single shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced
outside the service (ADR-0012).
## Open questions
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
egress IPs (affects Pattern A vs B and HA of the gateway).
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
WS-NET implementation detail; the kill-switch acceptance test is the gate.
- Whether a containerized **Lightway-UDP** client is viable (else OpenVPN UDP is the mechanism).
+19 -12
View File
@@ -39,18 +39,19 @@ service comes back with its data. This is verified per service, not assumed.
## Bulk media strategy (R-STATE-04 — the Plex problem)
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
block the core platform.
- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
node-bound; requires its own backup plan.
- **Object storage** for some content — generally poor fit for Plex direct play.
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
backing can be filled in later without touching service definitions.
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
(qbittorrent) stage to the bulk tier and hand off to the arr stack.
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
when the backing is chosen.
## Backup & disaster recovery (R-STATE-03)
@@ -86,6 +87,12 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
## Resolved
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
## Open questions
- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
and its own resilience. Needed by **Phase 4**, not Phase 2.
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
+23 -2
View File
@@ -28,6 +28,25 @@ services (R-TLS-01/02/03).
needs no new cert), with per-service certs available where isolation warrants.
- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)).
## DNS-01 challenge access (OQ-4 — the one open item)
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record. **That covers
reachability but not certificate issuance**: ACME DNS-01 requires cert-manager to **write
`_acme-challenge` TXT records**, and the existing wildcard does not match two-label-deep names like
`plex.home.keithjackson.dev`. So a **challenge-access decision is still required**:
- **Option A — scoped Cloudflare API token** (Zone·DNS·Edit for `keithjackson.dev`), stored
SOPS-encrypted ([07](07-security-and-secrets.md)). cert-manager writes TXT challenges and
ExternalDNS manages the `*.home.keithjackson.dev` records — **fully automated**. *Recommended
(simplest).*
- **Option B — acme-dns delegation**: a one-time `_acme-challenge` **CNAME** in Cloudflare points to
an in-cluster acme-dns server; cert-manager writes only to acme-dns, so **no standing Cloudflare
credential** lives in the cluster. More moving parts; public `*.home` records then need the
one-time wildcard record instead of ExternalDNS.
Either option supports the wildcard `*.home.keithjackson.dev` cert and internal-only issuance
(R-TLS-03). This is the remaining blocker for the TLS/DNS workstream (Phase 2).
## Renewal & revocation (R-TLS-02)
- cert-manager renews well before expiry automatically; no human action.
@@ -57,5 +76,7 @@ CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
- Onboarding a new service requires no manual certificate step.
## Open questions
- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01.
- Wildcard vs. per-service certs as the default (pinned in ADR-0006).
- **OQ-4** — DNS-01 challenge access: **Option A (scoped Cloudflare API token, recommended)** vs
**Option B (acme-dns delegation)**. Provider is Cloudflare (confirmed). This is the only item still
blocking Phase 2 TLS.
- Wildcard vs. per-service certs default — **resolved: wildcard** `*.home.keithjackson.dev` (ADR-0006).
+26 -8
View File
@@ -44,13 +44,27 @@ Internal and external answers differ:
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
Gateway.
## Edge integration
## Edge integration (single SOHO router — OQ-1 resolved)
- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address
and applies multi-WAN inbound rules as needed (OQ-1).
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only
ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400,
Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step
(the SOHO device has no automation) — the analogue of the old pfSense manual-config exception.
Confirm the router can port-forward (basic NAT; almost all can).
- No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)).
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium
L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up.
- **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not
available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice.
## DNS record depth (note for `*.home.keithjackson.dev`)
A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `home.keithjackson.dev`
but **not** `plex.home.keithjackson.dev`. Consequences:
- **`internal` services need no public record** — PiHole resolves `<svc>.home.keithjackson.dev` to
the internal Gateway VIP (split-horizon). This is the majority of services.
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`. That is provided by
either a one-time `*.home.keithjackson.dev` → public-IP record, or by **ExternalDNS via the DNS
API** — both hinge on OQ-4 (see [08](08-tls-and-certificates.md)).
## Interaction with default-deny
@@ -65,7 +79,11 @@ not let other in-cluster services reach it unless an explicit allow rule exists.
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
- The root `home.keithjackson.dev` serves the C2 UI.
## Resolved
- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**.
## Open questions
- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and
inbound WAN rules).
- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up.
- **OQ-4** — DNS challenge/record access: Cloudflare **API token** (automated ExternalDNS + DNS-01)
vs **acme-dns delegation** (one-time CNAME, no standing credential). Determines whether public
`*.home.keithjackson.dev` records and ACME challenges are automated or partly manual. See
[08](08-tls-and-certificates.md).