jakman85andClaude Opus 4.8
95b4b6af21
Gluetun does not support Private Internet Access over WireGuard (upstream: native support "cannot be added... slow work in progress"; confirmed live rejecting the provider on v3.40/v3.41/latest). PIA in Gluetun is OpenVPN-only. Amend ADR-0013 to PIA over OpenVPN-UDP. The egress ARCHITECTURE is unchanged (single shared gateway, central kill-switch/fail-closed, native PF, Cilium egress steering, active/standby) and R-NET-05/06/07 remain met; only the tunnel protocol changes. OpenVPN-UDP satisfies "non-TCP", and Gluetun's HTTP control API (health + forwarded-port lookup) is what the shared-gateway topology needs. thrnz/docker-wireguard-pia retained as a documented WireGuard fallback. Propagated the protocol correction to all active specs: ADR-0013 (+amendment note), decisions/README, docs/00, docs/01, docs/03, docs/07, agents/task-board. Superseded ADR-0008/0012 left immutable as historical record. Co-Authored-By: Claude Opus 4.8 <[email protected]>
10 KiB
10 KiB
01 — Architecture Overview
This document is the map. It names the chosen stack, shows how the layers fit, and points to the ADR that locks in each choice and the domain spec that details it. Do not change a technology choice here without a superseding ADR.
The stack at a glance
| Layer | Choice | Serves | ADR | Spec |
|---|---|---|---|---|
| Kubernetes distro | k3s (HA, embedded etcd) | R-PLAT-01/02 | ADR-0001 | 02 |
| CNI / network policy / egress | Cilium (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | ADR-0002 | 03 |
| Distributed storage | Longhorn | R-STATE-01..03 | ADR-0003 | 04 |
| GitOps engine | Flux CD | R-GITOPS-01..03 | ADR-0004 | 05 |
| VM workloads (IaaS) | KubeVirt (+ CDI) | R-PLAT-03 | ADR-0005 | 06 |
| TLS | cert-manager + Let's Encrypt (DNS-01) | R-TLS-01..03 | ADR-0006 | 08 |
| Secrets | SOPS + age (Flux-native) | R-SEC-01 | ADR-0007 | 07 |
| VPN egress | Single shared in-cluster PIA/Gluetun gateway (OpenVPN-UDP + native PF) + Cilium Egress Gateway | R-NET-05/06/07 | ADR-0013 (supersedes 0012/0008) | 03 |
| Automated updates | Flux image-update-automation (images) + Renovate (charts) | R-UPD-01..03 | ADR-0009 | 11 |
| DNS records | ExternalDNS (public via Cloudflare) + PiHole (internal) | R-DNS-01..03 | — | 09 |
| Ingress (north-south) | Gateway API implemented by Cilium | R-DNS-01, R-TLS-01 | ADR-0002 | 09 |
| Observability | kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager→SMTP | R-OBS-01..04 | — | 10 |
| Backup/DR | Velero + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | ADR-0003 | 04 |
| Node/infra provisioning | OpenTofu (libvirt) + cloud-init + Ansible (k3s bootstrap) | R-ENV-01..03 | ADR-0001 | 02 |
| Identity / SSO | Keycloak (OIDC + SAML) + oauth2-proxy forward-auth | R-IDP-01..06 | ADR-0014 | 15 |
| Command center UI | Homepage + Headlamp + Grafana panels (behind Keycloak SSO) | R-C2-01/02 | — | 12 |
| Conformance testing | Kyverno (L0 policy) + Chainsaw (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | ADR-0010, ADR-0011 | 14 |
Layer cake
┌──────────────────────────────────────────────────────────────────────────┐
│ Command & Control UI (home.keithjackson.dev root) │
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
├──────────────────────────────────────────────────────────────────────────┤
│ Workloads │
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
│ IaaS: KubeVirt VMs (anything that can't be a container) │
├──────────────────────────────────────────────────────────────────────────┤
│ Platform services │
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
│ kube-prometheus-stack + Loki + Alloy (observability) │
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
├──────────────────────────────────────────────────────────────────────────┤
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
├──────────────────────────────────────────────────────────────────────────┤
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
├──────────────────────────────────────────────────────────────────────────┤
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
├──────────────────────────────────────────────────────────────────────────┤
│ Nodes │
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
│ prod: 3× mini-PC bare metal, local disks │
├──────────────────────────────────────────────────────────────────────────┤
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
│ VPN uplink(s), split-horizon DNS │
└──────────────────────────────────────────────────────────────────────────┘
Two clearly-separated virtualization layers (do not conflate)
This platform has two kinds of VM, and confusing them is the most common design error:
-
Infrastructure VMs = the cluster nodes themselves. In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
02-environments-and-provisioning.md. -
Application VMs = IaaS workloads that run inside the cluster via KubeVirt. These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 ("not everything fits in a container"). See
06-vm-workloads-kubevirt.md.
Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
How a service flows through the platform (the golden path)
- Operator adds/edits a ServiceDefinition in the git catalog (see
13-service-catalog.md) declaring: kind (container|vm), image/source, storage, egress class (internet|vpn), exposure (internal|public), dependencies (explicit allow rules), update policy, and observability expectations. - Flux reconciles the change into the cluster.
- Longhorn provisions replicated persistent volumes; state is durable.
- Cilium applies default-deny plus the explicitly declared allow rules; assigns the correct egress path (local internet or VPN, with kill-switch).
- ExternalDNS publishes
<service>.home.keithjackson.dev; Gateway API routes north-south traffic; cert-manager issues a trusted TLS cert via ACME DNS-01. - kube-prometheus-stack + Loki scrape metrics/logs; Alertmanager emails on issues; the C2 UI lists and links the service.
- Flux image automation / Renovate keep the service on the latest stable version per policy — no human action.
- Velero + Longhorn backups protect the data; destroy/rebuild restores state.
Every numbered step maps to a domain spec and a workstream in roadmap/workstreams.md.
Design principles
- Declarative everything. If it can't be expressed as YAML reconciled from git, question it.
- One control plane. Containers and VMs share the Kubernetes surface; operators learn one model.
- Secure by default. Deny-by-default network, least-privilege identity, encrypted secrets, trusted TLS everywhere.
- Durable by default. No workload is "pet" hardware; state lives on replicated storage with tested restore.
- Hands-off operations. Updates, cert renewal, DNS, and drift correction are automated.
- Promotable. dev/test and prod are the same config with scoped overrides.
- Provable. Every requirement is verified by an automated, reproducible, behavior-based conformance test (14, ADR-0010); "done" is a green, requirement-tagged suite, not a manual demo.
- Spec-fixed. The specification is human-owned and immutable to agents (R-TEST-06); agents move the implementation and its tests toward the spec, never the spec toward the implementation.