jakman85andClaude Opus 4.8
a2cbdf656d
Operator answers folded into the specs: - OQ-3 VPN: ExpressVPN, cluster-controlled SINGLE shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced outside the service. New ADR-0012 supersedes ADR-0008 (Gluetun-sidecar default dropped). Central kill-switch. - OQ-1 WAN/edge: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). Multi-WAN out of scope; segmentation + VPN in-cluster; edge = NAT + manual port-forward. Gateway VIP resolved to Cilium L2 announcement. T-3.3 marked won't-do. - OQ-2 storage: bulk media lives outside the cluster (backing TBD, non-blocking) via a 'bulk' external-mount indirection; Plex metadata DB stays on Longhorn. Split out a separate OQ-BKP for the off-cluster DR backup target (Phase 4). - OQ-4 refined: wildcard record covers reachability but NOT ACME DNS-01 (needs TXT-write access) and does not match *.home.* depth. Still OPEN: Cloudflare API token (Option A) vs acme-dns delegation (Option B). T-2.4/T-2.5 blocked. Updated: docs 00,01,03,04,08,09; ADR-0008/0012 + index; task board. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
9.8 KiB
9.8 KiB
01 — Architecture Overview
This document is the map. It names the chosen stack, shows how the layers fit, and points to the ADR that locks in each choice and the domain spec that details it. Do not change a technology choice here without a superseding ADR.
The stack at a glance
| Layer | Choice | Serves | ADR | Spec |
|---|---|---|---|---|
| Kubernetes distro | k3s (HA, embedded etcd) | R-PLAT-01/02 | ADR-0001 | 02 |
| CNI / network policy / egress | Cilium (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | ADR-0002 | 03 |
| Distributed storage | Longhorn | R-STATE-01..03 | ADR-0003 | 04 |
| GitOps engine | Flux CD | R-GITOPS-01..03 | ADR-0004 | 05 |
| VM workloads (IaaS) | KubeVirt (+ CDI) | R-PLAT-03 | ADR-0005 | 06 |
| TLS | cert-manager + Let's Encrypt (DNS-01) | R-TLS-01..03 | ADR-0006 | 08 |
| Secrets | SOPS + age (Flux-native) | R-SEC-01 | ADR-0007 | 07 |
| VPN egress | Single shared in-cluster ExpressVPN gateway (OpenVPN UDP) + Cilium Egress Gateway | R-NET-05/06/07 | ADR-0012 (supersedes 0008) | 03 |
| Automated updates | Flux image-update-automation (images) + Renovate (charts) | R-UPD-01..03 | ADR-0009 | 11 |
| DNS records | ExternalDNS (public via Cloudflare) + PiHole (internal) | R-DNS-01..03 | — | 09 |
| Ingress (north-south) | Gateway API implemented by Cilium | R-DNS-01, R-TLS-01 | ADR-0002 | 09 |
| Observability | kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager→SMTP | R-OBS-01..04 | — | 10 |
| Backup/DR | Velero + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | ADR-0003 | 04 |
| Node/infra provisioning | OpenTofu (libvirt) + cloud-init + Ansible (k3s bootstrap) | R-ENV-01..03 | ADR-0001 | 02 |
| Command center UI | Homepage + Headlamp + Grafana panels | R-C2-01/02 | — | 12 |
| Conformance testing | Kyverno (L0 policy) + Chainsaw (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | ADR-0010, ADR-0011 | 14 |
Layer cake
┌──────────────────────────────────────────────────────────────────────────┐
│ Command & Control UI (home.keithjackson.dev root) │
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
├──────────────────────────────────────────────────────────────────────────┤
│ Workloads │
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
│ IaaS: KubeVirt VMs (anything that can't be a container) │
├──────────────────────────────────────────────────────────────────────────┤
│ Platform services │
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
│ kube-prometheus-stack + Loki + Alloy (observability) │
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
├──────────────────────────────────────────────────────────────────────────┤
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
├──────────────────────────────────────────────────────────────────────────┤
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
├──────────────────────────────────────────────────────────────────────────┤
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
├──────────────────────────────────────────────────────────────────────────┤
│ Nodes │
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
│ prod: 3× mini-PC bare metal, local disks │
├──────────────────────────────────────────────────────────────────────────┤
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
│ VPN uplink(s), split-horizon DNS │
└──────────────────────────────────────────────────────────────────────────┘
Two clearly-separated virtualization layers (do not conflate)
This platform has two kinds of VM, and confusing them is the most common design error:
-
Infrastructure VMs = the cluster nodes themselves. In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
02-environments-and-provisioning.md. -
Application VMs = IaaS workloads that run inside the cluster via KubeVirt. These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 ("not everything fits in a container"). See
06-vm-workloads-kubevirt.md.
Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
How a service flows through the platform (the golden path)
- Operator adds/edits a ServiceDefinition in the git catalog (see
13-service-catalog.md) declaring: kind (container|vm), image/source, storage, egress class (internet|vpn), exposure (internal|public), dependencies (explicit allow rules), update policy, and observability expectations. - Flux reconciles the change into the cluster.
- Longhorn provisions replicated persistent volumes; state is durable.
- Cilium applies default-deny plus the explicitly declared allow rules; assigns the correct egress path (local internet or VPN, with kill-switch).
- ExternalDNS publishes
<service>.home.keithjackson.dev; Gateway API routes north-south traffic; cert-manager issues a trusted TLS cert via ACME DNS-01. - kube-prometheus-stack + Loki scrape metrics/logs; Alertmanager emails on issues; the C2 UI lists and links the service.
- Flux image automation / Renovate keep the service on the latest stable version per policy — no human action.
- Velero + Longhorn backups protect the data; destroy/rebuild restores state.
Every numbered step maps to a domain spec and a workstream in roadmap/workstreams.md.
Design principles
- Declarative everything. If it can't be expressed as YAML reconciled from git, question it.
- One control plane. Containers and VMs share the Kubernetes surface; operators learn one model.
- Secure by default. Deny-by-default network, least-privilege identity, encrypted secrets, trusted TLS everywhere.
- Durable by default. No workload is "pet" hardware; state lives on replicated storage with tested restore.
- Hands-off operations. Updates, cert renewal, DNS, and drift correction are automated.
- Promotable. dev/test and prod are the same config with scoped overrides.
- Provable. Every requirement is verified by an automated, reproducible, behavior-based conformance test (14, ADR-0010); "done" is a green, requirement-tagged suite, not a manual demo.
- Spec-fixed. The specification is human-owned and immutable to agents (R-TEST-06); agents move the implementation and its tests toward the spec, never the spec toward the implementation.