# 01 — Architecture Overview This document is the map. It names the chosen stack, shows how the layers fit, and points to the ADR that locks in each choice and the domain spec that details it. **Do not change a technology choice here without a superseding ADR.** ## The stack at a glance | Layer | Choice | Serves | ADR | Spec | |---|---|---|---|---| | Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | | CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) | | Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | | GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) | | VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) | | TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) | | Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) | | VPN egress | **Single shared in-cluster PIA/Gluetun gateway** (WireGuard + native PF) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md) (supersedes 0012/0008) | [03](03-networking-isolation-and-egress.md) | | Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) | | DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) | | Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) | | Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) | | Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | | Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | | Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) | | Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) | ## Layer cake ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Command & Control UI (home.keithjackson.dev root) │ │ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │ ├──────────────────────────────────────────────────────────────────────────┤ │ Workloads │ │ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │ │ IaaS: KubeVirt VMs (anything that can't be a container) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Platform services │ │ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │ │ kube-prometheus-stack + Loki + Alloy (observability) │ │ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │ │ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │ ├──────────────────────────────────────────────────────────────────────────┤ │ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Nodes │ │ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │ │ prod: 3× mini-PC bare metal, local disks │ ├──────────────────────────────────────────────────────────────────────────┤ │ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │ │ VPN uplink(s), split-horizon DNS │ └──────────────────────────────────────────────────────────────────────────┘ ``` ## Two clearly-separated virtualization layers (do not conflate) This platform has **two** kinds of VM, and confusing them is the most common design error: 1. **Infrastructure VMs = the cluster nodes themselves.** In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See [`02-environments-and-provisioning.md`](02-environments-and-provisioning.md). 2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.** These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 ("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md). > Layer 1 is how the platform exists. Layer 2 is a feature the platform offers. ## How a service flows through the platform (the golden path) 1. Operator adds/edits a **ServiceDefinition** in the git catalog (see [`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source, storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies (explicit allow rules), update policy, and observability expectations. 2. **Flux** reconciles the change into the cluster. 3. **Longhorn** provisions replicated persistent volumes; state is durable. 4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct **egress path** (local internet or VPN, with kill-switch). 5. **ExternalDNS** publishes `.home.keithjackson.dev`; **Gateway API** routes north-south traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01. 6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the **C2 UI** lists and links the service. 7. **Flux image automation / Renovate** keep the service on the latest stable version per policy — no human action. 8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state. Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md). ## Design principles - **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it. - **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model. - **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets, trusted TLS everywhere. - **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with tested restore. - **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated. - **Promotable.** dev/test and prod are the same config with scoped overrides. - **Provable.** Every requirement is verified by an automated, reproducible, behavior-based conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md)); "done" is a green, requirement-tagged suite, not a manual demo. - **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move the implementation and its tests toward the spec, never the spec toward the implementation.