home-network-k3s/docs/01-architecture-overview.md

01 — Architecture Overview

This document is the map. It names the chosen stack, shows how the layers fit, and points to the ADR that locks in each choice and the domain spec that details it. Do not change a technology choice here without a superseding ADR.

The stack at a glance

Layer	Choice	Serves	ADR	Spec
Kubernetes distro	k3s (HA, embedded etcd)	R-PLAT-01/02	ADR-0001	02
CNI / network policy / egress	Cilium (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM)	R-NET-01..07	ADR-0002	03
Distributed storage	Longhorn	R-STATE-01..02	ADR-0003	04
Bulk media	NFS from a standalone TrueNAS on the production LAN	R-STATE-04	ADR-0018	04
PostgreSQL	CloudNativePG (CNPG)	R-STATE-01/03, R-IDP-05	ADR-0019	04
GitOps engine	Flux CD	R-GITOPS-01..03	ADR-0004	05
VM workloads (IaaS)	KubeVirt (+ CDI)	R-PLAT-03	ADR-0005	06
TLS	cert-manager + Let's Encrypt (DNS-01)	R-TLS-01..03	ADR-0006	08
Secrets	SOPS + age (Flux-native)	R-SEC-01	ADR-0007	07
VPN egress	Node-level PIA/OpenVPN tunnel on a single gateway node, steered by Cilium Egress Gateway (Cilium-enforced fail-closed)	R-NET-05/06/07	ADR-0015, ADR-0020 (provider 0013)	03
Automated updates	Renovate (images and charts), every change PR-gated	R-UPD-01..03	ADR-0023	11
DNS records	ExternalDNS (public via Cloudflare) + PiHole (explicit LAN answers)	R-DNS-01..03	—	09
Ingress (north-south)	Gateway API implemented by Cilium	R-DNS-01, R-TLS-01	ADR-0002	09
Public web ingress	Cloudflare Tunnel (outbound-only cloudflared)	R-DNS-04/05, R-SEC-04	ADR-0016	09
Remote access	WireGuard on a hostNetwork DaemonSet	R-SEC-02/04	ADR-0021	03
Observability	kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager→SMTP	R-OBS-01..04	—	10
Backup/DR	Velero (kopia) + CNPG Barman → Cloudflare R2; no Longhorn backup target	R-STATE-03	ADR-0018	04
Node/infra provisioning	Imaged Debian on bare metal + Ansible (baseline, k3s bootstrap)	R-ENV-01..03	ADR-0001, ADR-0017	02
Control-plane endpoint	kube-vip virtual IP	R-PLAT-01	ADR-0017	02
Node patching & reboots	security-only unattended-upgrades + kured	R-PLAT-01	—	02
Identity / SSO	Keycloak (OIDC + SAML) + per-app oauth2-proxy forward-auth sidecars	R-IDP-01..06	ADR-0014, ADR-0022	15
Command center UI	Homepage + Headlamp + Grafana panels (behind Keycloak SSO)	R-C2-01/02	—	12
Conformance testing	Kyverno (L0 policy) + Chainsaw (L1/L2/L3 behavior + fault-injection)	R-TEST-01..06	ADR-0010, ADR-0011	14
Delivery & merge gate	Gitea + Gitea Actions; the CI gate is the approver	R-GITOPS-01, R-TEST-03	ADR-0024	05
Image builds	CI-run rootless BuildKit, published to Gitea's registry	—	—	05
AI Program Manager	pm-agent (issues to pull requests through the gate)	—	ADR-0025	05
Hardware transcode	Intel GPU device plugin (Plex Quick Sync)	—	—	13

Layer cake

┌──────────────────────────────────────────────────────────────────────────┐
│  Command & Control UI  (home.keithjackson.dev root)                        │
│  Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
├──────────────────────────────────────────────────────────────────────────┤
│  Workloads                                                                 │
│   PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …)              │
│   IaaS: KubeVirt VMs (anything that can't be a container)                  │
├──────────────────────────────────────────────────────────────────────────┤
│  Platform services                                                         │
│   Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API           │
│   Keycloak (SSO/IdP) + oauth2-proxy (forward-auth)                         │
│   kube-prometheus-stack + Loki + Alloy (observability)                     │
│   Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets)      │
│   cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots)  │
├──────────────────────────────────────────────────────────────────────────┤
│  Storage: Longhorn (replicated block)  +  TrueNAS NFS bulk tier (ADR-0018) │
├──────────────────────────────────────────────────────────────────────────┤
│  Network: Cilium  (default-deny NetworkPolicy · Egress Gateway · Hubble)   │
│           VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound  │
├──────────────────────────────────────────────────────────────────────────┤
│  Kubernetes: k3s (3 nodes, HA control plane, embedded etcd)                │
├──────────────────────────────────────────────────────────────────────────┤
│  Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible)       │
├──────────────────────────────────────────────────────────────────────────┤
│  Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex        │
│  Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
└──────────────────────────────────────────────────────────────────────────┘

Two clearly-separated virtualization layers (do not conflate)

This platform has two kinds of VM, and confusing them is the most common design error:

1. Infrastructure VMs = the cluster nodes themselves. These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is historical shorthand for this layer, and it remains the place a node would be virtualized. See 02-environments-and-provisioning.md.

2. Application VMs = IaaS workloads that run inside the cluster via KubeVirt. These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 ("not everything fits in a container"). See 06-vm-workloads-kubevirt.md.

Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.

How a service flows through the platform (the golden path)

1. Operator adds/edits a ServiceDefinition in the git catalog (see 13-service-catalog.md) declaring: kind (container|vm), image/source, storage, egress class (internet|vpn), exposure (internal|public), dependencies (explicit allow rules), update policy, and observability expectations.
2. Flux reconciles the change into the cluster.
3. Longhorn provisions replicated persistent volumes, so state is durable; large media attaches from the NFS bulk tier.
4. Cilium applies default-deny plus the explicitly declared allow rules; assigns the correct egress path (local internet or VPN, with kill-switch).
5. A public service gets a proxied ExternalDNS record at <service>.home.keithjackson.dev and arrives through the Cloudflare Tunnel. An internal service has no public record and is answered on the LAN by PiHole. Gateway API routes north-south traffic for both; cert-manager issues a trusted TLS cert via ACME DNS-01.
6. kube-prometheus-stack + Loki scrape metrics/logs; Alertmanager emails on issues; the C2 UI lists and links the service.
7. Renovate keeps the service on the latest stable version per policy, through machine-authored changes that pass the merge gate.
8. Velero and CNPG Barman, writing off-cluster, protect the data; destroy/rebuild restores state.

Every numbered step maps to a domain spec and a workstream in roadmap/workstreams.md.

Design principles

• Declarative everything. If it can't be expressed as YAML reconciled from git, question it.
• One control plane. Containers and VMs share the Kubernetes surface; operators learn one model.
• Secure by default. Deny-by-default network, least-privilege identity, encrypted secrets, trusted TLS everywhere.
• Durable by default. No workload is "pet" hardware; state lives on replicated storage with tested restore.
• Hands-off operations. Updates, cert renewal, DNS, and drift correction are automated.
• Promotable. Configuration is one base plus an environment-scoped overlay.
• Provable. Every requirement is verified by an automated, reproducible, behavior-based conformance test (14, ADR-0010); "done" is a green, requirement-tagged suite, not a manual demo.
• Spec-fixed. The specification is human-owned and immutable to agents (R-TEST-06); agents move the implementation and its tests toward the spec, never the spec toward the implementation.