# 01 — Architecture Overview

This document is the map. It names the chosen stack, shows how the layers fit, and points to the
ADR that locks in each choice and the domain spec that details it. **Do not change a technology
choice here without a superseding ADR.**

## The stack at a glance

| Layer | Choice | Serves | ADR | Spec |
|---|---|---|---|---|
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
| Distributed storage | **Longhorn** | R-STATE-01..02 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
| Bulk media | **NFS** from a standalone **TrueNAS** on the production LAN | R-STATE-04 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
| PostgreSQL | **CloudNativePG (CNPG)** | R-STATE-01/03, R-IDP-05 | [ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md) | [04](04-storage-and-state.md) |
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
| VPN egress | **Node-level PIA/OpenVPN tunnel** on a single gateway node, steered by **Cilium Egress Gateway** (Cilium-enforced fail-closed) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), [ADR-0020](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
| Automated updates | **Renovate** (images and charts), every change PR-gated | R-UPD-01..03 | [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md) | [11](11-automated-updates.md) |
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (explicit LAN answers) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
| Public web ingress | **Cloudflare Tunnel** (outbound-only cloudflared) | R-DNS-04/05, R-SEC-04 | [ADR-0016](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md) | [09](09-dns-and-ingress.md) |
| Remote access | **WireGuard** on a hostNetwork DaemonSet | R-SEC-02/04 | [ADR-0021](../decisions/adr-0021-inbound-remote-access-wireguard.md) | [03](03-networking-isolation-and-egress.md) |
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
| Backup/DR | **Velero** (kopia) + **CNPG Barman** → Cloudflare R2; no Longhorn backup target | R-STATE-03 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
| Node/infra provisioning | **Imaged Debian** on bare metal + **Ansible** (baseline, k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
| Control-plane endpoint | **kube-vip** virtual IP | R-PLAT-01 | [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
| Node patching & reboots | security-only **unattended-upgrades** + **kured** | R-PLAT-01 | — | [02](02-environments-and-provisioning.md) |
| Identity / SSO | **Keycloak** (OIDC + SAML) + per-app **oauth2-proxy** forward-auth sidecars | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md), [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | [15](15-identity-and-access.md) |
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
| Delivery & merge gate | **Gitea** + **Gitea Actions**; the CI gate is the approver | R-GITOPS-01, R-TEST-03 | [ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md) | [05](05-gitops-and-delivery.md) |
| Image builds | CI-run rootless **BuildKit**, published to Gitea's registry | — | — | [05](05-gitops-and-delivery.md) |
| AI Program Manager | **pm-agent** (issues to pull requests through the gate) | — | [ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md) | [05](05-gitops-and-delivery.md) |
| Hardware transcode | **Intel GPU device plugin** (Plex Quick Sync) | — | — | [13](13-service-catalog.md) |

## Layer cake

```
┌──────────────────────────────────────────────────────────────────────────┐
│  Command & Control UI  (home.keithjackson.dev root)                        │
│  Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
├──────────────────────────────────────────────────────────────────────────┤
│  Workloads                                                                 │
│   PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …)              │
│   IaaS: KubeVirt VMs (anything that can't be a container)                  │
├──────────────────────────────────────────────────────────────────────────┤
│  Platform services                                                         │
│   Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API           │
│   Keycloak (SSO/IdP) + oauth2-proxy (forward-auth)                         │
│   kube-prometheus-stack + Loki + Alloy (observability)                     │
│   Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets)      │
│   cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots)  │
├──────────────────────────────────────────────────────────────────────────┤
│  Storage: Longhorn (replicated block)  +  TrueNAS NFS bulk tier (ADR-0018) │
├──────────────────────────────────────────────────────────────────────────┤
│  Network: Cilium  (default-deny NetworkPolicy · Egress Gateway · Hubble)   │
│           VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound  │
├──────────────────────────────────────────────────────────────────────────┤
│  Kubernetes: k3s (3 nodes, HA control plane, embedded etcd)                │
├──────────────────────────────────────────────────────────────────────────┤
│  Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible)       │
├──────────────────────────────────────────────────────────────────────────┤
│  Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex        │
│  Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
└──────────────────────────────────────────────────────────────────────────┘
```

## Two clearly-separated virtualization layers (do not conflate)

This platform has **two** kinds of VM, and confusing them is the most common design error:

1. **Infrastructure VMs = the cluster nodes themselves.**
   These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is
   historical shorthand for this layer, and it remains the place a node would be virtualized. See
   [`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).

2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
   These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03
   ("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).

> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.

## How a service flows through the platform (the golden path)

1. Operator adds/edits a **ServiceDefinition** in the git catalog (see
   [`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source,
   storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
   (explicit allow rules), update policy, and observability expectations.
2. **Flux** reconciles the change into the cluster.
3. **Longhorn** provisions replicated persistent volumes, so state is durable; large media attaches
   from the NFS **bulk tier**.
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
   **egress path** (local internet or VPN, with kill-switch).
5. A `public` service gets a proxied **ExternalDNS** record at `<service>.home.keithjackson.dev` and
   arrives through the **Cloudflare Tunnel**. An `internal` service has no public record and is
   answered on the LAN by **PiHole**. **Gateway API** routes north-south traffic for both;
   **cert-manager** issues a trusted TLS cert via ACME DNS-01.
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
   **C2 UI** lists and links the service.
7. **Renovate** keeps the service on the latest stable version per policy, through machine-authored
   changes that pass the merge gate.
8. **Velero and CNPG Barman**, writing off-cluster, protect the data; destroy/rebuild restores
   state.

Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).

## Design principles

- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it.
- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model.
- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets,
  trusted TLS everywhere.
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
  tested restore.
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
- **Promotable.** Configuration is one base plus an environment-scoped overlay.
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
  conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
  "done" is a green, requirement-tagged suite, not a manual demo.
- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move
  the implementation and its tests toward the spec, never the spec toward the implementation.