jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
125 lines
12 KiB
Markdown
125 lines
12 KiB
Markdown
# 01 — Architecture Overview
|
||
|
||
This document is the map. It names the chosen stack, shows how the layers fit, and points to the
|
||
ADR that locks in each choice and the domain spec that details it. **Do not change a technology
|
||
choice here without a superseding ADR.**
|
||
|
||
## The stack at a glance
|
||
|
||
| Layer | Choice | Serves | ADR | Spec |
|
||
|---|---|---|---|---|
|
||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||
| Distributed storage | **Longhorn** | R-STATE-01..02 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||
| Bulk media | **NFS** from a standalone **TrueNAS** on the production LAN | R-STATE-04 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||
| PostgreSQL | **CloudNativePG (CNPG)** | R-STATE-01/03, R-IDP-05 | [ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md) | [04](04-storage-and-state.md) |
|
||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||
| VPN egress | **Node-level PIA/OpenVPN tunnel** on a single gateway node, steered by **Cilium Egress Gateway** (Cilium-enforced fail-closed) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), [ADR-0020](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||
| Automated updates | **Renovate** (images and charts), every change PR-gated | R-UPD-01..03 | [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md) | [11](11-automated-updates.md) |
|
||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (explicit LAN answers) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||
| Public web ingress | **Cloudflare Tunnel** (outbound-only cloudflared) | R-DNS-04/05, R-SEC-04 | [ADR-0016](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md) | [09](09-dns-and-ingress.md) |
|
||
| Remote access | **WireGuard** on a hostNetwork DaemonSet | R-SEC-02/04 | [ADR-0021](../decisions/adr-0021-inbound-remote-access-wireguard.md) | [03](03-networking-isolation-and-egress.md) |
|
||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||
| Backup/DR | **Velero** (kopia) + **CNPG Barman** → Cloudflare R2; no Longhorn backup target | R-STATE-03 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||
| Node/infra provisioning | **Imaged Debian** on bare metal + **Ansible** (baseline, k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||
| Control-plane endpoint | **kube-vip** virtual IP | R-PLAT-01 | [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||
| Node patching & reboots | security-only **unattended-upgrades** + **kured** | R-PLAT-01 | — | [02](02-environments-and-provisioning.md) |
|
||
| Identity / SSO | **Keycloak** (OIDC + SAML) + per-app **oauth2-proxy** forward-auth sidecars | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md), [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | [15](15-identity-and-access.md) |
|
||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||
| Delivery & merge gate | **Gitea** + **Gitea Actions**; the CI gate is the approver | R-GITOPS-01, R-TEST-03 | [ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md) | [05](05-gitops-and-delivery.md) |
|
||
| Image builds | CI-run rootless **BuildKit**, published to Gitea's registry | — | — | [05](05-gitops-and-delivery.md) |
|
||
| AI Program Manager | **pm-agent** (issues to pull requests through the gate) | — | [ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md) | [05](05-gitops-and-delivery.md) |
|
||
| Hardware transcode | **Intel GPU device plugin** (Plex Quick Sync) | — | — | [13](13-service-catalog.md) |
|
||
|
||
## Layer cake
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ Command & Control UI (home.keithjackson.dev root) │
|
||
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Workloads │
|
||
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
|
||
│ IaaS: KubeVirt VMs (anything that can't be a container) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Platform services │
|
||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
|
||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||
│ Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets) │
|
||
│ cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Storage: Longhorn (replicated block) + TrueNAS NFS bulk tier (ADR-0018) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||
│ VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │
|
||
│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Two clearly-separated virtualization layers (do not conflate)
|
||
|
||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||
|
||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||
These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is
|
||
historical shorthand for this layer, and it remains the place a node would be virtualized. See
|
||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||
|
||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||
These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03
|
||
("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||
|
||
> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
|
||
|
||
## How a service flows through the platform (the golden path)
|
||
|
||
1. Operator adds/edits a **ServiceDefinition** in the git catalog (see
|
||
[`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source,
|
||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||
(explicit allow rules), update policy, and observability expectations.
|
||
2. **Flux** reconciles the change into the cluster.
|
||
3. **Longhorn** provisions replicated persistent volumes, so state is durable; large media attaches
|
||
from the NFS **bulk tier**.
|
||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||
**egress path** (local internet or VPN, with kill-switch).
|
||
5. A `public` service gets a proxied **ExternalDNS** record at `<service>.home.keithjackson.dev` and
|
||
arrives through the **Cloudflare Tunnel**. An `internal` service has no public record and is
|
||
answered on the LAN by **PiHole**. **Gateway API** routes north-south traffic for both;
|
||
**cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||
**C2 UI** lists and links the service.
|
||
7. **Renovate** keeps the service on the latest stable version per policy, through machine-authored
|
||
changes that pass the merge gate.
|
||
8. **Velero and CNPG Barman**, writing off-cluster, protect the data; destroy/rebuild restores
|
||
state.
|
||
|
||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||
|
||
## Design principles
|
||
|
||
- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it.
|
||
- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model.
|
||
- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets,
|
||
trusted TLS everywhere.
|
||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||
tested restore.
|
||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||
- **Promotable.** Configuration is one base plus an environment-scoped overlay.
|
||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||
- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move
|
||
the implementation and its tests toward the spec, never the spec toward the implementation.
|