jakman85andClaude Opus 4.8
2cca9beac7
Operator turned off ExpressVPN auto-renew and purchased PIA. The single-shared- gateway model is retained; provider/client/protocol change to PIA via Gluetun over WireGuard with native port forwarding (qbittorrent-natmap port-sync). Simpler than ExpressVPN: no custom OpenVPN config, just PIA user/pass SOPS-encrypted; Gluetun provides the built-in kill-switch. Updated: ADR-0013 (new) + ADR-0012 superseded + index; docs 00,01,03,07; task board (T-3.2p/T-3.2, OQ-3). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
110 lines
9.8 KiB
Markdown
110 lines
9.8 KiB
Markdown
# 01 — Architecture Overview
|
||
|
||
This document is the map. It names the chosen stack, shows how the layers fit, and points to the
|
||
ADR that locks in each choice and the domain spec that details it. **Do not change a technology
|
||
choice here without a superseding ADR.**
|
||
|
||
## The stack at a glance
|
||
|
||
| Layer | Choice | Serves | ADR | Spec |
|
||
|---|---|---|---|---|
|
||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||
| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||
| VPN egress | **Single shared in-cluster PIA/Gluetun gateway** (WireGuard + native PF) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md) (supersedes 0012/0008) | [03](03-networking-isolation-and-egress.md) |
|
||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||
|
||
## Layer cake
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ Command & Control UI (home.keithjackson.dev root) │
|
||
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Workloads │
|
||
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
|
||
│ IaaS: KubeVirt VMs (anything that can't be a container) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Platform services │
|
||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Nodes │
|
||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||
│ prod: 3× mini-PC bare metal, local disks │
|
||
├──────────────────────────────────────────────────────────────────────────┤
|
||
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
|
||
│ VPN uplink(s), split-horizon DNS │
|
||
└──────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Two clearly-separated virtualization layers (do not conflate)
|
||
|
||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||
|
||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||
In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and
|
||
configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
|
||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||
|
||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||
These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03
|
||
("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||
|
||
> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
|
||
|
||
## How a service flows through the platform (the golden path)
|
||
|
||
1. Operator adds/edits a **ServiceDefinition** in the git catalog (see
|
||
[`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source,
|
||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||
(explicit allow rules), update policy, and observability expectations.
|
||
2. **Flux** reconciles the change into the cluster.
|
||
3. **Longhorn** provisions replicated persistent volumes; state is durable.
|
||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||
**egress path** (local internet or VPN, with kill-switch).
|
||
5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south
|
||
traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||
**C2 UI** lists and links the service.
|
||
7. **Flux image automation / Renovate** keep the service on the latest stable version per policy —
|
||
no human action.
|
||
8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state.
|
||
|
||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||
|
||
## Design principles
|
||
|
||
- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it.
|
||
- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model.
|
||
- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets,
|
||
trusted TLS everywhere.
|
||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||
tested restore.
|
||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||
- **Promotable.** dev/test and prod are the same config with scoped overrides.
|
||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||
- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move
|
||
the implementation and its tests toward the spec, never the spec toward the implementation.
|