jakman85andClaude Opus 4.8
7598cff945
The in-cluster Gluetun-pod gateway (ADR-0013) cannot be steered: a pod's tun0 is invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own default route (breaks k3s/etcd). The tunnel itself works (PIA egress validated), but not the steering. New decision (ADR-0015): move the tunnel to a HOST OpenVPN systemd service on egress-gateway node(s) (Ansible-managed, split-tunnel so it doesn't touch the node default route), steer egress-class=vpn pods with a CiliumEgressGatewayPolicy (native eBPF, no client-pod changes), and enforce the kill-switch with a host nftables rule (fail-closed independent of Cilium). PIA/OpenVPN-UDP provider decision (ADR-0013) stands; the .ovpn transfers over; PF re-solved host-side (Phase 6). Amended for consistency: ADR-0013 (client+steering superseded), decisions/README, docs/00, docs/01, docs/03, docs/07, agents/task-board. Co-Authored-By: Claude Opus 4.8 <[email protected]>
10 KiB
10 KiB
01 — Architecture Overview
This document is the map. It names the chosen stack, shows how the layers fit, and points to the ADR that locks in each choice and the domain spec that details it. Do not change a technology choice here without a superseding ADR.
The stack at a glance
| Layer | Choice | Serves | ADR | Spec |
|---|---|---|---|---|
| Kubernetes distro | k3s (HA, embedded etcd) | R-PLAT-01/02 | ADR-0001 | 02 |
| CNI / network policy / egress | Cilium (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | ADR-0002 | 03 |
| Distributed storage | Longhorn | R-STATE-01..03 | ADR-0003 | 04 |
| GitOps engine | Flux CD | R-GITOPS-01..03 | ADR-0004 | 05 |
| VM workloads (IaaS) | KubeVirt (+ CDI) | R-PLAT-03 | ADR-0005 | 06 |
| TLS | cert-manager + Let's Encrypt (DNS-01) | R-TLS-01..03 | ADR-0006 | 08 |
| Secrets | SOPS + age (Flux-native) | R-SEC-01 | ADR-0007 | 07 |
| VPN egress | Node-level PIA/OpenVPN tunnel steered by Cilium Egress Gateway (host nftables kill-switch) | R-NET-05/06/07 | ADR-0015 (provider 0013) | 03 |
| Automated updates | Flux image-update-automation (images) + Renovate (charts) | R-UPD-01..03 | ADR-0009 | 11 |
| DNS records | ExternalDNS (public via Cloudflare) + PiHole (internal) | R-DNS-01..03 | — | 09 |
| Ingress (north-south) | Gateway API implemented by Cilium | R-DNS-01, R-TLS-01 | ADR-0002 | 09 |
| Observability | kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager→SMTP | R-OBS-01..04 | — | 10 |
| Backup/DR | Velero + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | ADR-0003 | 04 |
| Node/infra provisioning | OpenTofu (libvirt) + cloud-init + Ansible (k3s bootstrap) | R-ENV-01..03 | ADR-0001 | 02 |
| Identity / SSO | Keycloak (OIDC + SAML) + oauth2-proxy forward-auth | R-IDP-01..06 | ADR-0014 | 15 |
| Command center UI | Homepage + Headlamp + Grafana panels (behind Keycloak SSO) | R-C2-01/02 | — | 12 |
| Conformance testing | Kyverno (L0 policy) + Chainsaw (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | ADR-0010, ADR-0011 | 14 |
Layer cake
┌──────────────────────────────────────────────────────────────────────────┐
│ Command & Control UI (home.keithjackson.dev root) │
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
├──────────────────────────────────────────────────────────────────────────┤
│ Workloads │
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
│ IaaS: KubeVirt VMs (anything that can't be a container) │
├──────────────────────────────────────────────────────────────────────────┤
│ Platform services │
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
│ kube-prometheus-stack + Loki + Alloy (observability) │
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
├──────────────────────────────────────────────────────────────────────────┤
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
├──────────────────────────────────────────────────────────────────────────┤
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │
├──────────────────────────────────────────────────────────────────────────┤
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
├──────────────────────────────────────────────────────────────────────────┤
│ Nodes │
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
│ prod: 3× mini-PC bare metal, local disks │
├──────────────────────────────────────────────────────────────────────────┤
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
│ VPN uplink(s), split-horizon DNS │
└──────────────────────────────────────────────────────────────────────────┘
Two clearly-separated virtualization layers (do not conflate)
This platform has two kinds of VM, and confusing them is the most common design error:
-
Infrastructure VMs = the cluster nodes themselves. In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
02-environments-and-provisioning.md. -
Application VMs = IaaS workloads that run inside the cluster via KubeVirt. These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 ("not everything fits in a container"). See
06-vm-workloads-kubevirt.md.
Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
How a service flows through the platform (the golden path)
- Operator adds/edits a ServiceDefinition in the git catalog (see
13-service-catalog.md) declaring: kind (container|vm), image/source, storage, egress class (internet|vpn), exposure (internal|public), dependencies (explicit allow rules), update policy, and observability expectations. - Flux reconciles the change into the cluster.
- Longhorn provisions replicated persistent volumes; state is durable.
- Cilium applies default-deny plus the explicitly declared allow rules; assigns the correct egress path (local internet or VPN, with kill-switch).
- ExternalDNS publishes
<service>.home.keithjackson.dev; Gateway API routes north-south traffic; cert-manager issues a trusted TLS cert via ACME DNS-01. - kube-prometheus-stack + Loki scrape metrics/logs; Alertmanager emails on issues; the C2 UI lists and links the service.
- Flux image automation / Renovate keep the service on the latest stable version per policy — no human action.
- Velero + Longhorn backups protect the data; destroy/rebuild restores state.
Every numbered step maps to a domain spec and a workstream in roadmap/workstreams.md.
Design principles
- Declarative everything. If it can't be expressed as YAML reconciled from git, question it.
- One control plane. Containers and VMs share the Kubernetes surface; operators learn one model.
- Secure by default. Deny-by-default network, least-privilege identity, encrypted secrets, trusted TLS everywhere.
- Durable by default. No workload is "pet" hardware; state lives on replicated storage with tested restore.
- Hands-off operations. Updates, cert renewal, DNS, and drift correction are automated.
- Promotable. dev/test and prod are the same config with scoped overrides.
- Provable. Every requirement is verified by an automated, reproducible, behavior-based conformance test (14, ADR-0010); "done" is a green, requirement-tagged suite, not a manual demo.
- Spec-fixed. The specification is human-owned and immutable to agents (R-TEST-06); agents move the implementation and its tests toward the spec, never the spec toward the implementation.