docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -9,22 +9,32 @@ choice here without a superseding ADR.**
|
||||
| Layer | Choice | Serves | ADR | Spec |
|
||||
|---|---|---|---|---|
|
||||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..02 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Bulk media | **NFS** from a standalone **TrueNAS** on the production LAN | R-STATE-04 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||||
| PostgreSQL | **CloudNativePG (CNPG)** | R-STATE-01/03, R-IDP-05 | [ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md) | [04](04-storage-and-state.md) |
|
||||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Node-level PIA/OpenVPN tunnel** steered by **Cilium Egress Gateway** (host nftables kill-switch) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| VPN egress | **Node-level PIA/OpenVPN tunnel** on a single gateway node, steered by **Cilium Egress Gateway** (Cilium-enforced fail-closed) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), [ADR-0020](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Renovate** (images and charts), every change PR-gated | R-UPD-01..03 | [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (explicit LAN answers) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
| Public web ingress | **Cloudflare Tunnel** (outbound-only cloudflared) | R-DNS-04/05, R-SEC-04 | [ADR-0016](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md) | [09](09-dns-and-ingress.md) |
|
||||
| Remote access | **WireGuard** on a hostNetwork DaemonSet | R-SEC-02/04 | [ADR-0021](../decisions/adr-0021-inbound-remote-access-wireguard.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Identity / SSO | **Keycloak** (OIDC + SAML) + **oauth2-proxy** forward-auth | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md) | [15](15-identity-and-access.md) |
|
||||
| Backup/DR | **Velero** (kopia) + **CNPG Barman** → Cloudflare R2; no Longhorn backup target | R-STATE-03 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **Imaged Debian** on bare metal + **Ansible** (baseline, k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Control-plane endpoint | **kube-vip** virtual IP | R-PLAT-01 | [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Node patching & reboots | security-only **unattended-upgrades** + **kured** | R-PLAT-01 | — | [02](02-environments-and-provisioning.md) |
|
||||
| Identity / SSO | **Keycloak** (OIDC + SAML) + per-app **oauth2-proxy** forward-auth sidecars | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md), [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | [15](15-identity-and-access.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||||
| Delivery & merge gate | **Gitea** + **Gitea Actions**; the CI gate is the approver | R-GITOPS-01, R-TEST-03 | [ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md) | [05](05-gitops-and-delivery.md) |
|
||||
| Image builds | CI-run rootless **BuildKit**, published to Gitea's registry | — | — | [05](05-gitops-and-delivery.md) |
|
||||
| AI Program Manager | **pm-agent** (issues to pull requests through the gate) | — | [ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md) | [05](05-gitops-and-delivery.md) |
|
||||
| Hardware transcode | **Intel GPU device plugin** (Plex Quick Sync) | — | — | [13](13-service-catalog.md) |
|
||||
|
||||
## Layer cake
|
||||
|
||||
@@ -41,18 +51,17 @@ choice here without a superseding ADR.**
|
||||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||||
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
|
||||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||||
│ Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets) │
|
||||
│ cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||||
│ Storage: Longhorn (replicated block) + TrueNAS NFS bulk tier (ADR-0018) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||||
│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │
|
||||
│ VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Nodes │
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
│ Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │
|
||||
│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
|
||||
@@ -64,8 +73,8 @@ choice here without a superseding ADR.**
|
||||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||||
|
||||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||||
In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and
|
||||
configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
|
||||
These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is
|
||||
historical shorthand for this layer, and it remains the place a node would be virtualized. See
|
||||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||||
|
||||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||||
@@ -81,16 +90,20 @@ This platform has **two** kinds of VM, and confusing them is the most common des
|
||||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||||
(explicit allow rules), update policy, and observability expectations.
|
||||
2. **Flux** reconciles the change into the cluster.
|
||||
3. **Longhorn** provisions replicated persistent volumes; state is durable.
|
||||
3. **Longhorn** provisions replicated persistent volumes, so state is durable; large media attaches
|
||||
from the NFS **bulk tier**.
|
||||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||||
**egress path** (local internet or VPN, with kill-switch).
|
||||
5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south
|
||||
traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
5. A `public` service gets a proxied **ExternalDNS** record at `<service>.home.keithjackson.dev` and
|
||||
arrives through the **Cloudflare Tunnel**. An `internal` service has no public record and is
|
||||
answered on the LAN by **PiHole**. **Gateway API** routes north-south traffic for both;
|
||||
**cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||||
**C2 UI** lists and links the service.
|
||||
7. **Flux image automation / Renovate** keep the service on the latest stable version per policy —
|
||||
no human action.
|
||||
8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state.
|
||||
7. **Renovate** keeps the service on the latest stable version per policy, through machine-authored
|
||||
changes that pass the merge gate.
|
||||
8. **Velero and CNPG Barman**, writing off-cluster, protect the data; destroy/rebuild restores
|
||||
state.
|
||||
|
||||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||||
|
||||
@@ -103,7 +116,7 @@ Every numbered step maps to a domain spec and a workstream in [`roadmap/workstre
|
||||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||||
tested restore.
|
||||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||||
- **Promotable.** dev/test and prod are the same config with scoped overrides.
|
||||
- **Promotable.** Configuration is one base plus an environment-scoped overlay.
|
||||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||||
|
||||
Reference in new issue
Block a user