docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+22 -18
View File
@@ -2,7 +2,9 @@
Delivery is phased. Each phase has an **entry gate** (what must be true to start), **exit criteria**
(demonstrable outcomes tied to requirement IDs), and the **workstreams** ([workstreams.md](workstreams.md))
that execute it. **Everything is proven in dev/test; prod is design-only until Phase 7.**
that execute it. **Status: Phases 0–6 are delivered and Phase 7 is open** (see
[known gaps](../docs/16-known-gaps.md) for what remains unproven). Phases 1–6 were first built on a
dev/test cluster, decommissioned in 2026-09; prod is the only environment (ADR-0017).
A phase's exit criteria are **passing automated conformance tests** (per [`../AGENTS.md` §7](../AGENTS.md)
and [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not "manifests
@@ -26,12 +28,12 @@ every subsequent phase ships with tests, not after them.**
runs green with the coverage report wired**; no cluster yet.
**Serves:** R-GITOPS-02, R-ENV-02, R-TEST-01..04.
## Phase 1 — dev/test Cluster Bring-up (IaC)
## Phase 1 — Cluster Bring-up (IaC)
**Entry:** Phase 0 done.
**Do:** OpenTofu libvirt node VMs + cloud-init baseline + Ansible k3s HA install with **Cilium**;
disable bundled flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**.
**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a dev node VM can be
destroyed and rebuilt from IaC. **Prove:** node-loss re-provision.
**Do:** node baseline + Ansible k3s HA install with **Cilium**; disable bundled
flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**.
**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a node can be
destroyed and rebuilt from code. **Prove:** node-loss re-provision.
**Serves:** R-PLAT-01, R-ENV-01, R-GITOPS-01, ADR-0001/0002/0004/0007.
## Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress
@@ -46,26 +48,27 @@ split-horizon DNS foundation.
- Auto DNS record appears for a test route (R-DNS-02).
**Serves:** R-NET-01/02/03, R-STATE-01/02, R-TLS-01/02/03, R-DNS-01/02/03, ADR-0002/0003/0006.
## Phase 3 — Egress, VPN & Multi-WAN
## Phase 3 — Egress & VPN
**Entry:** Phase 2 done; OQ-1/OQ-3 answered enough to build.
**Do:** implement egress classes; VPN egress (Gluetun sidecar default) with **kill-switch**;
integrate edge-router multi-WAN split for the `internet` class.
**Do:** implement egress classes; node-level VPN egress steered by Cilium Egress Gateway with a
fail-closed **kill-switch** (ADR-0015, ADR-0020).
**Exit (demonstrate):**
- `vpn` test workload egresses via VPN IP; `internet` workload via home/WAN IP (R-NET-05).
- **Kill-switch:** tunnel forced down ⇒ `vpn` workload leaks nothing; `internet` unaffected (R-NET-06).
- Flipping `egress` class in git re-routes without workload rewrite (R-NET-07).
**Serves:** R-NET-04/05/06/07, ADR-0008.
**Serves:** R-NET-04/05/06/07, ADR-0013/0015/0020.
## Phase 4 — Observability, Backup/DR & Automated Updates
**Entry:** Phase 2 done (can overlap Phase 3).
**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero +
Longhorn off-cluster backup; Flux image-automation + Renovate; per-service update policy plumbing.
**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero
and CNPG Barman off-cluster backup to Cloudflare R2; Renovate for images and charts; per-service
update policy plumbing.
**Exit (demonstrate):**
- Injected fault emails an alert; dashboards render (R-OBS-01/02/03/04).
- DR drill: restore a namespace + volumes from off-cluster backup (R-STATE-03).
- A test service auto-advances a stable version on schedule with no human/AI edit; a bad update
auto-rolls-back and alerts (R-UPD-01/02/03).
**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0009/0003.
**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0003/0018/0019/0023.
## Phase 5 — IaaS Proof (KubeVirt)
**Entry:** Phases 2–3 done.
@@ -76,7 +79,7 @@ survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates.
## Phase 6 — Service Catalog Onboarding + C2 UI
**Entry:** Phases 2–4 done.
**Do:** onboard the 14 services via ServiceDefinitions in dependency order (see workstream WS-APPS);
**Do:** onboard the catalog services in dependency order (see workstream WS-APPS);
**bring Gitea and Keycloak online early in this phase** — Keycloak is the foundational IdP so
services adopt SSO (native OIDC / forward-auth) as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md));
and cut the GitOps source + CI/CD over to Gitea (Gitea
@@ -89,14 +92,15 @@ seerr↔plex dependency graph works and nothing undeclared does; `home.keithjack
with live links + health.
**Serves:** R-C2-01/02, all app-facing requirements, [`../docs/13-service-catalog.md`](../docs/13-service-catalog.md).
## Phase 7 — Production Readiness (design → build)
**Entry:** Phases 1–6 proven in dev/test; operator declares prod hardware ready.
## Phase 7 — Production Readiness
**Status:** open; opened 2026-08-31.
**Entry:** Phases 1–6 delivered; operator declares prod hardware ready.
**Do:** finalize bare-metal provisioning checklist; apply the **prod overlay** (no base edits);
full DR drill on prod; cutover plan.
full DR drill on prod; decommission dev/test.
**Exit:** prod 3× mini-PC cluster runs the same config via the prod overlay; DR proven; monitoring +
alerts live.
**Serves:** R-ENV-02/03, R-PLAT-01.
**Note:** Do not begin building on prod hardware before this phase is explicitly opened
**Note:** This phase was explicitly opened by the operator before any build on prod hardware
([`../AGENTS.md` §6](../AGENTS.md)).
---
+12 -11
View File
@@ -8,14 +8,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
| ID | Workstream | Owns (specs) | Serves | Depends on |
|---|---|---|---|---|
| **WS-INFRA** | Infra & cluster bring-up | [02](../docs/02-environments-and-provisioning.md) | R-PLAT-01, R-ENV-* | — |
| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
| **WS-GITOPS** | GitOps, CI, catalog renderer, image pipeline, PM agent | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
| **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA |
| **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA |
| **WS-SEC** | Secrets, TLS, RBAC, **identity/SSO**, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md), [15](../docs/15-identity-and-access.md) | R-SEC-*, R-TLS-*, R-IDP-* | WS-INFRA, WS-NET |
| **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE |
| **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS |
| **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE |
| **WS-APPS** | Service onboarding (13 services) | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS |
| **WS-APPS** | Service onboarding | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS |
| **WS-C2** | Command & Control UI | [12](../docs/12-command-center-ui.md) | R-C2-* | WS-OBS, WS-APPS, WS-SEC |
| **WS-TEST** | Conformance testing (cross-cutting) | [14](../docs/14-testing-and-conformance.md) | R-TEST-* | WS-INFRA, WS-GITOPS |
@@ -26,14 +26,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
contract in the system.**
- **Network baseline & policy rendering** (WS-NET) — default-deny + how `dependencies` and `egress`
compile to CiliumNetworkPolicies/egress routing. Consumed by WS-APPS/WS-VM.
- **StorageClasses** (WS-STORAGE) — `longhorn` (replicated) and `bulk` (media). Consumed by everyone
with state.
- **StorageClasses** (WS-STORAGE) — the `longhorn` family (replicated, including static and VM
variants) and `bulk` (media, NFS). Consumed by everyone with state.
- **Issuer + Gateway TLS** (WS-SEC + WS-NET) — wildcard cert + Gateway listeners that HTTPRoutes
attach to.
- **Observability contract** (WS-OBS) — how a service declares metrics/SLIs/alerts and gets
dashboards. Consumed by WS-APPS/WS-VM.
- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's
`update` block maps to.
- **Update policy contract** (WS-UPDATE) — the Renovate conventions a ServiceDefinition's `update`
block maps to.
- **SSO/auth contract** (WS-SEC) — how a ServiceDefinition's `auth` block (oidc / forward-auth /
native / groups) maps to a Keycloak client or oauth2-proxy protection ([15](../docs/15-identity-and-access.md)).
Consumed by WS-APPS/WS-C2.
@@ -61,17 +61,18 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
3. **keycloak (+ oauth2-proxy)** — **foundational IdP/SSO**, brought online early so later services
integrate SSO as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md)).
DR-critical + break-glass path. →
4. **plex** (largest; validates bulk tier + public exposure) →
4. **plex** (largest; validates bulk tier + L4 remote access) →
5. **qbittorrent, jackett** (VPN + kill-switch proof) → 6. **sonarr, radarr** (VPN + arr↔downloader
allows) → 7. **seerr** (cross-service allows to arr + plex) → 8. **nextcloud** (DR-critical data +
backups) → 9. **minecraft, unifi, semaphore, kasm** (remaining).
backups) → 9. **minecraft** (delivered as Crafty, platform-managed), **unifi** (a KubeVirt VM),
**semaphore** → **kasm** (not onboarded).
As each service onboards, it adopts SSO per its `auth` mode (native OIDC or forward-auth) —
[`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md).
**GitOps source migration** (WS-GITOPS, tied to step 2): local bootstrap source → self-hosted Gitea;
CI (Kyverno gate + Chainsaw conformance Job trigger) moves to Gitea Actions; PRs become the enforced
merge gate into `main`.
**GitOps source** (WS-GITOPS, tied to step 2): Flux follows the self-hosted, in-cluster Gitea; CI
(Kyverno gate + Chainsaw conformance Job trigger) runs on Gitea Actions; PRs are the enforced merge
gate into `main`.
Each onboarding is a full acceptance-gate pass ([`../agents/definition-of-done.md`](../agents/definition-of-done.md)),
not just a merged manifest.