docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
+22
-18
@@ -2,7 +2,9 @@
|
||||
|
||||
Delivery is phased. Each phase has an **entry gate** (what must be true to start), **exit criteria**
|
||||
(demonstrable outcomes tied to requirement IDs), and the **workstreams** ([workstreams.md](workstreams.md))
|
||||
that execute it. **Everything is proven in dev/test; prod is design-only until Phase 7.**
|
||||
that execute it. **Status: Phases 0–6 are delivered and Phase 7 is open** (see
|
||||
[known gaps](../docs/16-known-gaps.md) for what remains unproven). Phases 1–6 were first built on a
|
||||
dev/test cluster, decommissioned in 2026-09; prod is the only environment (ADR-0017).
|
||||
|
||||
A phase's exit criteria are **passing automated conformance tests** (per [`../AGENTS.md` §7](../AGENTS.md)
|
||||
and [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not "manifests
|
||||
@@ -26,12 +28,12 @@ every subsequent phase ships with tests, not after them.**
|
||||
runs green with the coverage report wired**; no cluster yet.
|
||||
**Serves:** R-GITOPS-02, R-ENV-02, R-TEST-01..04.
|
||||
|
||||
## Phase 1 — dev/test Cluster Bring-up (IaC)
|
||||
## Phase 1 — Cluster Bring-up (IaC)
|
||||
**Entry:** Phase 0 done.
|
||||
**Do:** OpenTofu libvirt node VMs + cloud-init baseline + Ansible k3s HA install with **Cilium**;
|
||||
disable bundled flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**.
|
||||
**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a dev node VM can be
|
||||
destroyed and rebuilt from IaC. **Prove:** node-loss re-provision.
|
||||
**Do:** node baseline + Ansible k3s HA install with **Cilium**; disable bundled
|
||||
flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**.
|
||||
**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a node can be
|
||||
destroyed and rebuilt from code. **Prove:** node-loss re-provision.
|
||||
**Serves:** R-PLAT-01, R-ENV-01, R-GITOPS-01, ADR-0001/0002/0004/0007.
|
||||
|
||||
## Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress
|
||||
@@ -46,26 +48,27 @@ split-horizon DNS foundation.
|
||||
- Auto DNS record appears for a test route (R-DNS-02).
|
||||
**Serves:** R-NET-01/02/03, R-STATE-01/02, R-TLS-01/02/03, R-DNS-01/02/03, ADR-0002/0003/0006.
|
||||
|
||||
## Phase 3 — Egress, VPN & Multi-WAN
|
||||
## Phase 3 — Egress & VPN
|
||||
**Entry:** Phase 2 done; OQ-1/OQ-3 answered enough to build.
|
||||
**Do:** implement egress classes; VPN egress (Gluetun sidecar default) with **kill-switch**;
|
||||
integrate edge-router multi-WAN split for the `internet` class.
|
||||
**Do:** implement egress classes; node-level VPN egress steered by Cilium Egress Gateway with a
|
||||
fail-closed **kill-switch** (ADR-0015, ADR-0020).
|
||||
**Exit (demonstrate):**
|
||||
- `vpn` test workload egresses via VPN IP; `internet` workload via home/WAN IP (R-NET-05).
|
||||
- **Kill-switch:** tunnel forced down ⇒ `vpn` workload leaks nothing; `internet` unaffected (R-NET-06).
|
||||
- Flipping `egress` class in git re-routes without workload rewrite (R-NET-07).
|
||||
**Serves:** R-NET-04/05/06/07, ADR-0008.
|
||||
**Serves:** R-NET-04/05/06/07, ADR-0013/0015/0020.
|
||||
|
||||
## Phase 4 — Observability, Backup/DR & Automated Updates
|
||||
**Entry:** Phase 2 done (can overlap Phase 3).
|
||||
**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero +
|
||||
Longhorn off-cluster backup; Flux image-automation + Renovate; per-service update policy plumbing.
|
||||
**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero
|
||||
and CNPG Barman off-cluster backup to Cloudflare R2; Renovate for images and charts; per-service
|
||||
update policy plumbing.
|
||||
**Exit (demonstrate):**
|
||||
- Injected fault emails an alert; dashboards render (R-OBS-01/02/03/04).
|
||||
- DR drill: restore a namespace + volumes from off-cluster backup (R-STATE-03).
|
||||
- A test service auto-advances a stable version on schedule with no human/AI edit; a bad update
|
||||
auto-rolls-back and alerts (R-UPD-01/02/03).
|
||||
**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0009/0003.
|
||||
**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0003/0018/0019/0023.
|
||||
|
||||
## Phase 5 — IaaS Proof (KubeVirt)
|
||||
**Entry:** Phases 2–3 done.
|
||||
@@ -76,7 +79,7 @@ survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates.
|
||||
|
||||
## Phase 6 — Service Catalog Onboarding + C2 UI
|
||||
**Entry:** Phases 2–4 done.
|
||||
**Do:** onboard the 14 services via ServiceDefinitions in dependency order (see workstream WS-APPS);
|
||||
**Do:** onboard the catalog services in dependency order (see workstream WS-APPS);
|
||||
**bring Gitea and Keycloak online early in this phase** — Keycloak is the foundational IdP so
|
||||
services adopt SSO (native OIDC / forward-auth) as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md));
|
||||
and cut the GitOps source + CI/CD over to Gitea (Gitea
|
||||
@@ -89,14 +92,15 @@ seerr↔plex dependency graph works and nothing undeclared does; `home.keithjack
|
||||
with live links + health.
|
||||
**Serves:** R-C2-01/02, all app-facing requirements, [`../docs/13-service-catalog.md`](../docs/13-service-catalog.md).
|
||||
|
||||
## Phase 7 — Production Readiness (design → build)
|
||||
**Entry:** Phases 1–6 proven in dev/test; operator declares prod hardware ready.
|
||||
## Phase 7 — Production Readiness
|
||||
**Status:** open; opened 2026-08-31.
|
||||
**Entry:** Phases 1–6 delivered; operator declares prod hardware ready.
|
||||
**Do:** finalize bare-metal provisioning checklist; apply the **prod overlay** (no base edits);
|
||||
full DR drill on prod; cutover plan.
|
||||
full DR drill on prod; decommission dev/test.
|
||||
**Exit:** prod 3× mini-PC cluster runs the same config via the prod overlay; DR proven; monitoring +
|
||||
alerts live.
|
||||
**Serves:** R-ENV-02/03, R-PLAT-01.
|
||||
**Note:** Do not begin building on prod hardware before this phase is explicitly opened
|
||||
**Note:** This phase was explicitly opened by the operator before any build on prod hardware
|
||||
([`../AGENTS.md` §6](../AGENTS.md)).
|
||||
|
||||
---
|
||||
|
||||
+12
-11
@@ -8,14 +8,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
| ID | Workstream | Owns (specs) | Serves | Depends on |
|
||||
|---|---|---|---|---|
|
||||
| **WS-INFRA** | Infra & cluster bring-up | [02](../docs/02-environments-and-provisioning.md) | R-PLAT-01, R-ENV-* | — |
|
||||
| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
|
||||
| **WS-GITOPS** | GitOps, CI, catalog renderer, image pipeline, PM agent | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
|
||||
| **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA |
|
||||
| **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA |
|
||||
| **WS-SEC** | Secrets, TLS, RBAC, **identity/SSO**, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md), [15](../docs/15-identity-and-access.md) | R-SEC-*, R-TLS-*, R-IDP-* | WS-INFRA, WS-NET |
|
||||
| **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE |
|
||||
| **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS |
|
||||
| **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE |
|
||||
| **WS-APPS** | Service onboarding (13 services) | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS |
|
||||
| **WS-APPS** | Service onboarding | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS |
|
||||
| **WS-C2** | Command & Control UI | [12](../docs/12-command-center-ui.md) | R-C2-* | WS-OBS, WS-APPS, WS-SEC |
|
||||
| **WS-TEST** | Conformance testing (cross-cutting) | [14](../docs/14-testing-and-conformance.md) | R-TEST-* | WS-INFRA, WS-GITOPS |
|
||||
|
||||
@@ -26,14 +26,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
contract in the system.**
|
||||
- **Network baseline & policy rendering** (WS-NET) — default-deny + how `dependencies` and `egress`
|
||||
compile to CiliumNetworkPolicies/egress routing. Consumed by WS-APPS/WS-VM.
|
||||
- **StorageClasses** (WS-STORAGE) — `longhorn` (replicated) and `bulk` (media). Consumed by everyone
|
||||
with state.
|
||||
- **StorageClasses** (WS-STORAGE) — the `longhorn` family (replicated, including static and VM
|
||||
variants) and `bulk` (media, NFS). Consumed by everyone with state.
|
||||
- **Issuer + Gateway TLS** (WS-SEC + WS-NET) — wildcard cert + Gateway listeners that HTTPRoutes
|
||||
attach to.
|
||||
- **Observability contract** (WS-OBS) — how a service declares metrics/SLIs/alerts and gets
|
||||
dashboards. Consumed by WS-APPS/WS-VM.
|
||||
- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's
|
||||
`update` block maps to.
|
||||
- **Update policy contract** (WS-UPDATE) — the Renovate conventions a ServiceDefinition's `update`
|
||||
block maps to.
|
||||
- **SSO/auth contract** (WS-SEC) — how a ServiceDefinition's `auth` block (oidc / forward-auth /
|
||||
native / groups) maps to a Keycloak client or oauth2-proxy protection ([15](../docs/15-identity-and-access.md)).
|
||||
Consumed by WS-APPS/WS-C2.
|
||||
@@ -61,17 +61,18 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
3. **keycloak (+ oauth2-proxy)** — **foundational IdP/SSO**, brought online early so later services
|
||||
integrate SSO as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md)).
|
||||
DR-critical + break-glass path. →
|
||||
4. **plex** (largest; validates bulk tier + public exposure) →
|
||||
4. **plex** (largest; validates bulk tier + L4 remote access) →
|
||||
5. **qbittorrent, jackett** (VPN + kill-switch proof) → 6. **sonarr, radarr** (VPN + arr↔downloader
|
||||
allows) → 7. **seerr** (cross-service allows to arr + plex) → 8. **nextcloud** (DR-critical data +
|
||||
backups) → 9. **minecraft, unifi, semaphore, kasm** (remaining).
|
||||
backups) → 9. **minecraft** (delivered as Crafty, platform-managed), **unifi** (a KubeVirt VM),
|
||||
**semaphore** → **kasm** (not onboarded).
|
||||
|
||||
As each service onboards, it adopts SSO per its `auth` mode (native OIDC or forward-auth) —
|
||||
[`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md).
|
||||
|
||||
**GitOps source migration** (WS-GITOPS, tied to step 2): local bootstrap source → self-hosted Gitea;
|
||||
CI (Kyverno gate + Chainsaw conformance Job trigger) moves to Gitea Actions; PRs become the enforced
|
||||
merge gate into `main`.
|
||||
**GitOps source** (WS-GITOPS, tied to step 2): Flux follows the self-hosted, in-cluster Gitea; CI
|
||||
(Kyverno gate + Chainsaw conformance Job trigger) runs on Gitea Actions; PRs are the enforced merge
|
||||
gate into `main`.
|
||||
|
||||
Each onboarding is a full acceptance-gate pass ([`../agents/definition-of-done.md`](../agents/definition-of-done.md)),
|
||||
not just a merged manifest.
|
||||
Reference in new issue
Block a user