diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 92e1abe..fed53d1 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3629,3 +3629,28 @@ never applied. Configuration was present in almost every case. folders on disk. - Deleting the orphaned kopia blobs from the two R2-tagging failures. - The stale workstation dnsmasq override (`sudo rm` — blocks local name resolution). + +--- + +## 2026-09-13 — DECIDED (owner): Velero+kopia to R2 IS the volume backup; no Longhorn backup target + +I had escalated this as a blocker for the Phase 7 DR drill: `settings.longhorn.io/backup-target` does +not exist on prod (a WS-STORAGE Phase-4 leftover), so there is no Longhorn-native volume backup and +no second restore path. + +**Owner's decision: that is fine.** Volume data is already protected by Velero + kopia to Cloudflare +R2, and that is accepted as the practical backup for R-STATE-03. No Longhorn backup target will be +configured. Corollary the owner also stated: **Longhorn configuration belongs in source**, which is +where it now lives — the I/O caps and `nodeDrainPolicy` landed in `platform/longhorn/helmrelease.yaml` +`defaultSettings` rather than as live-only settings (see the entry above). + +**What this makes true and what it does not.** The restore path is single: Velero's kopia file-system +backup through the node-agent to R2. That is a genuine off-cluster copy — the `dr-restore` test proves +it by asserting the restored PVC binds a *different* PersistentVolume than the original, so an +in-cluster survival cannot masquerade as a restore. What it is not is redundant: a failure in Velero's +R2 path has no Longhorn-side fallback, and the R2 target has already shown one incompatibility this +month (the `velero-plugin-for-aws` `x-amz-tagging` 501, fixed by pinning the plugin). Worth revisiting +only if that path proves unreliable, not on principle. + +Consequence for the roadmap: the DR drill is unblocked and Phase 7's remaining exit criterion is +simply to run it on prod and record the proof. diff --git a/README.md b/README.md index 7fd74c1..0c76d82 100644 --- a/README.md +++ b/README.md @@ -38,22 +38,46 @@ ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render ## Status -Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not -open** — `clusters/prod` is a design-only overlay. +Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in +progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the +apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration: +dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod +hardware rather than on Proxmox VMs. + +**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring + +alerts live*: + +| criterion | state | +|---|---| +| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only | +| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date | +| **DR proven** | **the open item** — R-STATE-03's only ledger proof was run on dev against MinIO | + +The DR gap is specific, not vague. `conformance/runs/ledger.jsonl` holds proofs for `dr-restore` +(R-STATE-03) and `node-loss` (R-STATE-02) that were both recorded **on the dev cluster**, which no +longer exists, and the dr-restore proof was against **MinIO** rather than today's Cloudflare R2 +target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod is what closes Phase 7. | Phase | Delivered | |---|---| | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI | -| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS | +| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) | | 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS | | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch | -| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled | +| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled | | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate | +| **7 Production readiness** | **IN PROGRESS.** Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: DR proven on prod.** | -**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap -(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`; -the full report is `conformance/_artifacts/coverage/coverage.md`. +**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed +2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with +`./conformance/run.sh --coverage-only`; the full report is +`conformance/_artifacts/coverage/coverage.md`. + +Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and +*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current +SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the +second. The running narrative — decisions, root causes, incidents, follow-ups — is [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.