diff --git a/README.md b/README.md index 0c76d82..4b224fa 100644 --- a/README.md +++ b/README.md @@ -51,12 +51,22 @@ alerts live*: |---|---| | prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only | | monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date | -| **DR proven** | **the open item** — R-STATE-03's only ledger proof was run on dev against MinIO | +| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only | -The DR gap is specific, not vague. `conformance/runs/ledger.jsonl` holds proofs for `dr-restore` -(R-STATE-03) and `node-loss` (R-STATE-02) that were both recorded **on the dev cluster**, which no -longer exists, and the dr-restore proof was against **MinIO** rather than today's Cloudflare R2 -target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod is what closes Phase 7. +**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 — +the first requirement in this repo to reach that state. Both of its verifiers are recorded: +`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`) +and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn` +class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy** +(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a +*different* PersistentVolume — so an in-cluster survival cannot pass as a restore. + +**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned +dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled +exercise rather than something to run in passing. + +Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a +bug but the honest answer, because proofs are claims about a specific tree on a specific cluster. | Phase | Delivered | |---|---| @@ -67,7 +77,7 @@ target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod i | 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled | | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate | -| **7 Production readiness** | **IN PROGRESS.** Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: DR proven on prod.** | +| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** | **Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed 2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with