From 268b0a26b4b2c51338f18d452b9eb487ac1d1de4 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 21:20:13 -0500 Subject: [PATCH] =?UTF-8?q?docs(readme):=20DR=20restore=20is=20proven=20on?= =?UTF-8?q?=20prod=20=E2=80=94=20Phase=207's=20remaining=20item=20is=20nod?= =?UTF-8?q?e-loss?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in this repo to reach that state, with both verifiers recorded on prod. Also states why the earlier PROVEN: 0 was the honest answer rather than a bug: a proof is a claim about a specific tree on a specific cluster, so coverage (51/51) and proof are two different questions. Co-Authored-By: Claude Opus 5 --- README.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 0c76d82..4b224fa 100644 --- a/README.md +++ b/README.md @@ -51,12 +51,22 @@ alerts live*: |---|---| | prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only | | monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date | -| **DR proven** | **the open item** — R-STATE-03's only ledger proof was run on dev against MinIO | +| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only | -The DR gap is specific, not vague. `conformance/runs/ledger.jsonl` holds proofs for `dr-restore` -(R-STATE-03) and `node-loss` (R-STATE-02) that were both recorded **on the dev cluster**, which no -longer exists, and the dr-restore proof was against **MinIO** rather than today's Cloudflare R2 -target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod is what closes Phase 7. +**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 — +the first requirement in this repo to reach that state. Both of its verifiers are recorded: +`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`) +and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn` +class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy** +(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a +*different* PersistentVolume — so an in-cluster survival cannot pass as a restore. + +**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned +dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled +exercise rather than something to run in passing. + +Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a +bug but the honest answer, because proofs are claims about a specific tree on a specific cluster. | Phase | Delivered | |---|---| @@ -67,7 +77,7 @@ target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod i | 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled | | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate | -| **7 Production readiness** | **IN PROGRESS.** Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: DR proven on prod.** | +| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** | **Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed 2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with