docs(readme): DR restore is proven on prod — Phase 7's remaining item is node-loss
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in
this repo to reach that state, with both verifiers recorded on prod.

Also states why the earlier PROVEN: 0 was the honest answer rather than a bug:
a proof is a claim about a specific tree on a specific cluster, so coverage
(51/51) and proof are two different questions.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 21:20:13 -05:00
1 parent 4f2ca8c1a3
commit 268b0a26b4
1 file changed
+16 -6
+16 -6
View File
@@ -51,12 +51,22 @@ alerts live*:
|---|---|
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
| **DR proven** | **the open item** — R-STATE-03's only ledger proof was run on dev against MinIO |
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
The DR gap is specific, not vague. `conformance/runs/ledger.jsonl` holds proofs for `dr-restore`
(R-STATE-03) and `node-loss` (R-STATE-02) that were both recorded **on the dev cluster**, which no
longer exists, and the dr-restore proof was against **MinIO** rather than today's Cloudflare R2
target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod is what closes Phase 7.
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
exercise rather than something to run in passing.
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
| Phase | Delivered |
|---|---|
@@ -67,7 +77,7 @@ target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod i
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
| **7 Production readiness** | **IN PROGRESS.** Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: DR proven on prod.** |
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with