fix(conformance): the ledger stamped every proof env=dev regardless of cluster

record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.

Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.

env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3629,3 +3629,28 @@ never applied. Configuration was present in almost every case.
   folders on disk.
 - Deleting the orphaned kopia blobs from the two R2-tagging failures.
 - The stale workstation dnsmasq override (`sudo rm` — blocks local name resolution).
+
+---
+
+## 2026-09-13 — DECIDED (owner): Velero+kopia to R2 IS the volume backup; no Longhorn backup target
+
+I had escalated this as a blocker for the Phase 7 DR drill: `settings.longhorn.io/backup-target` does
+not exist on prod (a WS-STORAGE Phase-4 leftover), so there is no Longhorn-native volume backup and
+no second restore path.
+
+**Owner's decision: that is fine.** Volume data is already protected by Velero + kopia to Cloudflare
+R2, and that is accepted as the practical backup for R-STATE-03. No Longhorn backup target will be
+configured. Corollary the owner also stated: **Longhorn configuration belongs in source**, which is
+where it now lives — the I/O caps and `nodeDrainPolicy` landed in `platform/longhorn/helmrelease.yaml`
+`defaultSettings` rather than as live-only settings (see the entry above).
+
+**What this makes true and what it does not.** The restore path is single: Velero's kopia file-system
+backup through the node-agent to R2. That is a genuine off-cluster copy — the `dr-restore` test proves
+it by asserting the restored PVC binds a *different* PersistentVolume than the original, so an
+in-cluster survival cannot masquerade as a restore. What it is not is redundant: a failure in Velero's
+R2 path has no Longhorn-side fallback, and the R2 target has already shown one incompatibility this
+month (the `velero-plugin-for-aws` `x-amz-tagging` 501, fixed by pinning the plugin). Worth revisiting
+only if that path proves unreliable, not on principle.
+
+Consequence for the roadmap: the DR drill is unblocked and Phase 7's remaining exit criterion is
+simply to run it on prod and record the proof.

README.md

@@ -38,22 +38,46 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
-open** — `clusters/prod` is a design-only overlay.
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **the open item** — R-STATE-03's only ledger proof was run on dev against MinIO |
+
+The DR gap is specific, not vague. `conformance/runs/ledger.jsonl` holds proofs for `dr-restore`
+(R-STATE-03) and `node-loss` (R-STATE-02) that were both recorded **on the dev cluster**, which no
+longer exists, and the dr-restore proof was against **MinIO** rather than today's Cloudflare R2
+target. `--status` therefore reports `PROVEN: 0`. Re-proving those two on prod is what closes Phase 7.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
 | 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS.** Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: DR proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.

conformance/lib/record_run.py

@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
     return "unknown"
 
 
+def context_env() -> str:
+    """The env a run TARGETED, derived from the live kube context — never guessed.
+
+    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
+    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
+    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
+    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
+    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
+    a proof that lies about where it was proven is worse than no proof, because the ledger's
+    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
+
+    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
+    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
+    name: an unreadable context must not silently attribute a proof to an environment.
+    """
+    try:
+        p = subprocess.run(["kubectl", "config", "current-context"],
+                           capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired):
+        return "unknown"
+    if p.returncode != 0:
+        return "unknown"
+    ctx = (p.stdout or "").strip()
+    if not ctx:
+        return "unknown"
+    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
+    # unexpected context is visible in the ledger rather than being coerced into a known name.
+    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
+
+
 def server_kube_version() -> str:
     """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
     for args in (["kubectl", "version", "-o", "json"],
@@ -883,7 +913,9 @@ def main() -> int:
                     help=f"append-only ledger (default {DEFAULT_LEDGER})")
     ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
     ap.add_argument("--repo", type=Path, default=REPO_ROOT)
-    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
+                    help="env the run targeted; derived from the kube context when "
+                         "unset (never defaulted to a cluster name)")
     ap.add_argument("--chainsaw-version", default=None,
                     help="override the versions.env pin (recorded verbatim)")
     ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),

conformance/run.sh

@@ -201,7 +201,7 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
   # to the real ledger; operators should never set it.
   "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
-    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
     --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
   log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
   log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
@@ -239,7 +239,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    --env "${CONF_ENV:-dev}" \
+    ${CONF_ENV:+--env "$CONF_ENV"} \
     "$@"
 }