Revert "feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex"

This reverts commit 9137bbd0c3.

.gitignore

@@ -44,3 +44,6 @@ infra/tofu/.r2.env
 
 # Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
 infra/ansible/collections/
+
+# standup.sh DNS snapshots — run artifacts, not authored state
+.standup-dns/

DECISIONS-LOG.md

@@ -1758,3 +1758,279 @@ into a transient hard Kustomization failure.
 - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
   with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
   Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
+
+## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched
+
+Operator asleep; standing instruction was "finish out what is necessary and I want it working by
+the morning." **Prod is serving.** Dev was verified unharmed at every step.
+
+### Final state
+
+| | prod | dev |
+|---|---|---|
+| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** |
+| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` |
+| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) |
+| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready |
+| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged |
+| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** |
+
+End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`,
+keycloak `302` (login), gitea `200`, grafana `302` (SSO).
+
+### What was actually blocking, and what it taught
+
+**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a
+Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.**
+
+1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation
+   limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers
+   with no API types to watch. The failed install then tripped `install.remediation.retries: 3`,
+   whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller
+   cannot advance from. Cleared with `helm uninstall` + a cold reinstall.
+2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR
+   *and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until
+   `cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier
+   can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev**
+   (its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying
+   only the CR; the CRD appeared in 10s.
+
+> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own
+> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation
+> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a
+> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster.
+
+### The armed footgun, disarmed
+
+Prod's external-dns Deployment was running `--policy=sync --registry=txt
+--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`,
+the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the
+records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was
+`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a
+guard, it is luck.**
+
+The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The
+fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported
+`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the
+current config, so it would never have upgraded on its own.
+
+**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets
+replicas to 1 and applies new args in one operation — there is no window to check first. So
+`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=<owner>`
+appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into
+a precondition.
+
+### Three bugs in my own verification — recorded because a check you cannot trust is worse than none
+
+1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline
+   was discarded between steps and the before/after comparison — the check that detects damage to the
+   other cluster — could never run across invocations. Now repo-relative and gitignored.
+2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing
+   pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev`
+   differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots
+   now strip CRs and sort under a fixed collation.
+3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script
+   claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight
+   is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is
+   what actually causes harm.
+
+### I also fell into the pfSense trap — ACTION REQUIRED
+
+The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new
+`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare
+correctly returns `192.168.2.20`.
+
+**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the
+override, or add one pointing `prod.home.*` at `192.168.2.20`.
+
+It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches
+`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that
+as a broken prod cert before checking the resolver. **This is the same trap that produced the false
+healthy reading during the incident. Never verify DNS or TLS here through a local resolver.**
+
+### Still open for the operator
+
+- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes
+  it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup
+  coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`).
+- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a
+  rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
+- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease
+  change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
+- **Six L2 tests hardcode the apex** and must be parameterized per environment.
+- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.
+
+## 2026-09-05 (afternoon) — **PM autonomous session.** Backups to R2, CDI deadlock repaired, coverage 82%→88% — and a guard I broke
+
+Operator away ~2h, PM directed to work autonomously through agents. Four agents ran;
+every result was re-verified by the PM before commit. **Nothing merged, nothing pushed.**
+Both clusters healthy throughout — dev 13/13, prod 12/13 (velero deliberately suspended).
+
+Commits: `32d6766` R2 · `765ad03` CDI · `59ffbaf` L2 params · `8f9b2e4` s3check · `fe45d0f` coverage.
+
+### THE MISTAKE, AND WHY IT MATTERS MORE THAN THE FEATURES
+
+In `32d6766` I changed the backup placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
+`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`. The guards in `standup.sh` matched
+`TBD-*` — a **prefix** glob. `fnmatch('https://TBD-R2-…','TBD-*')` is **False**, so all three
+guards silently died, and **`resume` would have enabled velero on prod against a nonexistent
+hostname**: installs clean, reports healthy, no backup exists. Precisely the failure the guard
+existed to prevent, disarmed by a value change in a *different file*.
+
+**The class, now seen three times today** (this; the CRD count that read a blank line as `1`;
+two coverage checks that passed when perturbed): *a check that degrades to success*. The
+structural lesson is narrower and worth keeping — **a guard keyed on a value's SHAPE, where the
+value lives in another file, has no test tying the two together.** Fixed by substring match;
+a test for the coupling is **not** yet written and should be.
+
+### Operator decisions waiting
+
+1. **R-TLS-02 "revoked" is NOT satisfied — a real platform gap.** The spec requires certs be
+   "issued, renewed, **and revoked** without manual steps"; docs/08 reads revocation as a
+   declarative removal. That needs `enableCertificateOwnerRef: true`, which
+   `platform/cert-manager/helmrelease.yaml` does not set (chart default false). **Verified on
+   both live clusters: `gateway/wildcard-home-tls` has ZERO ownerReferences** — deleting the
+   Certificate leaves the private key serving indefinitely.
+   **Deliberately not flipped.** The wildcard Certificate is under `./platform`, which runs
+   `prune: true`. With owner refs ON, a transient prune of that Certificate — *exactly what
+   happened to the CDI StorageProfiles today* — deletes the SERVING TLS Secret for every host,
+   and reissuance is bounded by LE's duplicate-certificate limit (5/week per identical name set),
+   risking **no trusted cert for a week**. Security gap vs plausible week-long outage: operator's
+   call. The L2 test asserts the requirement as written and **will fail until decided** — it was
+   not softened.
+2. **R2 needs the two token values.** `infra/BACKUP-R2.md` is the procedure. The **account id is
+   already committed** (`infra/tofu/backend.r2.hcl` → `7ce452a1b3d6c86daf2c020edcb82d37`), so if
+   the buckets go in that account only the token is missing. Placeholder kept on purpose.
+3. **The apex flip is also the legacy-prod 443 cutover.** The SOHO router NATs inbound 443 to the
+   pfSense WAN for the non-k3s prod. §10a sequence P does not cover repointing it.
+
+### Merge preconditions — read before merging anything here
+
+- **CDI split:** `flux get kustomization platform-config` must be **Ready on the target first**.
+  Otherwise the two StorageProfiles are pruned and sit at CDI defaults until it recovers (prod was
+  in that state hours earlier). Measured exposure: CDI unaided infers **ReadWriteOnce**
+  (`longhorn-static`) where we author **ReadWriteMany** (`longhorn-vm`) — so a DataVolume created
+  in the ~30–60s window is **permanently non-live-migratable**. `cloneStrategy` is *not* at risk.
+- **R2 switch RESETS dev's PITR window** — fresh kopia repo, new CNPG base backups, the 30-day
+  Barman window rebuilds from zero. The MinIO copy is retained and restorable.
+- **R-STATE-03 is PROVEN MinIO-ONLY** and its ledger row predates the R2 move. `BACKUP-R2.md` §5
+  is the re-proof and it ends in a **restore**, not an upload.
+
+### Findings worth keeping
+
+- **Second-order CRDs need a third pass.** A CR whose CRD is created by a *controller at runtime*
+  cannot share a Kustomization with resources of that CRD. `cdi-operator` registers exactly one
+  CRD; `storageprofiles`/`datavolumes`/`datasources` appear only after it reconciles the CDI CR.
+  Audit found `StorageProfile` was the only second-order kind in `platform-config`.
+- **`dns-exposure`'s teardown is ASYNCHRONOUS.** It reports PASS when chainsaw deletes the
+  namespace; the Cloudflare record is reaped later by external-dns's 1-minute sync. A green run
+  does **not** prove cleanup — if external-dns were suspended the record would persist silently.
+  (Observed, then observed clearing.)
+- **The apex was not the only per-environment literal in the L2 suite** — two DNS tests hardcoded
+  dev's LB range `192.168.1.53-.60`. They now assert the *observed* `GATEWAY_VIP` / assigned LB IP,
+  which is stricter as well as portable.
+- **Velero's kopia backend has no checksum knob**, so BSL metadata and PVC-data R2 compatibility
+  are **separate questions**. CNPG's boto3 1.35.60 is below the 1.36 default-checksum line — clear
+  today, but **a CNPG image bump is an R2 re-verification event** (barman-cloud has no escape hatch).
+- **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into
+  the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI.
+  Confirmed no plaintext reached disk.
+
+## 2026-09-06 — Prod solidified. Node 2 at full 32 GB, storage recovered, four latent cold-start bugs closed
+
+Operator away; PM directed to oversee, delegate, and pay down debt while skipping anything needing
+their input. Five agents ran; every result re-verified by the PM before commit. **Nothing merged,
+nothing pushed. Both clusters healthy.**
+
+Commits: `04569ab` node-maintenance runbook · `da3e9cc` Kyverno/CDI · `a984442` timezone + guard test.
+
+### PROD IS FUNCTIONING
+
+| | |
+|---|---|
+| Nodes | 3/3 Ready, **all three at ~`32638256Ki`** — node 2's DIMM upgrade landed, matching 1 and 3 to within 4 KiB |
+| Flux | **12/13 tiers Ready** (`velero` suspended: no backup endpoint) |
+| Pods | **112 Running**, 0 Running-but-NotReady |
+| Storage | **17/17 volumes healthy**, replicas symmetric 17-running+1-stopped per node |
+| TLS | LE wildcard `*.prod.home.keithjackson.dev` Ready |
+| etcd | 3 members, survived an ungraceful power-down without losing a member |
+
+The memattr saga (§0b) is **closed on the hardware side**: with the 7.1.8 backports kernel, all three
+nodes address their full 32 GB.
+
+### THE FINDING THAT MATTERS MOST — a failed drain silently stops storage self-healing
+
+`kubectl drain` **cordons before it fails**, and Longhorn's `disable-scheduling-on-cordoned-node` is
+`true`. After prod-2's memory-upgrade reboot, **all 17 volumes sat `attached/degraded` with prod-2
+holding 2 of 18 replicas and ZERO rebuilds in flight** — nothing driving recovery, every pod
+`Running`, no alert. `kubectl uncordon` started rebuilds immediately; converged 17/17 in ~6 minutes.
+
+**Uncordon is not tidying up — it is what re-enables redundancy.** Full runbook + post-maintenance
+checklist in `infra/PROD-STANDUP.md` §Node maintenance, including why `drain` can never complete here
+(instance-manager PDBs at 0 allowed disruptions; at 3 nodes/3 replicas replica eviction has nowhere to
+go, which is structural, not a setting).
+
+### FOUR BUGS OF ONE SHAPE — *a thing that looks applied and isn't*
+
+Worth naming as a class, because it has now appeared four times in two days:
+
+1. **Kyverno's own CRDs** — `crds.install: false`, nothing else installed them; pods crash-looped 55×.
+2. **CDI StorageProfiles** — CR and its dependent resource in the same Flux pass; the dry-run aborts
+   the tier, so the CR never applies and its CRD never appears. *Repaired in the graph* (`765ad03`).
+3. **The placeholder guard** — a `TBD-*` prefix glob stopped matching when the value it guards changed
+   *in a different file*; `resume` would have enabled velero against a nonexistent host. Now tested
+   (`a984442`).
+4. **PolicyException** — the CRD is installed but every controller runs
+   `--enablePolicyException=false`. An exception would apply cleanly through Flux and be **silently
+   ignored**. Verified live on prod.
+
+Two more of the same family: a coverage check that passed when perturbed (asked a question always
+satisfiable), and `ci/touched-requirements.sh` having **no `bootstrap/*` entry**, which scoped the L0
+run away from the very check guarding the placeholder class. Both fixed.
+
+**The rule:** dev's age hides cold-start defects. Anything that only runs on a fresh cluster has never
+been exercised on dev, and prod is the first honest test. And: *a guard keyed on the shape of a value
+that lives in another file needs a test tying the two together.*
+
+### Kyverno blocking CDI — fixed by SATISFYING the policy, not exempting it
+
+Prod's UniFi VM had no disk: Kyverno denied CDI's generated `importer-*` pod for missing the
+`auth-mode` / `egress-class` labels, which CDI cannot set. Fixed with a **mutate** policy that adds
+them when the requester is ServiceAccount `cdi/cdi-sa` — identity from AdmissionReview userInfo, **not
+a name prefix** (a prefix is attacker-choosable; the test proves it with a forged
+`importer-prime-forged-by-an-app` fixture that is still denied). Two-way proof: same pod, different
+requester, opposite outcomes. Kyverno L0 17 → 29 tests.
+
+Rejected `exclude` scoped to the same SA because any userInfo match forces `background: false`, which
+would drop background scanning of R-NET-05/R-IDP-02 for **all existing pods** to fix one pod class.
+
+### Conformance 82.4% → 88.2%
+
+45/51 scaffolded, 0 safety-critical gaps. Six L2 tests made environment-agnostic (they derive
+`BASE_DOMAIN`/`GATEWAY_VIP` from the cluster they run against, so they cannot drift); R-TLS-02,
+R-ENV-01 and R-ENV-03 covered — the last two matter because the Phase-7 entry-gate note recorded
+R-ENV-01/02/03 as uncovered while prod provisioning was proceeding.
+
+### STAGED FOR ONE PUSH — `prod/deliver-fixes`
+
+Prod reconciles bootstrap-Gitea `main@c8c1d29` and **cannot take the working branch** — that carries
+the DNS inversion, which would move prod to the apex and collide with dev. `prod/deliver-fixes` is
+`c8c1d29` + the CDI graph fix + the Kyverno mutate policy: a clean fast-forward, verified free of any
+cluster-vars/externaldns/overlay change, prod's `BASE_DOMAIN` untouched, render checked. **Not pushed
+— that changes what a live cluster reconciles.**
+
+### Waiting on the operator (deliberately skipped)
+
+- **R2 credentials.** Two token values; the account id is already committed in
+  `infra/tofu/backend.r2.hcl`. `infra/BACKUP-R2.md` is the procedure. **Prod has no backup coverage.**
+- **`enableCertificateOwnerRef`** — R-TLS-02's "revoked" clause is unmet; both clusters' TLS Secrets
+  have zero ownerReferences. Not flipped because the wildcard Certificate lives under a `prune: true`
+  tier, so a transient prune would delete the serving Secret and reissuance is capped at 5/week.
+- **Node timezone** — authored (`Etc/UTC`, matching dev), `--check` confirms the drift, **not applied**:
+  a live clock change during storage resync is not an unattended action.
+- **Database HA** — all three CNPG clusters are `instances: 1` and all three primaries are on prod-1.
+  Losing that node takes Keycloak, Nextcloud and Semaphore together. `instances: 2` spends resources.
+- **The apex flip** is also the legacy-prod 443 cutover; §10a sequence P does not cover the SOHO NAT.
+- **R-STATE-04** stays a gap: `bulk` is a non-provisioning stub pending **OQ-2**.
+- Orphaned `prime-be3f9d1c-…` 40 GiB scratch PVC from the blocked imports — operator cleanup.

README.md

@@ -69,6 +69,17 @@ infra/devcontainer/dev.sh run make -C ci gate   # the same gate CI runs
 infra/devcontainer/dev.sh                  # interactive shell (sops, flux, kubectl, tofu, ansible)
 ```
 
+`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
+target prod instead:
+
+```bash
+infra/devcontainer/dev.sh run kubectl get nodes                # dev (default)
+CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod
+```
+
+Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
+`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
+
 The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
 so the spec-immutability check can actually run. See
 [`infra/devcontainer/README.md`](infra/devcontainer/README.md).

apps/vms/unifi/30-virtualmachine.yaml

@@ -102,7 +102,7 @@ spec:
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
               packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]

apps/vms/unifi/50-service-l4.yaml

@@ -46,7 +46,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

apps/wave-0/gitea.yaml

@@ -90,9 +90,9 @@ spec:
               name: gitea-bootstrap
               key: secret-key
         - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                 \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
@@ -207,7 +207,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/jackett.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/nextcloud-db.yaml

@@ -79,7 +79,11 @@ spec:
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
+      # NAME — and both environments have a `nextcloud-db`. Sharing one MinIO with a shared path
+      # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
+      # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:

apps/wave-0/pihole.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/plex.yaml

@@ -183,7 +183,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/qbittorrent.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/semaphore-db.yaml

@@ -80,7 +80,11 @@ spec:
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
+      # NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path
+      # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
+      # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:

apps/wave-1/gitea-actions.yaml

@@ -165,7 +165,7 @@ spec:
               exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:

apps/wave-1/nextcloud.yaml

@@ -104,13 +104,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -125,9 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -228,7 +228,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/radarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/semaphore.yaml

@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -231,7 +231,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/sonarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-2/seerr.yaml

@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

bootstrap/flux/README.md

@@ -11,7 +11,8 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
+| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
 | `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
@@ -74,6 +75,49 @@ flux get kustomizations -A                                  # all 11 must return
 **Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
 simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
 
+## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
+
+`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
+stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
+`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
+`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
+HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
+
+The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
+Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
+because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
+
+```
+helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
+  | yq 'select(.kind == "CustomResourceDefinition")'
+  | kubectl apply --server-side --force-conflicts
+```
+
+- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
+  read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
+  **constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
+  helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
+  of that one line carries this step with it. There is nothing here that can drift.
+- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
+  HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
+  HelmRelease over field managers. The `yq` filter is load-bearing.
+- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
+  hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
+  `serverside-applied` and exits 0.
+- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
+  `null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
+  exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
+  waited on for `condition=Established`. There is **no skip flag** on purpose.
+- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
+  repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
+
+Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
+
+```bash
+# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
+bash bootstrap/flux/preapply-crds.sh
+```
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
@@ -98,7 +142,7 @@ tofu apply → cloud-init → Ansible (k3s + Cilium)          [Infra worker, T-1
         ▼  (final Ansible play calls the entrypoints below)
 1. bootstrap/sops/age-keygen.sh          # operator generates age key OUTSIDE repo
 2. bootstrap/sops/age-key-secret.md      # operator installs private key as sops-age Secret
-3. bootstrap/flux/bootstrap.sh           # flux install → verify sops-age → apply -k clusters/dev
+3. bootstrap/flux/bootstrap.sh           # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
         │
         ▼
 Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
@@ -118,8 +162,10 @@ live in `infra/devcontainer/versions.env`).
    template). `bootstrap.sh` **fails fast** if this Secret is absent.
 3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
 
-`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
+`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
-never handles key material — it only *verifies* the Secret exists.
+`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
+exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
+both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
 
 ## Handoff — clusters owner / PM (files this worker may not edit)
 

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -30,10 +33,19 @@
 #                         the single apps.yaml was retired). Letting
 #                         bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
 #                         applying the authored clusters/dev overlay is the reconciling path.
-#   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
+#   2. CRD pre-apply   — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
+#                         platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
+#                         (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
+#                         which STALLS the HelmRelease and blocks the `platform` Kustomization and
+#                         every tier below it). With CRD install off, NOTHING else installs them, so
+#                         a cold cluster can never reconcile `platform` without this step. The chart
+#                         version is DERIVED from that same HelmRelease — no second pin. Dev only
+#                         ever had these CRDs because they were applied BY HAND; this is that step,
+#                         in git, idempotent and version-derived. See ./preapply-crds.sh.
+#   3. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +64,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,10 +77,13 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
-for bin in flux kubectl; do
+# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
+# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
+# Flux controllers are installed, instead of half-way through.
+for bin in flux kubectl helm yq; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
@@ -93,7 +111,7 @@ done
 #
 # ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
 # strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
-echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
 flux_install_args=(--namespace="${FLUX_NAMESPACE}")
 [ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
 
@@ -124,18 +142,29 @@ else
   kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
-# --- 2. verify the out-of-band age key Secret exists -------------------------
+# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
+# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
+# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
+# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
+# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
+# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
+# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
+# cluster that looks bootstrapped and reconciles nothing.
+echo "==> [2/4] pre-applying chart CRDs (server-side)"
+bash "${script_dir}/preapply-crds.sh"
+
+# --- 3. verify the out-of-band age key Secret exists -------------------------
 # We DO NOT create it here (it holds the private key). The operator installs it per
 # ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
-echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
+echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
 if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
   echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
   echo "       Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 4. apply the authored environment entrypoint ----------------------------
-echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
+echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 
 echo

bootstrap/flux/preapply-crds.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+#
+# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
+# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
+# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
+# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
+# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
+#
+# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
+# =============================================================================
+#   *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
+# =============================================================================
+#
+# WHY THIS EXISTS
+#
+#   platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
+#   the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
+#   annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
+#   STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
+#   it (the kps-cascade failure mode; see the HelmRelease's own comment).
+#
+#   The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
+#   Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
+#   cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
+#   undocumented manual step this repo exists to eliminate. This script is that step, in git,
+#   version-derived and idempotent.
+#
+# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
+#
+#   Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
+#
+#     chart name      spec.chart.spec.chart            (HelmRelease)
+#     chart version   spec.chart.spec.version           (HelmRelease)  e.g. "3.9.x"
+#     chart repo url  spec.url                          (HelmRepository)
+#     namespace       metadata.namespace                (HelmRelease)
+#
+#   The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
+#   constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
+#   through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
+#   one file and this step follows automatically — it cannot drift, because there is nothing here to
+#   drift from.
+#
+# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
+#
+#   The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
+#   into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
+#   ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
+#   filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
+#   (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
+#
+# IDEMPOTENCY
+#
+#   `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
+#   every later run reports "serverside-applied" and exits 0, and field ownership is taken from
+#   whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
+#   hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
+#   Server-side apply does not write the last-applied annotation, which is the whole reason the
+#   5.6 MiB bundle can be applied at all.
+#
+# LOUD FAILURE (this script must never degrade to a successful no-op)
+#
+#   `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
+#   class of bug this repo has been bitten by more than once:
+#     1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
+#     2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
+#        filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
+#     3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
+#        to mean "the API type exists", not merely "kubectl returned 0".
+#
+set -euo pipefail
+
+# --- config (env-overridable) ------------------------------------------------
+# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
+# Flux reconciles, so it is the file this step must agree with.
+CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
+# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
+CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
+
+echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
+
+# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
+# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
+# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
+for bin in helm yq kubectl; do
+  command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
+done
+
+helmrelease="${repo_root}/${CRD_HELMRELEASE}"
+[ -f "${helmrelease}" ] || {
+  echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
+  exit 1
+}
+
+# --- 1. derive the chart coordinates from the authored manifests --------------
+chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
+chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
+chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
+chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
+
+# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
+# "null" for a missing path and an empty string for a missing document, so both are rejected.
+for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
+  key="${pair%%:*}"; val="${pair#*:}"
+  case "${val}" in
+    ""|null|*$'\n'*)
+      echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
+      echo "       The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
+      exit 1
+      ;;
+  esac
+done
+
+echo "    chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
+echo "    (version is the constraint Flux resolves — NOT a second copy pinned here)"
+
+# --- 2. render the chart and keep ONLY the CRD documents ----------------------
+render_dir="$(mktemp -d)"
+trap 'rm -rf "${render_dir}"' EXIT
+crd_file="${render_dir}/crds.yaml"
+
+# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
+# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
+echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
+helm template "${chart_name}" "${chart_name}" \
+  --repo "${chart_repo}" \
+  --version "${chart_version}" \
+  --namespace "${chart_ns}" \
+  --include-crds \
+  --set crds.install=true \
+  | yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
+
+# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
+# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
+mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
+crd_count="${#crd_names[@]}"
+
+# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
+[ "${crd_count}" -gt 0 ] || {
+  echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
+  echo "       The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
+  echo "       applying nothing would look like success and leave 'platform' blocked forever." >&2
+  exit 1
+}
+echo "    ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
+
+# --- 3. server-side apply (idempotent upsert) ---------------------------------
+# --server-side: no last-applied annotation, so the size limit does not apply.
+# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
+#   Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
+echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
+kubectl apply --server-side --force-conflicts -f "${crd_file}"
+
+# --- 4. prove the API types actually exist -----------------------------------
+# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
+# controllers and the platform Kustomization actually need.
+echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
+crd_refs=()
+for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
+kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
+  "${crd_refs[@]}" >/dev/null || {
+  echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
+  echo "       Inspect: kubectl get crd | grep ${chart_name}" >&2
+  exit 1
+}
+
+echo "==> ${crd_count} ${chart_name} CRDs applied and Established."

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

bootstrap/standup.sh

@@ -0,0 +1,988 @@
+#!/usr/bin/env bash
+#
+# standup.sh — bring a bootstrapped cluster's reconciliation graph all the way up.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): it reads everything it needs from the cluster it is pointed at and
+# from the authored manifests. There is no dev/prod fork. What makes it safe to run against prod is
+# the MANDATORY --expect-server guard, not a name in the filename.
+#
+# WHY THIS EXISTS
+#   `bootstrap/flux/bootstrap.sh` installs Flux and applies the graph. It does NOT finish the job when
+#   tiers have been deliberately suspended (a DNS/TLS migration window, an incident recovery) or when
+#   a HelmRelease has wedged. Those last steps were living in shell history and in a human's head,
+#   which is the same failure the Kyverno CRD pre-apply had. This is that sequence, in git, ordered,
+#   idempotent, and gated.
+#
+# THE ORDERING IS LOAD-BEARING. Read infra/PROD-STANDUP.md before deviating from it.
+#
+# USAGE
+#   bootstrap/standup.sh --kubeconfig <path> --expect-server <url> [--confirm-destructive] <step...>
+#
+#   steps:  preflight  crds  unwedge  externaldns  s3check  resume  verify   (or: all)
+#
+#   --expect-server is REQUIRED and is compared against the live API server before ANY command runs.
+#   Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and
+#   shares a Cloudflare zone with prod. The guard is not optional and there is no override flag.
+#
+#   --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
+#
+#   --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
+#   cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
+#   `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
+#   nothing else and applies nothing. See infra/BACKUP-R2.md.
+#
+set -euo pipefail
+
+# ---------------------------------------------------------------------------- args
+KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE=""
+STEPS=()
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --kubeconfig)          KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;;
+    --expect-server)       EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;;
+    --confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;;
+    --s3-endpoint)         S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;;
+    -h|--help)             sed -n '2,30p' "$0"; exit 0 ;;
+    -*)                    echo "ERROR: unknown flag '$1'" >&2; exit 2 ;;
+    *)                     STEPS+=("$1"); shift ;;
+  esac
+done
+
+[ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; }
+[ -n "${EXPECT_SERVER}" ]  || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; }
+[ "${#STEPS[@]}" -gt 0 ]   || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; }
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))"
+cd "${repo_root}"
+
+export KUBECONFIG="${KUBECONFIG_ARG}"
+[ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; }
+
+# Where DNS snapshots go. This MUST be a path that survives between invocations: each `dev.sh run` is
+# a FRESH container, so a $TMPDIR default silently threw the baseline away and made `verify`'s
+# before/after comparison — the check that detects damage to the OTHER cluster's records — impossible
+# to run across steps. The repo is bind-mounted, so a repo-relative dir persists on the host.
+# Gitignored: these are run artifacts, not authored state.
+SNAP_DIR="${STANDUP_SNAP_DIR:-${repo_root}/.standup-dns}"
+mkdir -p "${SNAP_DIR}"
+
+say()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
+
+for bin in kubectl flux helm yq sops; do
+  command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
+done
+
+# ------------------------------------------------------------------ THE SAFETY GUARD
+# Checked once, before any step, and re-checked by every mutating helper. The failure mode this
+# prevents is running a prod bring-up against the LIVE dev cluster, whose public DNS records prod
+# has already destroyed once (2026-09-04).
+live_server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
+[ "${live_server}" = "${EXPECT_SERVER}" ] || die \
+  "TARGET MISMATCH — refusing to run.
+       --expect-server : ${EXPECT_SERVER}
+       live API server : ${live_server}
+       The kubeconfig does not point where you think it does. Nothing has been changed."
+say "target confirmed: ${live_server}"
+
+# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
+# each wait is meaningful instead of blocking on a dependency that is still suspended.
+#   flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
+#                                     -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
+#
+# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
+# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
+# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
+# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
+# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
+# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
+TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
+# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
+# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
+
+cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
+
+k_ready()  { kubectl -n flux-system get kustomization "$1" \
+               -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true; }
+k_msg()    { kubectl -n flux-system get kustomization "$1" \
+               -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' 2>/dev/null || true; }
+k_susp()   { kubectl -n flux-system get kustomization "$1" -o jsonpath='{.spec.suspend}' 2>/dev/null || true; }
+
+wait_ready() { # <kustomization> <timeout-seconds>
+  local n="$1" t="${2:-600}" i=0
+  while [ "$i" -lt "$t" ]; do
+    [ "$(k_ready "$n")" = "True" ] && { info "$n Ready (${i}s)"; return 0; }
+    sleep 10; i=$((i+10))
+  done
+  warn "$n did NOT reach Ready in ${t}s: $(k_msg "$n")"
+  return 1
+}
+
+# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable.
+# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard
+# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false
+# healthy reading during the 2026-09-04 incident and hid the damage for a while.
+dns_snapshot() { # <output-file>
+  local out="$1" tok zid page tp
+  tok="$(sops -d platform/externaldns/cloudflare-api-token.sops.yaml \
+         | yq '.stringData."api-token"' | tr -d '"')"
+  [ -n "${tok}" ] && [ "${tok}" != "null" ] || die "could not read the Cloudflare token (SOPS/age key available?)"
+  zid="$(curl -sS -H "Authorization: Bearer ${tok}" \
+        'https://api.cloudflare.com/client/v4/zones?name=keithjackson.dev' | yq '.result[0].id' | tr -d '"')"
+  [ -n "${zid}" ] && [ "${zid}" != "null" ] || die "Cloudflare zone lookup failed"
+  page=1; : > "${out}"
+  while :; do
+    local r; r="$(curl -sS -H "Authorization: Bearer ${tok}" \
+      "https://api.cloudflare.com/client/v4/zones/${zid}/dns_records?per_page=100&page=${page}")"
+    printf '%s' "${r}" | yq -o=json '.result' \
+      | yq -r '.[] | (.type + "|" + .name + "|" + .content)' | tr -d '"' >> "${out}"
+    tp="$(printf '%s' "${r}" | yq '.result_info.total_pages' | tr -d '"')"
+    [ "${page}" -ge "${tp:-1}" ] && break
+    page=$((page+1))
+  done
+  # Strip CRs, and sort under a FIXED collation. Both matter, and both were learned the hard way:
+  #   - a stray \r makes every line differ from a baseline captured without one;
+  #   - locale-dependent sort ordering puts `*.keithjackson.dev` in a different position on the host
+  #     than in the container, so a diff reports pairs of IDENTICAL lines as changed. That produced a
+  #     "RECORDS OUTSIDE ... CHANGED" alarm on a zone where nothing had changed at all.
+  # A regression check that cries wolf is worse than none — it trains you to skip the one alarm that
+  # matters. LC_ALL=C makes the ordering reproducible everywhere.
+  tr -d '\r' < "${out}" | grep -vE '^\s*$' | LC_ALL=C sort -o "${out}"
+  info "$(wc -l < "${out}") records -> ${out}"
+}
+
+dns_diff() { # <before> <after>  — returns non-zero if anything changed
+  if diff -u "$1" "$2" > "${SNAP_DIR}/dns.diff"; then
+    info "Cloudflare zone IDENTICAL — nothing created, modified or deleted."
+    return 0
+  fi
+  warn "Cloudflare zone CHANGED:"; cat "${SNAP_DIR}/dns.diff"
+  return 1
+}
+
+# ======================================================================= s3check plumbing
+# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
+# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
+# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
+# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
+# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
+# hold, from the network position they actually occupy.
+
+# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
+#   - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
+#     so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
+#   - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
+#     (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
+#     discovered by a silently-failing backup at 03:00;
+#   - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
+#     already broken one probe in this repo — and it cannot be made to send the checksum headers, so
+#     it cannot test for the trap at all.
+S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
+S3CHECK_RUN_ID="$(date +%s)"
+
+# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
+# must never leave a pod behind on a live cluster.
+S3CHECK_PODS=()
+s3_cleanup_pods() {
+  local p
+  for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
+    kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
+  done
+  S3CHECK_PODS=()
+}
+trap s3_cleanup_pods EXIT
+
+# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
+# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
+# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
+# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
+s3_is_placeholder() { # <endpoint>
+  case "$1" in
+    ""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
+  esac
+  return 1
+}
+
+# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
+# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
+# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
+# value with a trailing space or a stray quote is a defect, not something to paper over.
+s3_safe_token() { # <label> <value>
+  case "$2" in
+    *[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
+       That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
+  esac
+}
+
+# ---------------------------------------------------------------- the in-pod probe
+# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
+# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
+#   - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
+#   - every failure prints `S3CHECK-FAIL ...` and is counted;
+#   - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
+# The caller independently knows how many assertions this environment must produce and re-counts the
+# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
+# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
+# past; that class of bug is designed out here rather than commented about.)
+s3_probe_script() {
+  cat <<'PROBE'
+set -u
+umask 077
+A=0; F=0
+ok()   { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
+bad()  { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
+note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
+verdict() {
+  if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
+    printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
+  fi
+  printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
+}
+
+for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
+  eval "x=\${${v}-}"
+  [ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
+done
+
+export AWS_EC2_METADATA_DISABLED=true
+export AWS_PAGER=""
+export AWS_DEFAULT_REGION="${S3REGION}"
+export AWS_REGION="${S3REGION}"
+export AWS_MAX_ATTEMPTS=2
+export AWS_RETRY_MODE=standard
+export AWS_CONFIG_FILE=/tmp/s3check-aws-config
+# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
+# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
+# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
+# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
+# future CLI version.
+{
+  echo '[default]'
+  echo 'request_checksum_calculation = when_required'
+  echo 'response_checksum_validation = when_required'
+  echo 's3 ='
+  echo '    addressing_style = path'
+} > "${AWS_CONFIG_FILE}"
+
+# ---- (1) the credential loads at all -------------------------------------------------
+# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
+# byte counts and lengths leave this pod.
+case "${CRED_MODE}" in
+  file)
+    cf="${AWS_SHARED_CREDENTIALS_FILE-}"
+    if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
+      bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
+    elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
+      bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
+        the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
+    elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
+      || ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
+      bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
+    else
+      ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
+    fi
+    ;;
+  env)
+    if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
+      bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
+    else
+      ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
+    fi
+    ;;
+  *) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
+esac
+[ "${F}" -eq 0 ] || verdict     # no point talking to S3 with no credential
+
+OUT=""; RC=0
+awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
+
+detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
+
+# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
+diagnose() { # <operation label>
+  local op="$1"
+  case "${OUT}" in
+    *InvalidArgument*|*checksum*|*Checksum*)
+      bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
+        AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
+        Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
+        every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
+        backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
+        checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
+    *"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
+    *"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
+    *"Connection reset"*|*"Read timeout"*)
+      bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
+        install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
+        while no backup exists. This is the reading that matters: the workstation's own reachability
+        is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
+        pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
+        or give this environment its own endpoint." ;;
+    *NoSuchBucket*|*"(404)"*|*"Not Found"*)
+      bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
+        (R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
+    *InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
+      bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
+        different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
+    *AccessDenied*|*"(403)"*|*Forbidden*)
+      bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
+        needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
+    *) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
+  esac
+  detail
+}
+
+PAY=/tmp/s3check.payload
+GOT=/tmp/s3check.got
+head -c 4096 /dev/urandom | base64 > "${PAY}"
+[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
+
+TRAP_DONE=0
+for t in ${S3TARGETS}; do
+  B="${t%%:*}"; P="${t#*:}"
+  [ "${P}" = "${B}" ] && P=""
+  if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
+
+  # ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
+  #          WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
+  #          needs it and a correctly bucket-scoped R2 token may not grant it.
+  args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
+  [ -n "${P}" ] && args+=(--prefix "${P}/")
+  awsx "${args[@]}"
+  if [ "${RC}" -eq 0 ]; then
+    ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
+  else
+    diagnose "LIST ${B}"
+    continue      # a short assertion count is how the caller learns this target was abandoned
+  fi
+
+  # ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
+  #          nothing about whether an object can be STORED (wrong scope, checksum rejection and
+  #          read-only tokens all pass a HEAD and fail a backup).
+  awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
+  else diagnose "PUT ${B}/${K}"; continue; fi
+
+  rm -f "${GOT}"
+  awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
+  else diagnose "GET ${B}/${K}"
+       aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
+       continue; fi
+
+  sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
+  hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
+  hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
+  if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
+    ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
+  else
+    bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
+  fi
+
+  awsx s3api delete-object --bucket "${B}" --key "${K}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
+  else diagnose "DELETE ${B}/${K}"; fi
+
+  awsx s3api head-object --bucket "${B}" --key "${K}"
+  if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
+  else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
+
+  # ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
+  #          checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
+  #          that line is LOAD-BEARING. Reported either way so the operator learns which world this
+  #          endpoint lives in before, not after, enabling velero.
+  if [ "${TRAP_DONE}" -eq 0 ]; then
+    TRAP_DONE=1
+    TK="${K}.checksum-trap-probe"
+    OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
+           aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
+    if [ "${RC}" -eq 0 ]; then
+      note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
+        present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
+      aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
+        || note "checksum probe: could NOT delete ${TK} — remove it by hand"
+    else
+      case "${OUT}" in
+        *InvalidArgument*)
+          note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
+        x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
+        checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
+        velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
+        *) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
+      esac
+    fi
+  fi
+done
+
+verdict
+PROBE
+}
+
+# ---------------------------------------------------------------- run one probe pod
+# <ns> <file|env> <secret> <space-separated bucket:prefix targets>  <endpoint> <region>
+s3_probe() {
+  local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
+  # $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
+  # would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
+  local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
+
+  # What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
+  # byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
+  # pod's own claim is never the only source of the expected count.
+  local t n=0
+  for t in ${targets}; do n=$((n+1)); done
+  local min=$((1 + 6 * n))
+
+  local volsjson="" mountsjson="" creds=""
+  case "${mode}" in
+    file)  # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
+      volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
+      mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
+      creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
+    env)   # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
+      creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
+    *) die "internal: unknown probe mode '${mode}'" ;;
+  esac
+
+  local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
+  [ -n "${b64}" ] || die "internal: probe script encoded to nothing"
+
+  # egress-class label: harmless here (the velero namespace is not an app namespace) but required by
+  # the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
+  # the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
+  # the check would report "unknown" instead of a result.
+  local overrides
+  overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
+
+  say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}'  → ${min} assertions required"
+  kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
+  S3CHECK_PODS+=("${ns}/${pod}")
+  kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
+      --overrides="${overrides}" >/dev/null 2>&1 \
+    || die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
+       That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
+
+  local i=0 phase=""
+  while [ "${i}" -lt 300 ]; do
+    phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+    case "${phase}" in Succeeded|Failed) break ;; esac
+    sleep 5; i=$((i+5))
+  done
+  case "${phase}" in
+    Succeeded|Failed) ;;
+    *) local reason
+       reason="$(kubectl -n "${ns}" get pod "${pod}" \
+                 -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
+       die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
+       NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
+  esac
+
+  local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
+  [ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
+       an empty log is the shape of a probe that never ran, not of a healthy backup target."
+  printf '%s\n' "${log}" | while IFS= read -r l; do printf '    %s\n' "${l}"; done
+
+  # ---- the four independent accounting checks that make a silent no-op impossible ----
+  local okcount failcount resline nres verd a f
+  okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
+  failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
+  nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
+  [ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
+       A missing verdict line means the probe died mid-way; it is never a pass."
+  resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
+  read -r _ verd a f <<<"${resline}"
+  a="${a#assertions=}"; f="${f#failures=}"
+  case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
+  case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
+  [ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
+       Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
+       or CNPG backups against this endpoint until they are all gone."
+  [ "${f}" = "0" ]          || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
+  [ "${failcount}" = "0" ]  || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
+  [ "${a}" -ge "${min}" ]   || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
+       (1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
+       degrade-to-a-successful-no-op this check exists to prevent."
+  [ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
+       lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
+  [ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
+  info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
+}
+
+# ============================================================================ steps
+step_preflight() {
+  say "PREFLIGHT (read-only)"
+  info "nodes Ready: $(kubectl get nodes --no-headers | awk '$2=="Ready"' | wc -l)/$(kubectl get nodes --no-headers | wc -l)"
+  info "flux controllers Running: $(kubectl -n flux-system get pods --no-headers | awk '$3=="Running"' | wc -l)"
+  # Count BOTH groups the Kyverno chart ships. Grepping only 'kyverno.io' undercounts by 2 — the
+  # policy-report types are in the wgpolicyk8s.io group — which reads as "CRDs are missing" when the
+  # pre-apply in fact succeeded. Expect 22 for kyverno 3.9.x.
+  info "kyverno CRDs present: $(kubectl get crd --no-headers 2>/dev/null | grep -cE 'kyverno\.io|wgpolicyk8s\.io' || true) (expect 22 for kyverno 3.9.x)"
+
+  local base owner filter s3 acme
+  base="$(cvar BASE_DOMAIN)"; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  filter="$(cvar EXTERNALDNS_DOMAIN_FILTER)"; s3="$(cvar S3_BACKUP_ENDPOINT)"; acme="$(cvar ACME_CLUSTER_ISSUER)"
+  say "cluster-vars guards"
+  info "BASE_DOMAIN                = ${base}"
+  info "EXTERNALDNS_TXT_OWNER_ID   = ${owner}"
+  info "EXTERNALDNS_DOMAIN_FILTER  = ${filter}"
+  info "ACME_CLUSTER_ISSUER        = ${acme}"
+  info "S3_BACKUP_ENDPOINT         = ${s3}"
+
+  # THE GUARD THAT MATTERS. Two clusters sharing one Cloudflare zone are isolated by txtOwnerId and
+  # by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the
+  # parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the
+  # single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records.
+  # preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the
+  # target guard, because that one means you are pointed at the wrong cluster. Everything else is
+  # reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting
+  # on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health
+  # report behind a condition that was, for dev, simply the long-standing status quo.
+  if [ -z "${owner}" ]; then
+    warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'.
+       Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse."
+  elif [ "${owner}" = "unset-see-cluster-vars" ]; then
+    warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply.
+       'externaldns' will refuse until a real per-environment value is set."
+  elif [ "${owner}" = "default" ]; then
+    # NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records
+    # are registered under it and cannot be renamed without orphaning them. Dying on it would make
+    # this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in
+    # a script whose whole claim is that it has none. preflight only READS, so it warns; the
+    # MUTATING path (step_externaldns) still refuses outright.
+    warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value.
+       That is correct for DEV (its records are registered under it) and WRONG for any second cluster
+       sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the
+       2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and
+       policy: sync deleted the records behind them."
+  else
+    info "owner id is distinct and non-default: OK"
+  fi
+
+  if s3_is_placeholder "${s3}"; then
+    warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
+       velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
+       so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
+       apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
+  else
+    info "S3 backup endpoint set: OK"
+  fi
+
+  # COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
+  # any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
+  # *and* CDI StorageProfiles in the same pass — but the StorageProfile CRD does not exist until the
+  # cdi-operator has reconciled that CR. On an established cluster the CRDs are long since there and
+  # this is invisible; on a COLD one it is a deadlock, and it is the same class of bug as Kyverno's
+  # 5.7 MiB CRD bundle. Report it here so `resume` does not burn 15 minutes discovering it.
+  say "CRDs that later tiers apply resources against"
+  local crd missing=0
+  for crd in storageprofiles.cdi.kubevirt.io cdis.cdi.kubevirt.io kubevirts.kubevirt.io \
+             volumesnapshotclasses.snapshot.storage.k8s.io; do
+    if kubectl get crd "${crd}" >/dev/null 2>&1; then
+      printf '    %-52s present\n' "${crd}"
+    else
+      printf '    \033[33m%-52s MISSING\033[0m\n' "${crd}"; missing=1
+    fi
+  done
+  if [ "${missing}" = "1" ]; then
+    warn "a MISSING crd above will fail platform-config's dry-run and block every tier below it.
+       CDI's own CRDs (StorageProfile, DataVolume, ...) are created by cdi-operator only AFTER it
+       reconciles the CDI CR. Check that the CDI CR is applied and cdi-operator is Available; if the
+       CRD appeared only recently this can also be a stale discovery cache in kustomize-controller,
+       which a forced reconcile clears."
+    kubectl -n cdi get cdi,deploy 2>/dev/null | head -12 || info "(no cdi namespace yet)"
+  fi
+
+  say "tiers"
+  local n
+  for n in flux-system cilium "${TIER_ORDER[@]}" velero; do
+    printf '    %-16s suspend=%-6s ready=%-6s %s\n' "$n" "$(k_susp "$n")" "$(k_ready "$n")" "$(k_msg "$n" | head -c 90)"
+  done
+
+  say "HelmReleases not Ready"
+  kubectl get helmrelease -A \
+    -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message \
+    --no-headers 2>/dev/null | grep -v ' True ' || info "(all Ready)"
+
+  # SUSPENDED releases are invisible above: a suspended HelmRelease keeps whatever Ready condition it
+  # last had, so a STALE `Ready=True` hides the fact that Flux is not managing it at all. external-dns
+  # is deliberately in this state until `standup.sh externaldns` runs.
+  say "HelmReleases SUSPENDED (Flux is not reconciling these — a stale Ready=True means nothing)"
+  kubectl get helmrelease -A \
+    -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend --no-headers 2>/dev/null \
+    | awk '$3=="true"' || true
+}
+
+step_crds() {
+  say "CRD PRE-APPLY (idempotent)"
+  # Charts whose CRD bundle exceeds kubectl's 256 KiB annotation limit set crds.install=false, so
+  # nothing installs them and `platform` can never go Ready. See bootstrap/flux/preapply-crds.sh.
+  bash "${script_dir}/flux/preapply-crds.sh"
+
+  # ---------------------------------------------------------------- the CDI cold-start deadlock
+  # SAME CLASS OF BUG, DIFFERENT OPERATOR. `platform-config` applies the CDI CR *and*
+  # platform/kubevirt/cdi-storageprofiles.yaml in ONE pass. But StorageProfile's CRD does not exist
+  # until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before
+  # applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD
+  # is never created, and platform-config can never converge. A true deadlock, invisible on an
+  # established cluster (the CRDs are long since there) and fatal on a cold one.
+  #
+  # Breaking it needs exactly one thing: get the CDI CR in. Then cdi-operator creates the remaining
+  # CDI CRDs and platform-config's next reconcile dry-runs clean. We apply ONLY the CR — the minimum
+  # — and leave every other object to Flux. The CR carries no ${} tokens, so no substitution is
+  # needed and applying it here cannot diverge from what Flux would render.
+  local cdi_cr="${repo_root}/platform/kubevirt/cdi-cr.yaml"
+  if kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1; then
+    info "storageprofiles CRD already present — CDI is past its cold start"
+  elif [ ! -f "${cdi_cr}" ]; then
+    warn "expected ${cdi_cr} and it is missing — skipping the CDI unblock"
+  else
+    say "CDI cold-start unblock: applying the CDI CR so cdi-operator can create its CRDs"
+    kubectl get ns cdi >/dev/null 2>&1 || die "namespace 'cdi' absent — the platform tier has not applied the CDI operator yet. Run 'resume' far enough to make platform Ready first."
+    kubectl apply --server-side --force-conflicts -f "${cdi_cr}"
+    local i=0
+    while [ "$i" -lt 300 ]; do
+      kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 && break
+      sleep 10; i=$((i+10))
+    done
+    kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 \
+      || die "cdi-operator did not create storageprofiles.cdi.kubevirt.io within ${i}s.
+       Inspect: kubectl -n cdi get cdi,pods,deploy — platform-config stays blocked until it exists."
+    kubectl wait --for=condition=Established --timeout=60s crd/storageprofiles.cdi.kubevirt.io >/dev/null
+    info "storageprofiles.cdi.kubevirt.io created and Established (${i}s)"
+  fi
+}
+
+step_unwedge() {
+  say "UNWEDGE STUCK HELM RELEASES"
+  # A Helm release left in uninstalling/pending-* cannot be advanced by helm-controller: it reports
+  # StateError ("unable to determine state for release with status 'uninstalling'") forever, which
+  # keeps `platform` (wait: true) not-Ready and every tier below it blocked.
+  #
+  # How prod got here: Kyverno's install timed out because its pods were crash-looping with no CRDs;
+  # install.remediation.retries:3 then fired an uninstall, and that uninstall timed out too.
+  #
+  # Pre-applied CRDs live OUTSIDE the Helm release and SURVIVE this — that is precisely why the
+  # pre-apply is a separate step and must run before this one.
+  local found=0 ns rel st
+  while read -r ns rel st; do
+    [ -n "${ns:-}" ] || continue
+    found=1
+    warn "${ns}/${rel} is '${st}'"
+    [ "${CONFIRM_DESTRUCTIVE}" = "1" ] || {
+      info "DRY RUN. This DELETES the release's Kubernetes objects and lets Flux install cold."
+      info "Re-run with --confirm-destructive to act."
+      continue
+    }
+    say "clearing ${ns}/${rel}"
+    # --no-hooks and --wait=false: the WAIT is what hung the original remediation, so do not repeat it.
+    helm uninstall "${rel}" -n "${ns}" --no-hooks --wait=false || warn "helm uninstall returned non-zero; continuing to check state"
+    if helm list -n "${ns}" --all -q 2>/dev/null | grep -qx "${rel}"; then
+      warn "${ns}/${rel} still present after uninstall — inspect manually, NOT deleting release secrets by hand."
+    else
+      info "${ns}/${rel} cleared"
+      flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile of ${ns}/${rel} did not complete in 10m"
+    fi
+  done < <(
+    for ns in $(kubectl get helmrelease -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do
+      helm list -n "${ns}" --all -o json 2>/dev/null \
+        | yq -r '.[] | select(.status | test("uninstalling|pending-install|pending-upgrade|pending-rollback")) | "'"${ns}"' " + .name + " " + .status' 2>/dev/null || true
+    done
+  )
+  [ "${found}" = "1" ] || info "no wedged releases found."
+}
+
+step_externaldns() {
+  say "EXTERNAL-DNS — SAFE UPGRADE (prove the owner id BEFORE the pod can run)"
+  # THE HAZARD: a Helm upgrade sets replicas back to the chart default (1) and applies the new args in
+  # ONE operation. There is no window in which to check the result first. If --txt-owner-id is missing
+  # the pod starts as owner `default`, claims the other cluster's TXT registry entries, and under
+  # `policy: sync` DELETES the records behind them. That is the 2026-09-04 incident.
+  #
+  # THE FIX IS TO VERIFY IN ADVANCE, NOT AFTER: render the chart with the LIVE HelmRelease values and
+  # assert the flag appears. Rendering is deterministic and touches nothing, so this converts a race
+  # into a precondition.
+  local ns=external-dns rel=external-dns
+  local owner chart ver repo vals
+  owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  [ -n "${owner}" ] && [ "${owner}" != "default" ] && [ "${owner}" != "unset-see-cluster-vars" ] \
+    || die "refusing: EXTERNALDNS_TXT_OWNER_ID is '${owner}'"
+
+  chart="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.chart}')"
+  ver="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.version}')"
+  repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' platform/externaldns/helmrepository.yaml | tr -d '"')"
+  [ -n "${repo}" ] || die "could not derive the external-dns chart repo url"
+
+  vals="${SNAP_DIR}/ed-values.yaml"
+  kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.values}' | yq -P > "${vals}"
+  info "chart=${chart} version=${ver} repo=${repo}"
+  info "live spec.values.txtOwnerId = $(yq '.txtOwnerId' "${vals}")"
+
+  say "rendering the chart with the LIVE values to prove the flag"
+  local rendered="${SNAP_DIR}/ed-rendered.yaml"
+  helm template "${rel}" "${chart}" --repo "${repo}" --version "${ver}" -n "${ns}" -f "${vals}" > "${rendered}"
+  local args; args="$(yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].args[]' "${rendered}")"
+  printf '%s\n' "${args}" | sed 's/^/      /'
+  printf '%s\n' "${args}" | grep -qx -- "--txt-owner-id=${owner}" \
+    || die "the render does NOT contain --txt-owner-id=${owner}.
+       An upgrade would start external-dns as owner 'default' and delete the other cluster's records.
+       REFUSING. Fix the chart values key before retrying."
+  info "PROVEN: the upgrade will carry --txt-owner-id=${owner}"
+
+  say "snapshotting the Cloudflare zone (baseline for the post-check)"
+  dns_snapshot "${SNAP_DIR}/dns-before.txt"
+
+  say "resuming + forcing the upgrade"
+  kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.suspend}' | grep -q true \
+    && flux resume helmrelease "${rel}" -n "${ns}" || info "(HelmRelease was not suspended)"
+  flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile did not complete in 10m"
+
+  say "verifying the LIVE deployment args"
+  local live; live="$(kubectl -n "${ns}" get deploy "${rel}" -o jsonpath='{.spec.template.spec.containers[0].args}' 2>/dev/null || true)"
+  printf '    %s\n' "${live}"
+  printf '%s' "${live}" | grep -q -- "--txt-owner-id=${owner}" \
+    || { kubectl -n "${ns}" scale deploy "${rel}" --replicas=0 || true
+         die "live args LACK --txt-owner-id=${owner}. Scaled to 0. Investigate before scaling up."; }
+  info "live deployment carries --txt-owner-id=${owner}"
+
+  say "post-check: did anything in the shared zone change?"
+  sleep 45   # external-dns --interval=1m; give it a full cycle to act before judging it safe
+  dns_snapshot "${SNAP_DIR}/dns-after.txt"
+  dns_diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-after.txt" \
+    || warn "records changed — expected ONLY additions under this cluster's own BASE_DOMAIN ($(cvar BASE_DOMAIN)).
+       Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION."
+}
+
+step_s3check() {
+  say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
+  # Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
+  # the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
+  # egress differ — and a healthy host-side reading has already misled this project once.
+  #
+  # WHAT THIS PROVES (and it fails closed on every one of them):
+  #   1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
+  #      (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
+  #   2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
+  #   3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
+  #      write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
+  #   4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
+  #
+  # WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
+  # against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
+  local ep live region
+  live="$(cvar S3_BACKUP_ENDPOINT)"
+  info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
+  if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
+    # Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
+    # backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
+    # the first evidence is silently-missing backups.
+    ep="${S3_ENDPOINT_OVERRIDE}"
+    info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
+  else
+    ep="${live}"
+  fi
+  if s3_is_placeholder "${ep}"; then
+    die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
+       cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
+       to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
+  fi
+  s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
+  case "${ep}" in
+    http://*|https://*) ;;
+    *) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
+  esac
+
+  # Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
+  # in cluster-vars silently builds a BSL with no region.
+  region="$(cvar S3_BACKUP_REGION)"
+  case "${ep}" in
+    *.r2.cloudflarestorage.com*)
+      if [ "${region}" != "auto" ]; then
+        warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
+       exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
+       velero, or the BackupStorageLocation is built with the wrong region."
+      fi
+      region=auto ;;
+    *)
+      if [ -z "${region}" ]; then
+        region="us-east-1"
+        warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
+       ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
+      fi ;;
+  esac
+  s3_safe_token S3_BACKUP_REGION "${region}"
+
+  # Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
+  # whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
+  local vbucket vprefix cpath cbucket cprefix rest
+  vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
+             platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
+  [ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
+    || die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
+  vprefix="$(cvar VELERO_BACKUP_PREFIX)"
+
+  cpath="$(cvar CNPG_BACKUP_PATH)"
+  if [ -z "${cpath}" ]; then
+    warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
+       substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
+       default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
+    cpath="s3://cnpg"
+  fi
+  case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
+  rest="${cpath#s3://}"; cbucket="${rest%%/*}"
+  if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
+  cprefix="${cprefix%/}"
+  [ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
+
+  s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
+  s3_safe_token cnpg-bucket   "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
+
+  say "what is about to be proven"
+  info "endpoint    ${ep}"
+  info "region      ${region}"
+  info "velero      bucket '${vbucket}'  prefix '${vprefix:-<bucket root>}'   (authored bucket + VELERO_BACKUP_PREFIX)"
+  info "cnpg        bucket '${cbucket}'  prefix '${cprefix:-<bucket root>}'   (from CNPG_BACKUP_PATH)"
+  info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
+
+  local ran=0 keys
+  # (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
+  if kubectl get ns velero >/dev/null 2>&1 \
+     && kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
+    # List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
+    # into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
+    # cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
+    # silently looked like "the key is missing".
+    keys="$(kubectl -n velero get secret velero-cloud-credentials \
+            -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
+    printf '%s\n' "${keys}" | grep -qx 'cloud' \
+      || die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
+       velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
+       at RUNTIME while the HelmRelease reports Ready."
+    s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
+    ran=$((ran+1))
+  else
+    warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
+       be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
+  fi
+
+  # (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
+  #     of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
+  #     ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
+  local pick cns csec
+  pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
+          | grep -- '-db-backup-credentials$' | head -n 1 || true)"
+  if [ -n "${pick}" ]; then
+    cns="${pick%%/*}"; csec="${pick#*/}"
+    s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
+    ran=$((ran+1))
+  else
+    warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
+       NOT be proven."
+  fi
+
+  [ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
+       proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
+       nothing to check'."
+
+  say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
+  info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
+  info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
+}
+
+step_resume() {
+  say "RESUME TIERS (dependency order)"
+  local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)"
+  for n in "${TIER_ORDER[@]}"; do
+    if [ "$(k_susp "$n")" = "true" ]; then
+      info "resuming $n"; flux resume kustomization "$n" --timeout=1s >/dev/null 2>&1 || true
+    else
+      info "$n already unsuspended"
+    fi
+    wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
+  done
+
+  # The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
+  # was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
+  # which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
+  # RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
+  # to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
+  if s3_is_placeholder "${s3}"; then
+    warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
+       Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
+       that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
+       it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
+  else
+    info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
+    wait_ready velero 900 || warn "velero did not converge"
+  fi
+}
+
+step_verify() {
+  say "VERIFY"
+  local base; base="$(cvar BASE_DOMAIN)"
+  info "BASE_DOMAIN = ${base}"
+  say "kustomizations"
+  kubectl get kustomization -A \
+    -o custom-columns=NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status --no-headers
+  say "certificates (must be Ready before any hostname is trusted)"
+  kubectl get certificate -A --no-headers 2>/dev/null || info "(none yet)"
+  say "pods not Running/Completed"
+  kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' || true
+  info "total pods: $(kubectl get pods -A --no-headers | wc -l)"
+  say "gateway / LoadBalancer VIPs"
+  kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true
+  say "published DNS for this cluster's own domain"
+  local owner; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  dns_snapshot "${SNAP_DIR}/dns-verify.txt"
+  info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)"
+
+  # Every TXT registry entry this cluster publishes must carry OUR owner id. One reading another
+  # cluster's id means we have claimed its records and `policy: sync` will delete them next cycle.
+  local foreign
+  foreign="$(grep "${base}" "${SNAP_DIR}/dns-verify.txt" | grep 'heritage=external-dns' \
+             | grep -v "external-dns/owner=${owner}" || true)"
+  if [ -n "${foreign}" ]; then
+    warn "TXT entries under ${base} NOT owned by '${owner}' — this cluster has claimed records it does not own:"
+    printf '%s\n' "${foreign}" | sed 's/^/      /'
+  else
+    info "every TXT registry entry under ${base} reads owner=${owner}: OK"
+  fi
+
+  # THE REGRESSION CHECK. Additions under our own BASE_DOMAIN are expected. ANY change to a name
+  # outside it means we have touched the other cluster's zone — the 2026-09-04 incident.
+  if [ -f "${SNAP_DIR}/dns-before.txt" ]; then
+    say "regression check against the pre-standup baseline"
+    local outside
+    outside="$(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" \
+               | grep -E '^[<>]' | grep -v "${base}" || true)"
+    if [ -n "${outside}" ]; then
+      warn "RECORDS OUTSIDE ${base} CHANGED — investigate before doing anything else:"
+      printf '%s\n' "${outside}" | sed 's/^/      /'
+    else
+      info "no record outside ${base} was added, changed or removed: OK"
+      info "added under ${base}: $(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" | grep -c '^>' || true)"
+    fi
+  else
+    warn "no ${SNAP_DIR}/dns-before.txt baseline — cannot prove the other cluster's records are untouched.
+       Take one BEFORE the next standup so this check can run."
+  fi
+}
+
+# ============================================================================ dispatch
+for s in "${STEPS[@]}"; do
+  case "$s" in
+    preflight)   step_preflight ;;
+    s3check)     step_s3check ;;
+    crds)        step_crds ;;
+    unwedge)     step_unwedge ;;
+    externaldns) step_externaldns ;;
+    resume)      step_resume ;;
+    verify)      step_verify ;;
+    all)         step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;;
+    *)           die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;;
+  esac
+done
+
+say "done: ${STEPS[*]}"

catalog/gitea.yaml

@@ -24,9 +24,9 @@ env:
         name: gitea-bootstrap
         key: secret-key
   - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
           # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
           # the moment its OIDC guard was fixed and the block actually ran for the first time.
           exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -178,7 +178,7 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)

catalog/jackett.yaml

@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
   image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -25,7 +25,7 @@ storage:
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/keycloak.yaml

@@ -18,7 +18,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -14,7 +14,7 @@ storage:
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)

catalog/plex.yaml

@@ -72,7 +72,7 @@ storage:
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -40,7 +40,7 @@ storage:
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/radarr.yaml

@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

catalog/render.py

@@ -44,7 +44,15 @@ except ImportError:  # pragma: no cover - explicit, actionable failure
 
 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label

catalog/seerr.yaml

@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/sonarr.yaml

@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

ci/touched-requirements.sh

@@ -183,6 +183,15 @@ path_requirements() {
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
+    # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
+    # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
+    # enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
+    # already-mapped path scoped the L0 run to that other path's requirements and silently
+    # excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
+    # precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
+    # stopped matching when the value it guards changed, in a different file) is exactly the
+    # class that omission would hide.
+    bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
     .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
     *) : ;;
   esac

clusters/README.md

@@ -97,6 +97,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 
+### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
+
+The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
+controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
+and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
+reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
+before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
+**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
+`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
+have existed for months); it cost 12h on prod's cold start.
+
+So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
+platform-config`), and a rule:
+
+> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
+> **controller at runtime** rather than by a manifest in git.
+
+`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
+`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
+profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
+
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own

clusters/base/apps-vms.yaml

@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
+# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
+# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
+# adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
   wait: false
   dependsOn:
     - name: platform-config
+    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
+    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
+    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
+    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
+    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
+    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
+    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
+    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
+    #
+    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
+    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
+    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
+    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
+    #
+    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
+    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
+    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
+    #
+    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
+    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
+    # cannot silently drop this tier's KubeVirt/StorageClass gate.
+    - name: platform-config-cdi
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/cluster-vars.yaml

@@ -17,9 +17,63 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # subdomain, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
+  #   prod  -> prod.home.keithjackson.dev
+  #
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
+  BASE_DOMAIN: "home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  #
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
+  # — it can never delete another environment's records — and the fallback is to set prod's value
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -47,10 +101,37 @@ data:
   # ship the literal string into the cluster.
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
+  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
+  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
+  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
+  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
+  # what kube-vip was stood up to provide.
+  #
+  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
+  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
+  #
+  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
+  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
+  # move. Read it as "the API host Cilium dials", which is what it always meant.
+  #
+  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
+  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
+  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
+  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
+  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
+  # block at the top of group_vars/all.yml.
+  #
+  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
+  # in both envs, so it is NOT a knob.
+  #
+  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
+  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
+  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
+  K3S_API_HOST: "192.168.1.49"
 
   # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
   # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -66,14 +147,43 @@ data:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
 
-  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
+  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
-  # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
+  # Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
-  # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
+  # backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
-  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
+  #
-  S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
+  # OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
+  # being until a better on-prem backup solution is created". Two reasons beyond cost:
+  #   1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
+  #      pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
+  #   2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
+  #      physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
+  #      genuinely off-site.
+  # Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
+  # 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
+  # stay free and R-STATE-03 has no cost disincentive to stay proven.
+  #
+  # The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
+  # means pointing this knob back at http://192.168.1.9:9000 temporarily.
+  S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
+
+  # S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
+  # This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
+  # MinIO and wrong for R2, which is why it is a knob now.
+  S3_BACKUP_REGION: "auto"
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
@@ -113,6 +223,39 @@ data:
   VELERO_BACKUP_SCHEDULE: "0 3 * * *"        # cron; overlays may differ
   VELERO_RETENTION: "168h"                   # dev short | prod long
 
+  # BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
+  # other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
+  #
+  # Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
+  #
+  #   VELERO  — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
+  #             `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
+  #             itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
+  #             therefore adopt dev's backups and delete them on schedule.
+  #   CNPG    — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
+  #             CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
+  #             so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
+  #             interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
+  #             dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
+  #
+  # Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
+  # worth explaining, because the earlier reasoning was the opposite:
+  #
+  #   While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
+  #   backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
+  #   restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
+  #
+  #   Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
+  #   reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
+  #   free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
+  #   colliding with a real environment's backups.
+  #
+  # *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
+  # the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
+  # one change; do not split the commit.
+  VELERO_BACKUP_PREFIX: "dev"                # dev: "dev" | prod: "prod"
+  CNPG_BACKUP_PATH: "s3://cnpg/dev"          # dev: s3://cnpg/dev | prod: s3://cnpg/prod
+
   # --- placement ---
   PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os"   # base no-op selector; prod may pin roles
   PLATFORM_NODE_SELECTOR_VALUE: "linux"

clusters/base/kustomization.yaml

@@ -14,6 +14,15 @@ resources:
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
+  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
+  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
+  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
+  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
+  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
+  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
+  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
+  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
   # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
   # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.

clusters/base/platform-config-cdi.yaml

@@ -0,0 +1,135 @@
+---
+# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
+# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
+#
+# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
+#
+# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
+# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
+# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
+# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
+# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
+# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
+# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
+#
+# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
+# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
+# graph order itself, so a cold cluster needs no human in the loop.
+#
+# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
+# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
+# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
+# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
+# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
+# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
+# kind. One edge, no pre-apply.
+#
+# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
+# and everything else waits on: putting CDI's health on that critical path is the mistake the
+# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
+# StorageProfiles that were in the wrong pass.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config-cdi
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
+  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
+  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
+  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
+  # cache is one retryInterval. Keep it short.
+  retryInterval: 2m
+  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
+  # image to pull and no controller to become Available, so a slow apply here means a real error, not
+  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
+  timeout: 2m
+  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
+  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
+  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
+  # second path also declares. See the transition note at the bottom of this file.
+  prune: true
+  # TRUE, but understand exactly what it buys and what it does NOT.
+  #
+  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
+  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
+  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
+  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
+  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
+  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
+  # `apps-vms` waits on it.
+  #
+  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
+  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
+  # storageprofile-controller computes that status from our spec override plus inference. So Ready
+  # here means "the spec override is accepted by the API server", not "CDI has folded it into
+  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
+  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
+  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
+  wait: true
+  dependsOn:
+    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
+    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
+    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
+    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config-cdi
+  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
+  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
+  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
+  # clusters/dev or clusters/prod for this tier.
+  #
+  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
+  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
+  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
+  #      whose Kustomization declares substituteFrom. Omitting it would move
+  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
+  #      could ship as a literal string, precisely the class of bug that check exists to stop.
+  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
+  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
+  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
+  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
+  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
+  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
+  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
+  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
+  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
+  #
+  # Why that is acceptable here — and it is NOT the platform/cilium case:
+  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
+  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
+  #     cluster-scoped CDI CRs.
+  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
+  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
+  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
+  #     absent".
+  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
+  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
+  #     is disturbed and no data path is touched.
+  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
+  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
+  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
+  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
+  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
+  #     could.
+  #
+  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
+  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
+  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
+  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
+  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
+  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
+  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
+  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
+  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
+  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
+  # `prune: true` in a second commit.

clusters/dev/kustomization.yaml

@@ -148,6 +148,33 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
+      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
+      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
+      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
+      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
+      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
+      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
+      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
+      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
+      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
+      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
+      # registry under owner `default` (the external-dns chart default, which is what both clusters
+      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
+      # BY this string: any other value and it stops recognising every record it manages, then
+      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
+      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "default"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -160,11 +187,25 @@ patches:
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
-      # k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
+      # k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
-      # (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
+      # the address kube-vip floats, NOT the primary node .50.
+      # [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
+      # (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
+      #
+      # *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
+      # Every other value in this block is what dev is running right now. This one is not:
+      # the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
+      # DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
+      # AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
+      # in parity for exactly this reason). Do not push this to dev-live casually, and not
+      # while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
+      # but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
+      # already (k3s_tls_sans) and dev's own kubeconfig validates against it via
+      # tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
+      # same LAN as the nodes and ARP-announced by kube-vip on every server.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.1.50"
+        value: "192.168.1.49"
       # Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
       # (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
       - op: replace
@@ -181,15 +222,34 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
+      # OFF-CLUSTER BACKUP TARGET — no op here any more, deliberately.
-      # barmanObjectStore and Velero's backupStorageLocation write to.
+      #
-      - op: replace
+      # This used to pin the on-Proxmox MinIO (http://192.168.1.9:9000). As of 2026-09-05 the operator
-        path: /data/S3_BACKUP_ENDPOINT
+      # moved backups to Cloudflare R2 "for the time being until a better on-prem backup solution is
-        value: "http://192.168.1.9:9000"
+      # created", and R2 is a single environment-INDEPENDENT endpoint: both clusters write to the same
+      # buckets and are isolated by VELERO_BACKUP_PREFIX / CNPG_BACKUP_PATH (dev vs prod), not by URL.
+      # Under R-ENV-02 a value that does not differ per environment belongs in base and must NOT be
+      # restated here — two copies of one endpoint is just a rotation waiting to drift.
+      #
+      # *** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts EMPTY, so Velero writes a
+      # fresh kopia repository and CNPG takes new base backups; the 30-day Barman window rebuilds from
+      # zero. The existing MinIO copy is NOT deleted and stays restorable — point S3_BACKUP_ENDPOINT
+      # back at http://192.168.1.9:9000 (and VELERO_BACKUP_PREFIX back to "") to reach it. Do not merge
+      # this while dev's proven restore is the only recovery path you are relying on.
       # dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -115,8 +115,20 @@ patches:
             name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
-  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
   #
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
   # Two independent reasons, both consequences of the split-horizon design:
   #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
   #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
@@ -126,13 +138,46 @@ patches:
   #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
   #      gitea.home.keithjackson.dev to itself and never reach a repository.
   #
-  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
-  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
-  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
   #
   # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
   # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
   # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository
@@ -140,7 +185,9 @@ patches:
         name: flux-system
         namespace: flux-system
       spec:
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   - target:
@@ -154,9 +201,87 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # ==========================================================================================
+      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
+      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
+      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
+      # of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
+      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
+      #
+      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
+      #   prod  -> prod.home.keithjackson.dev    (this knob)
+      #
+      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
+      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
+      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
+      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
+      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      #
+      # THE APEX INVERSION IS DELIBERATELY NOT ON THIS BRANCH. Moving prod to the bare apex (and dev
+      # to dev.home.) is a LIVE MIGRATION of the dev cluster, not a variable change — see
+      # infra/PROD-PROVISIONING.md §10a. It is kept as a single commit on branch
+      # `migration/dns-apex-inversion` (9137bbd) and is applied as step D3 of that runbook, with the
+      # operator present. Everything else on this branch is independent of it, which is the whole
+      # reason it was split out.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "prod.home.keithjackson.dev"
+      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
+      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
+      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
+      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
+      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
+      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
+      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
+      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
+        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
+        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
+        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
+        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
+        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
+      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
+      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -222,14 +347,32 @@ patches:
       - op: replace
         path: /data/CILIUM_LB_POOL_STOP
         value: "192.168.2.30"
-      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
-      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
+      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
-      # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
+      #
-      # platform/cilium/helmrelease.yaml. Not yet decided.
+      # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
+      # read "Not yet decided" while shipping .10 — the founder's own address — and that is what
+      # the LIVE prod cluster is running: all three cilium-agents carry
+      # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
+      # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
+      # the datapath keeps running off Cilium's cache while no service or policy update lands, with
+      # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
+      # control plane is, which is the whole reason kube-vip exists here.)
+      #
+      # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
+      # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
+      # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
+      # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
+      # `--server` join needs a host that is up before any VIP is floated. Two addresses, two
+      # variables, deliberately.
+      #
+      # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
+      # LB-IPAM can never be handed the same IP — asserted by
+      # conformance/tests/l0-static/ha-control-plane-as-code.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.2.10"
+        value: "192.168.2.9"
+      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
       # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
       # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
@@ -240,10 +383,38 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
-      # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
+      # ------------------------------------------------------------------------------------------
-      # network — a prod backup written there is not off-cluster DR and would mix two environments'
+      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05).
-      # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
+      #
+      # THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule
+      # stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev
+      # address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it
+      # is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and
+      # overriding it with an identical value would only create two places to change on a rotation —
+      # the exact drift this repo has been bitten by before. Isolation does NOT come from the
+      # endpoint; it comes from the two path knobs below.
+      #
+      # WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network
+      # 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN
+      # 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod
+      # egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s.
+      # Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and
+      # unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely
+      # off-site. Verify prod can reach the new endpoint before resuming velero:
+      #   bootstrap/standup.sh ... --s3-endpoint <the R2 endpoint> s3check
+      #
+      # THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was
+      # per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster
+      # used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have
+      # keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and
+      # TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman
+      # stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml.
+      # ------------------------------------------------------------------------------------------
       - op: replace
-        path: /data/S3_BACKUP_ENDPOINT
+        path: /data/VELERO_BACKUP_PREFIX
-        value: "TBD-PROD-S3-BACKUP-ENDPOINT"
+        value: "prod"
+      # Barman objects land at s3://cnpg/prod/<clusterName>/; dev's at s3://cnpg/dev/<clusterName>/.
+      - op: replace
+        path: /data/CNPG_BACKUP_PATH
+        value: "s3://cnpg/prod"
       # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.

conformance/incluster/rbac.yaml

@@ -57,6 +57,13 @@ rules:
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
+    # LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
+    # agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
+    # oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
+    # time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
+    # chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
+    # If this rule is ever narrowed to the test's own namespace, those bindings fail and the
+    # tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods/exec]

conformance/lib/status.py

@@ -297,6 +297,109 @@ _EXECUTABLE_RE = re.compile(
 _BLOCKED_RE = re.compile(
     r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
 
+# ---------------------------------------------------------- scaffold-marker detection
+# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
+# ---------------------------------------------------
+# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
+# is placeholder detection has to name the idioms it hunts for:
+# `l0-static/placeholder-guard-coupling/check.sh` carries
+# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
+# A whole-file substring test read that as the test declaring itself unfinished, classified
+# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
+# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
+# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
+#
+# The rule below is derived from the corpus (43 tests), not invented. Every genuine
+# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
+# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
+# two structural positions, and no executable test does either:
+#
+#   1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
+#      repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
+#      `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
+#      `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
+#   2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
+#      bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
+#      message is a note about work that is missing. A marker that is the ENTIRE content of
+#      a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
+#      marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
+#
+# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
+# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
+# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
+# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
+# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
+# carrying any trailing text all classify as scaffold.
+#
+# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
+# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
+# several idioms — is still called a scaffold. That is the conservative direction, and the
+# author's remedy is the same structural discipline the corpus already uses: keep each
+# idiom a standalone literal in a list.
+
+# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
+# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
+# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
+# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
+# `placeholder.*`), and none of them is used here to mark a placeholder body.
+# Case-insensitive: no English word contains either token, and the corpus carries no
+# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
+_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
+
+# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
+# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
+# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
+# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
+# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
+_STATUS_SCAFFOLD_RE = re.compile(
+    r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
+
+# A quoted literal whose entire content is one marker word: the marker as data.
+_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
+
+
+def _split_doc_header(text: str) -> tuple[str, str]:
+    """Split a test file into (documentation header, body).
+
+    Both file kinds in this tree open with a banner comment block that is PROSE about the
+    test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
+    prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
+    placeholder idioms when the test hunts them, so an incidental mention there is
+    documentation, not a placeholder body.
+
+    The body starts at the first line that is neither blank nor a `#` comment:
+      *.sh    -> `set -euo pipefail` (the shebang is a comment)
+      *.yaml  -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
+    Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
+    with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
+    # TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
+    """
+    lines = text.splitlines(keepends=True)
+    for i, ln in enumerate(lines):
+        s = ln.strip()
+        if s and not s.startswith("#"):
+            return "".join(lines[:i]), "".join(lines[i:])
+    return text, ""          # no executable line at all: it is all header, body is empty
+
+
+def scaffold_marker(text: str) -> str:
+    """Why `text` declares itself a placeholder body, or "" if it does not.
+
+    Returns a short human-readable reason (used verbatim in the hygiene report) so the
+    defect is actionable rather than a bare path.
+    """
+    head, body = _split_doc_header(text)
+    m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
+    if m:
+        return m.group(0).strip().lstrip("#").strip()[:80]
+    # Blank out markers held as pure data before looking for a marker that has a message.
+    m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
+    if m2:
+        line = body[:m2.start()].count("\n")
+        snippet = body.splitlines()[line].strip()
+        return f"placeholder marker in body: {snippet[:80]}"
+    return ""
+
 
 def classify_test(t: dict) -> dict:
     """Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
@@ -304,14 +407,18 @@ def classify_test(t: dict) -> dict:
     A tagged test is only worth counting if it can actually assert something. Rules, in
     order, chosen so ambiguity NEVER upgrades a test:
 
-      scaffold             the body contains TODO markers (a placeholder), *even if* it
+      scaffold             the test DECLARES itself a placeholder — a `# STATUS: scaffold`
-                           is annotated executable — an annotation must never outrank the
+                           header line, or a TODO/FIXME marker carrying a message in the
-                           body. That mismatch is also reported as a hygiene defect.
+                           body (see `scaffold_marker` for the rule and the corpus it was
-      executable           annotated `conformance.home.k3s/executable: "true"` and no TODOs.
+                           derived from) — *even if* it is annotated executable. An
-      executable-untagged  no TODOs and a real runnable body, but the annotation is
+                           annotation must never outrank the body. That mismatch is also
-                           missing (true of the L1 render tests, which execute a
+                           reported as a hygiene defect.
-                           render-check.sh + golden diff, of the L0 static checks, and
+      executable           annotated `conformance.home.k3s/executable: "true"` and no
-                           of the META self-check).
+                           scaffold declaration.
+      executable-untagged  no scaffold declaration and a real runnable body, but the
+                           annotation is missing (true of the L1 render tests, which
+                           execute a render-check.sh + golden diff, of the L0 static
+                           checks, and of the META self-check).
                            Counted as executable, and flagged as an annotation gap.
       scaffold             anything else.
     """
@@ -322,16 +429,26 @@ def classify_test(t: dict) -> dict:
     # downgrades the test to a scaffold — an annotation must never outrank the body,
     # whichever file the body lives in.
     extra = ""
+    scripts: list[tuple[str, str]] = []
     for script in ("render-check.sh", "check.sh"):
         sp = p.parent / script
         if sp.is_file():
-            extra += sp.read_text()
+            body = sp.read_text()
+            extra += body
+            scripts.append((script, body))
     declared = bool(_EXECUTABLE_RE.search(text))
-    has_todo = "TODO" in text or "TODO" in extra
+    # Each file is classified on ITS OWN structure — the yaml and the script have different
+    # header blocks, so they must not be concatenated before the header/body split.
+    reason = scaffold_marker(text)
+    for script, body in scripts:
+        if reason:
+            break
+        r = scaffold_marker(body)
+        reason = f"{script}: {r}" if r else ""
     bm = _BLOCKED_RE.search(text)
     has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
 
-    if has_todo:
+    if reason:
         state = "scaffold"
     elif declared:
         state = "executable"
@@ -348,7 +465,8 @@ def classify_test(t: dict) -> dict:
         "name": t["name"], "path": rel, "layer": t["layer"],
         "state": state,
         "declared_executable": declared,
-        "annotation_conflict": bool(declared and has_todo),
+        "annotation_conflict": bool(declared and reason),
+        "scaffold_reason": reason,
         "blocked_by": bm.group(1).strip().strip('"') if bm else "",
     }
 
@@ -640,6 +758,10 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
     # --- hygiene defects this tool owns -------------------------------------------
     hygiene = {
         "annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
+        # Same set, with WHY each one was called a placeholder body, so the fix is obvious
+        # from the report instead of requiring the reader to re-derive the rule.
+        "annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
+                                       for t in classified if t["annotation_conflict"]},
         "executable_untagged": [t["path"] for t in classified
                                 if t["state"] == "executable-untagged"],
         "registry_not_in_spec": [],
@@ -910,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     else:
         o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
                  "this is a skip, not a pass)")
-    o.append(f"- tests annotated executable but containing TODO bodies: "
+    o.append("- tests annotated executable but declaring a placeholder body: "
-             f"{h['annotation_conflicts'] or 'none'}")
+             + (", ".join(f"`{k}` ({v})"
+                          for k, v in sorted(h["annotation_conflict_reasons"].items()))
+                if h["annotation_conflicts"] else "none"))
     o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
              f"{h['executable_untagged'] or 'none'}")
 
@@ -930,9 +1054,12 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
              "reconciled |")
     o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
              "(the authoritative tag, same source as `coverage.md`) | exact |")
-    o.append("| executable vs scaffold | TODO markers in the test body outrank the "
+    o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
-             "`executable` annotation; a real body without the annotation still counts "
+             "TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
-             "and is flagged | conservative — ambiguity never upgrades |")
+             "quoted datum — a placeholder detector's own vocabulary — is not one); "
+             "either outranks the `executable` annotation, and a real body without the "
+             "annotation still counts and is flagged | conservative — ambiguity never "
+             "upgrades |")
     o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
              "every verifying test: JUnit result **plus** that test's own pass sentinel "
              "token grepped from the run log | exact where a run was recorded, refused "
@@ -1127,6 +1254,70 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+        # === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
+        # A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
+        # coupling` — a 32-assertion executable check whose SUBJECT is placeholder
+        # detection, so it must name the idioms it hunts — as a scaffold, and through it
+        # reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
+        # assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
+        # real placeholder body must still be caught. Both halves are load-bearing; the
+        # second is what stops this fix from degrading into "always executable".
+        yml = ("metadata:\n  annotations:\n"
+               "    conformance.home.k3s/executable: \"true\"\n"
+               "spec:\n  steps:\n    - script:\n"
+               "        content: bash ./check.sh\n")
+        f.write_text(yml)
+        detector = td2 / "check.sh"
+        # Shaped exactly like the real check: a prose header that discusses the idioms,
+        # then executable code holding them as a vocabulary list.
+        detector.write_text(
+            "#!/usr/bin/env bash\n"
+            "# L0 static check — placeholder vocabulary\n"
+            "#   * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
+            "# STATUS: EXECUTABLE. A pure offline assertion script.\n"
+            "set -euo pipefail\n"
+            "python3 - <<'EOF'\n"
+            "SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
+            "assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
+            "EOF\n")
+        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
+        check("a marker in HEADER PROSE does not downgrade an executable check",
+              c["state"] == "executable" and not c["annotation_conflict"])
+        check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
+              c["scaffold_reason"] == "")
+
+        # ... and the protection is intact: a real placeholder body is still caught, in
+        # each of the three shapes the corpus actually uses.
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "echo \"TODO: assert the PVC is bound\"\n")
+        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
+        check("a TODO carrying a message in check.sh IS still a scaffold",
+              c["state"] == "scaffold" and c["annotation_conflict"])
+        check("the scaffold reason names the file and the offending line",
+              c["scaffold_reason"].startswith("check.sh:")
+              and "TODO: assert the PVC is bound" in c["scaffold_reason"])
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
+        check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
+              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
+              == "scaffold")
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "echo \"FIXME the extraction before trusting this\"\n")
+        check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
+              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
+              == "scaffold")
+        # The dedicated declaration slot outranks a body that would otherwise look real —
+        # this is how all three genuine scaffolds in the tree declare themselves.
+        detector.unlink()
+        f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
+                     "# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
+        c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
+        check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
+              c["state"] == "scaffold" and c["annotation_conflict"])
+        check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
+              scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
+                              "assertion script; covers no scaffold.\nset -e\n") == "")
+
     # ---------------------------------------------------------------- PROOF / ledger
     # These are the assertions that stand between this report and a fake green.
     check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
@@ -1341,8 +1532,11 @@ def main() -> int:
                              f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
             rc = 1
         if h["annotation_conflicts"]:
-            sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
+            sys.stderr.write(
-                             f"bodies: {h['annotation_conflicts']}\n")
+                "[status] STRICT: tests annotated executable that declare a placeholder "
+                "body: " + "; ".join(f"{k} ({v})" for k, v in
+                                     sorted(h["annotation_conflict_reasons"].items()))
+                + "\n")
             rc = 1
         if rep["safety_critical_without_executable_l3"]:
             sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "

conformance/tests/README.md

@@ -93,9 +93,52 @@ and coverage extraction exact and robust.
   break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
 - **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
   the PM — do not relax the assertion or the requirement.
+- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
+  Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
+  so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
+  binding over the same ConfigMap Flux substitutes from:
+
+  ```yaml
+  spec:
+    bindings:
+      - name: baseDomain
+        value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
+  ```
+
+  Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
+  `($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
+  full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
+  which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
+  over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
+  the Service was assigned) — it is both stricter and environment-independent. Never take the
+  environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
+  ends up silently asserting the other cluster's expectations. L1 goldens are the one
+  exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
+
+### How a scaffold declares itself (and how `status.py` tells one from a verifier)
 
 Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
-later roadmap phase delivers the capability under test. `lib/status.py` treats such a
+later roadmap phase delivers the capability under test. `lib/status.py` classifies each
-marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
+test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
-outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
+`render-check.sh`, `check.sh` — and either signal below **outranks**
-can never be reported as an executable test.
+`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
+an executable test. Declare a scaffold in **both** positions:
+
+1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
+   declared state must be the first token after `STATUS:` (executable tests say
+   `# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
+2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
+   non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
+   or an inline comment on a missing fixture:
+   `file: fixtures/smtp-sink.yaml   # TODO(T-4.1): hermetic SMTP capture sink`.
+
+What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
+mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
+NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
+placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
+`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
+classify that 32-assertion check as a scaffold and, through it, report the safety-critical
+R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
+idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
+alternation, several idioms in one string) still classifies as a scaffold, because
+ambiguity here only ever downgrades.

conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml

@@ -0,0 +1,106 @@
+# =============================================================================
+# L0 STATIC — the five auditability proofs can be produced on demand  (R-SEC-03)  [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
+#          operator can prove isolation, egress path, cert validity, and backup integrity)"),
+#          made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
+#          must be able to prove": Isolation (denied-probe test), Egress path (egress IP
+#          check), Kill-switch (fault-injection test), TLS (cert status + external check),
+#          Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
+#          one-time checks", with the acceptance gate "The five auditability proofs above can
+#          be produced on demand."
+#
+# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
+# -----------------------------------------------------------------------------------
+# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
+# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
+# failure modes are asserted instead, each by DRIVING the real machinery:
+#   (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
+#       the spec's own words require (fault injection / a DR drill => L3), taken from
+#       lib/status.py's own verdict so this check cannot disagree with the status report;
+#   (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
+#       must resolve to real test dirs AND pass the merge gate's coverage assertion;
+#   (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
+#       attribute — a proof that passes but records nothing leaves the operator with a green
+#       terminal and no evidence;
+#   (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
+#       real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
+#       token, and a ledger row claiming `proven` at the repository's ROOT commit — and
+#       requires record_run.py to report not-proven and status.py to report `stale-proof`. It
+#       also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
+#       ci/ or run.sh ran before this test existed.
+# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
+# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
+# so a sixth proof added to the spec fails here instead of being silently unverified.
+#
+# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
+# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
+# anyone needs it.
+#
+# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
+# WHEN  lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
+#       lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
+#       and two synthetic forgeries are presented to them,
+# THEN  each of docs/07's five proofs maps to an executable test at the required layer, the
+#       documented one-command invocation selects and asserts them, each is capturable as a
+#       ledger row, and neither a self-skip nor a stale row can be recorded as proof.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
+# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
+# it directly, and this Chainsaw Test wraps the SAME script.
+#
+# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
+# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
+# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
+# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
+# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
+# assertion here was NOT relaxed to accommodate them (R-TEST-06).
+#
+# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
+#   * That the five proofs PASS. This asserts they are present, strong, runnable by one
+#     command, capturable, and that the chain refuses forgeries — not that isolation holds
+#     today (R-NET-01's test) or that the restore works (R-STATE-03's).
+#   * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
+#     suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
+#   * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
+#     would need signed history.
+#   * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
+#     requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
+#     not re-asserted here.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: auditability-proofs-producible
+  labels:
+    conformance.home.k3s/R-SEC-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-SEC-03"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree and the harness's own tooling, when status.py, the requirement
+      resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
+      tools' self-tests are driven and two synthetic forgeries are presented to them, then each
+      of docs/07's five auditability proofs has an executable test at the layer the spec
+      requires, is selectable and asserted by the documented one-command invocation, is
+      capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
+      evidence.
+spec:
+  steps:
+    - name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
+        conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh
+# ---------------------------------------------------------------------------------------------
+# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
+# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
+# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
+# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
+# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
+# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
+# the wrong trade (R-TEST-06).

conformance/tests/l0-static/auditability-proofs-producible/check.sh

@@ -0,0 +1,431 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — auditability-proofs-producible
+# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
+#         prove isolation, egress path, cert validity, and backup integrity)."
+#         — spec docs/00-vision-and-scope.md).
+#
+# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
+# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
+#   - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
+#     denied-probe test.
+#   - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
+#   - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
+#   - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
+#     status + external check.
+#   - **Backups**: "irreplaceable data is restorable" — via a DR drill.
+#   "These proofs are part of the platform acceptance suite, not one-time checks."
+# and its acceptance gate: "The five auditability proofs above can be produced on demand."
+#
+# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
+# ------------------------------------------------------------
+# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
+# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
+# would keep passing while every proof rotted into a scaffold. Four separate failure modes
+# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
+# by DRIVING the real machinery rather than by reading files:
+#
+#   (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
+#       requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
+#       where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
+#       executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
+#       check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
+#       repository's own status report.
+#   (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
+#       --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
+#       their coverage (that flag is the merge gate's own path: it passes --require to
+#       coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
+#       gate wiring break, R-SEC-03 fails.
+#   (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
+#       with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
+#       declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
+#       record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
+#       instead of silently making the requirement unprovable.
+#   (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
+#       that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
+#       real tools, in a throwaway directory:
+#         * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
+#           carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
+#           test. record_run.py must refuse to call it `proven`.
+#         * a ledger row claiming `proven` at a commit that PREDATES the requirement's
+#           implementing code — status.py must degrade it to `stale-proof`, never green.
+#       Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
+#       in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
+#       were themselves unexercised in the merge gate until this check.
+#
+#   The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
+#   repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
+#   mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
+#   than quietly ignoring it.
+#
+# WHY L0: every clause above is a property of the repository and its own tooling, provable
+# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
+# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
+# with one.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
+#   - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
+#     documented command, capturable as evidence, and that the evidence chain refuses
+#     forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
+#     restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
+#     proof, which is what is asserted.
+#   - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
+#     suite queries Hubble, and this check does not pretend otherwise: it asserts the
+#     denied-probe half, which is the half that is automated. The Hubble half remains an
+#     operator-interactive capability — a real, named gap in R-SEC-03's coverage.
+#   - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
+#     report's prose, the ledger's provenance fields) is a usability property, not a testable
+#     one.
+#   - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
+#     nothing here can prove a row was never removed from it (that would need signed history).
+#   - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
+#     the five bullets docs/07 enumerates. The five are what the spec made concrete and are
+#     what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
+#     tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
+export SPEC_REPO
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+CONF = ROOT / "conformance"
+LIB = CONF / "lib"
+FAILURES: list[str] = []
+NOTES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def note(msg):
+    NOTES.append(msg)
+
+
+def die():
+    print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+
+
+def run(cmd, **kw) -> subprocess.CompletedProcess:
+    return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
+
+
+# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
+# key  = the bullet label docs/07 uses (verified against the spec below when it is mounted)
+# reqs = the requirement(s) whose verifying test IS that proof
+# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
+#        test" / "a DR drill", an L2 happy-path test does not discharge the proof.
+PROOFS = {
+    "Isolation":   {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
+                    "why": '"service X cannot reach service Y" — via Hubble flow logs + a '
+                           "denied-probe test"},
+    "Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
+                    "why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
+    "Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
+                    "why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
+    "TLS":         {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
+                    "why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
+                           "— via cert status + external check"},
+    "Backups":     {"reqs": ["R-STATE-03"], "layers": {"L3"},
+                    "why": '"irreplaceable data is restorable" — via a DR drill'},
+}
+PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
+
+# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
+spec = Path(os.environ.get("SPEC_REPO", ""))
+sec_doc = spec / "docs" / "07-security-and-secrets.md"
+if sec_doc.is_file():
+    text = sec_doc.read_text()
+    m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
+                  re.M | re.S)
+    check(m is not None,
+          "docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
+          "the proof list this check is coupled to has moved or been removed, so the mapping "
+          "below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
+    if m:
+        labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
+        check(set(labels) == set(PROOFS),
+              f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
+              f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
+              "test, and a proof this check does not know about would be silently unverified")
+        check(len(labels) == 5,
+              f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
+              "gate says 'The five auditability proofs above can be produced on demand' — the "
+              "prose and the list have drifted apart")
+else:
+    note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
+         "re-derived from docs/07, so the mapping below was used as written. Every other "
+         "assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
+
+# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
+# Driven through lib/status.py so this check and the repository's own status report cannot
+# disagree about what is executable and what is a scaffold.
+tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
+try:
+    out = tmp / "status"
+    st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
+              "--registry", str(CONF / "requirements" / "registry.yaml"),
+              "--tests", str(CONF / "tests"), "--out", str(out)])
+    check(st.returncode == 0,
+          f"lib/status.py exited {st.returncode} — the report an operator would use to find "
+          f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
+    status = {}
+    sj = out / "status.json"
+    if sj.is_file():
+        status = json.loads(sj.read_text())
+    check(bool(status.get("by_requirement")),
+          "lib/status.py produced no per-requirement report — the audit posture of each "
+          "requirement is not derivable, so R-SEC-03 cannot be assessed at all")
+    by_req = status.get("by_requirement", {})
+
+    for label, spec_proof in sorted(PROOFS.items()):
+        for rid in spec_proof["reqs"]:
+            rec = by_req.get(rid)
+            check(rec is not None,
+                  f"proof '{label}' maps to {rid}, which the status report does not know — "
+                  "either the registry lost the requirement or this mapping is stale")
+            if not rec:
+                continue
+            tests = (rec.get("verification") or {}).get("tests") or []
+            executable = [t for t in tests if t.get("state", "").startswith("executable")]
+            check(bool(executable),
+                  f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
+                  f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
+                  f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
+                  "operator cannot prove this property")
+            at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
+            check(bool(at_layer),
+                  f"proof '{label}' ({rid}) has no executable test at layer(s) "
+                  f"{sorted(spec_proof['layers'])} — it has "
+                  f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
+                  f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
+                  "weaker layer does not discharge it (a rendering check is not a drill)")
+
+    # ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
+    # The resolver: --requirements must translate the five R-IDs into real test dirs.
+    resolver = run(["bash", "-c",
+                    f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
+                    f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
+    dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
+    check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
+          f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
+          f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
+          "acceptance gate depends on ('produced on demand') cannot select the proofs")
+
+    # The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
+    # zero exit here is the repository's own statement that all five proofs are covered.
+    gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
+    check(gate.returncode == 0,
+          f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
+          f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
+          f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
+
+    # ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
+    # Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
+    # token can pass forever and never be recorded as proof of anything.
+    sys.path.insert(0, str(LIB))
+    os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
+    sys.dont_write_bytecode = True
+    import record_run as rr                                   # noqa: E402
+
+    smap = rr.sentinel_map(CONF / "tests")
+    check(len(smap) >= 10,
+          f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
+          "grip on the test tree, so the assertions below would assert nothing")
+    for label, spec_proof in sorted(PROOFS.items()):
+        for rid in spec_proof["reqs"]:
+            rec = by_req.get(rid) or {}
+            tests = (rec.get("verification") or {}).get("tests") or []
+            names = [t.get("name") for t in tests
+                     if t.get("state", "").startswith("executable")
+                     and t.get("layer") in spec_proof["layers"]]
+            for name in names:
+                ent = smap.get(name)
+                check(ent is not None,
+                      f"proof '{label}': record_run.py does not know the test {name!r} — its "
+                      "result could never be attributed to a ledger row")
+                if not ent:
+                    continue
+                if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
+                    continue          # offline layers are truthful by JUnit alone
+                check(bool(ent.get("pass")),
+                      f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
+                      "so lib/record_run.py can never record it as proven. It would go green "
+                      "in the terminal and leave the operator with no evidence — exactly the "
+                      "gap the ledger exists to close (docs/07: the proofs are 'part of the "
+                      "platform acceptance suite, not one-time checks')")
+                check(bool(ent.get("terminal_pass")),
+                      f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
+                      "record_run.py cannot decide its verdict")
+
+    # =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
+    # Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
+    for tool in ("record_run.py", "status.py"):
+        r = run([sys.executable, str(LIB / tool), "--selftest"])
+        check(r.returncode == 0,
+              f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
+              "self-skipped or stale run from being recorded as proof are broken, so nothing "
+              f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
+
+    # ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
+    # This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
+    forge_target, forge_skip = None, None
+    for label, spec_proof in sorted(PROOFS.items()):
+        for rid in spec_proof["reqs"]:
+            for name, ent in smap.items():
+                if rid in (ent.get("requirements") or []) and ent.get("skip") \
+                        and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
+                    forge_target, forge_skip = name, ent["skip"][0]
+                    break
+            if forge_target:
+                break
+        if forge_target:
+            break
+    check(forge_target is not None,
+          "no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
+          "could not be mounted. That is not a pass: it means this check could not verify "
+          "that a self-skip is refused as proof, which is the whole point of the ledger")
+    if forge_target:
+        run_dir = tmp / "20260101T000000Z-forged"
+        run_dir.mkdir(parents=True)
+        (run_dir / "report.junit.xml").write_text(
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
+            f'<testcase name="{forge_target}" classname="forged"/>'
+            "</testsuite></testsuites>\n")
+        (run_dir / "run.log").write_text(
+            f"=== RUN   chainsaw/{forge_target}\n"
+            f"    === STDOUT\n        {forge_skip}\n")
+        forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
+                      "--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
+                      "--tests", str(CONF / "tests"), "--repo", str(ROOT)])
+        verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
+                              forged.stdout, re.M)
+        check(verdicts and all(v != "proven" for v in verdicts),
+              f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
+              "a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
+              f"({forge_skip}). A self-skip must never become proof — with this hole every "
+              "audit proof in docs/07 could be produced by a cluster that skipped it")
+        check(not (tmp / "never-written.jsonl").exists(),
+              "record_run.py --dry-run WROTE to the ledger — a tool that records while "
+              "claiming not to cannot be trusted to record only real runs")
+
+    # ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
+    # status.py must degrade it to stale-proof (not green), or an old green run would keep
+    # vouching for code it never executed.
+    root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
+    old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
+    check(len(old_sha) == 40,
+          "the repository's root commit could not be resolved, so the stale-proof guard could "
+          "not be exercised (that is a gap in this check, not a pass)")
+    if len(old_sha) == 40 and forge_target:
+        stale_ledger = tmp / "stale.jsonl"
+        rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
+        stale_ledger.write_text(json.dumps({
+            "ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
+            "chainsaw_version": "v0.0.0", "kube_version": "unknown",
+            "layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
+            "requirements": [rid], "junit_result": "passed",
+            "sentinel": (smap[forge_target].get("terminal_pass") or "X"),
+            "proof": "proven", "run_id": "20260101T000000Z-forged",
+            "note": "synthetic row written by the R-SEC-03 conformance check",
+        }) + "\n")
+        out2 = tmp / "status-stale"
+        st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
+                   "--registry", str(CONF / "requirements" / "registry.yaml"),
+                   "--tests", str(CONF / "tests"), "--out", str(out2),
+                   "--ledger", str(stale_ledger)])
+        check(st2.returncode == 0,
+              f"lib/status.py failed against a synthetic ledger: "
+              f"{(st2.stderr or st2.stdout).strip()[:300]}")
+        s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
+        rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
+        per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
+        check(per.get(forge_target) == "stale-proof",
+              f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
+              f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
+              "proof from a run that predates the code it vouches for is not evidence, and "
+              "honouring it would let one old green run certify every later change")
+        check((rec2.get("proof") or {}).get("state") != "proven",
+              f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
+              "freshness guard is not holding")
+
+    # ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
+    ledger = CONF / "runs" / "ledger.jsonl"
+    if ledger.is_file():
+        rows, bad = [], []
+        for i, line in enumerate(ledger.read_text().splitlines(), 1):
+            if not line.strip():
+                continue
+            try:
+                rows.append(json.loads(line))
+            except json.JSONDecodeError:
+                bad.append(i)
+        check(not bad,
+              f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
+              "is not machine-readable, so no proof can be derived from it")
+        for r in rows:
+            for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
+                        "run_id"):
+                check(key in r,
+                      f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
+                      "provenance cannot be audited")
+            if r.get("proof") == "proven":
+                check(bool(r.get("sentinel")),
+                      f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
+                      "the sentinel is the only thing separating a real pass from a self-skip")
+                check(r.get("test") in smap,
+                      f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
+                      "test in the suite any more — a proof of something that no longer exists")
+                check("-dirty" not in str(r.get("git_sha", "")),
+                      f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
+                      f"({r.get('git_sha')}) — what ran is not what is committed")
+    else:
+        note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
+             "were skipped; every other assertion ran, and the forgery probes above use "
+             "synthetic ledgers, not this file")
+finally:
+    shutil.rmtree(tmp, ignore_errors=True)
+
+# --------------------------------------------------------------------------- verdict
+# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
+check(CHECKS >= 30,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    die()
+for n in NOTES:
+    print(f"  note: {n}")
+print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
+      f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
+      f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
+      "and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
+      "attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
+      "self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
+      "R-SEC-03's producibility half only: whether each proof PASSES is its own "
+      "requirement's test, and the Hubble half of the isolation proof is automated nowhere "
+      "(see the header's gap list).")
+PY

conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml

@@ -0,0 +1,75 @@
+# =============================================================================
+# L0 STATIC — dev/test nodes are provisioned as code, end to end  (R-ENV-01)  [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
+#          created, baselined, cluster-joined, and managed as code"); the four stages
+#          are enumerated in docs/02 §"Node lifecycle stages".
+#
+# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
+# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
+# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
+# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
+# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
+# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
+# to any node), not by reading files: a chain that does not resolve fails the check.
+#
+# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
+# The dev group names are read from the playbooks' own default host patterns, the node
+# names are generated from the IaC module's prefix/count variables, and the two sets are
+# compared against each other — so the assertion survives a rename or a resize.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the provisioning chain is parsed and (where offline execution is possible) driven,
+# THEN  the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
+#       provider — a cloud provider fails), from a for_each map rather than a count index,
+#       each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
+#       node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
+#       the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
+#       declaratively; one k3s role founds the cluster with --cluster-init and joins the
+#       rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
+#       install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
+#       resolves and targets every dev node at every stage.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
+# where present — ansible-playbook in no-connection modes; no cluster, no network);
+# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
+# wraps the SAME script.
+#
+# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
+#   * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
+#     hypervisor access, neither available offline.
+#   * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
+#     parsed, not evaluated; a network-allowed CI stage would close this.
+#   * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
+#   * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
+#     tests/l3-fault-injection/node-loss (R-STATE-02).
+#   * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
+#     Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
+#     not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
+#     divergence rather than hiding it.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: dev-nodes-provisioned-as-code
+  labels:
+    conformance.home.k3s/R-ENV-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-01"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree, when the dev provisioning chain is parsed and driven
+      offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
+      and handed to Flux by one code path that covers all of them — with no node outside
+      it and no stage missing.
+spec:
+  steps:
+    - name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
+        conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh

@@ -0,0 +1,442 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
+# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
+#         baselined, cluster-joined, and managed as code." — spec
+#         docs/00-vision-and-scope.md; the four stages are enumerated in
+#         docs/02-environments-and-provisioning.md §"Node lifecycle stages
+#         (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
+#           1. Create   — OpenTofu instantiates the VM
+#           2. Baseline — OS baseline and prerequisites
+#           3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
+#              k3s bundled flannel/traefik/servicelb/kube-proxy
+#           4. Manage   — GitOps-managed by Flux; node config changes are made via IaC
+#              and re-applied, never hand-edited on the box.)
+#
+# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
+# ----------------------------------------------------------------------
+# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
+# cannot answer it: a Kubernetes node object looks identical whether it was built by
+# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
+# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
+# provisioning path exists for all four stages and covers every dev node — and, for the
+# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
+# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
+# does not actually resolve fails here. The IaC half is asserted by parsing the module:
+# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the provisioning chain is parsed and (where offline execution is possible)
+#       actually driven,
+# THEN  CREATE:  the IaC module targets a LOCAL KVM hypervisor (every declared provider
+#                is a KVM-family hypervisor provider — no cloud provider may appear),
+#                instantiates its nodes from a for_each map (never a count index, so one
+#                node can be replaced without disturbing its siblings), and gives each
+#                node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
+#                cloud-init seed carrying the login user + SSH key;
+#       AND      the node set it creates is EXACTLY the dev node set Ansible manages —
+#                names derived from the module's own prefix/count, so no node can be
+#                hand-added to the inventory or hand-created in the hypervisor;
+#       BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
+#                modules persisted AND loaded, sysctls, the Longhorn data disk formatted
+#                and mounted, swap off, iscsid enabled), driven by variables with
+#                defaults rather than per-host special cases;
+#       JOIN:    one role installs k3s for both roles of every node — `--cluster-init`
+#                on the primary, `--server` join on the secondaries — with flannel,
+#                kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
+#                installed as the CNI afterwards;
+#       MANAGE:  site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
+#                `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
+#                `--list-hosts` shows every one of those stages targeting EVERY dev
+#                node, so no node is outside the code path.
+#
+# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
+# The dev group names come from the playbooks' own default host patterns, the node names
+# from the IaC module's prefix/count variables, and the two sets are compared to each
+# other. This check therefore keeps working when the estate is renamed or resized.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
+#   - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
+#     it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
+#     access. `tofu plan` against the live estate is the operator's own check.
+#   - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
+#     the provider from the registry; L0 is offline by contract, so the module is parsed,
+#     not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
+#     with a network-allowed stage, not here.
+#   - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
+#     live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
+#   - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
+#     line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
+#   - That the baseline actually ran on a node (cluster/host state), and that the
+#     hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
+#     "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
+#     network topology fact.
+#   - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
+#     cloud-init applying the OS baseline. The implementation moved the rich baseline into
+#     the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
+#     because Proxmox provisioning is API-only and a custom cloud-init snippet needs
+#     SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
+#     "baselined ... as code", not that cloud-init be the tool, so this check asserts the
+#     STAGE, wherever it lives, and records the divergence rather than hiding it.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+NOTES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def note(msg):
+    NOTES.append(msg)
+
+
+# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
+# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
+# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
+# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
+# narrow: every other non-zero exit is a real failure. The parse assertions always run.
+ENV_ONLY_FAILURE = re.compile(
+    r"vault password|vault-id|Attempting to decrypt"
+    r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
+
+TOFU = ROOT / "infra" / "tofu"
+ANS = ROOT / "infra" / "ansible"
+check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
+check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
+if not TOFU.is_dir() or not ANS.is_dir():
+    print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+
+TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
+check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
+
+# ============================================================ STAGE 1 — CREATE (IaC)
+# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
+# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
+# here rather than being noticed later. Each entry is justified, not just listed:
+#   bpg/proxmox        Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
+#   dmacvicar/libvirt  libvirt/KVM directly (the pre-Proxmox substrate).
+KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
+providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
+                 [(m.group(1), m.group(2)) for m in
+                  re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
+check(len(providers) > 0,
+      "infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
+      "cannot be checked at all")
+for alias, source in sorted(providers.items()):
+    check(source.lower() in KVM_PROVIDERS,
+          f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
+          f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
+
+# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
+vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
+                       TF, re.S)
+check(len(vm_blocks) >= 1,
+      "no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
+for rtype, rname, body in vm_blocks:
+    check("for_each" in body,
+          f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
+          "replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
+          "'created ... as code' would not survive a rebuild")
+    check(not re.search(r"^\s*count\s*=", body, re.M),
+          f'resource "{rtype}" "{rname}" uses a positional count index')
+    for want, why in (
+        (r"\bcpu\b", "CPU"),
+        (r"\bmemory\b", "RAM"),
+        (r"network_device|network_interface", "a NIC"),
+        (r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
+    ):
+        check(re.search(want, body) is not None,
+              f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
+              "disks, NICs and a cloud-init seed to come from IaC")
+    disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
+    check(disks >= 2,
+          f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
+          "disk PLUS a dedicated data disk for Longhorn")
+    check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
+          f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
+          "reached by the BASELINE stage without a hand step")
+
+# Node identity comes from variables, and those variables define the whole node set.
+def tf_default(var: str):
+    m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
+    if not m:
+        return None
+    d = re.search(r'default\s*=\s*(.+)', m.group(1))
+    return d.group(1).strip() if d else None
+
+
+prefix_raw = tf_default("node_name_prefix")
+count_raw = tf_default("node_count")
+ips_raw = tf_default("node_ips")
+check(prefix_raw is not None and count_raw is not None,
+      "infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
+      "from code, so the created nodes cannot be compared with the managed ones")
+prefix = (prefix_raw or "").strip('"')
+try:
+    node_count = int((count_raw or "0").strip())
+except ValueError:
+    node_count = 0
+check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
+if ips_raw is not None:
+    n_ips = len(re.findall(r'"[^"]+"', ips_raw))
+    check(n_ips >= node_count,
+          f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
+          "cannot address every node it creates")
+check((TOFU / "outputs.tf").is_file() and
+      re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
+      "infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
+      "told the node facts by hand instead of consuming them from IaC")
+
+# ------------------------------------------------- the dev node set, derived twice over
+# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
+# restates the estate's naming.
+PLAYBOOKS = ANS / "playbooks"
+
+
+def default_patterns(playbook: str) -> list[str]:
+    p = PLAYBOOKS / playbook
+    if not p.is_file():
+        return []
+    return re.findall(r"default\('([^']+)'\)", p.read_text())
+
+
+dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
+check(len(dev_groups) > 0,
+      "no playbook declares a default host pattern — the dev target is not expressed in code")
+
+INV = ANS / "inventory" / "hosts.ini"
+check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
+ini_groups: dict[str, list[str]] = {}
+children: dict[str, list[str]] = {}
+cur = None
+for line in (INV.read_text().splitlines() if INV.is_file() else []):
+    s = line.strip()
+    if not s or s.startswith("#"):
+        continue
+    m = re.match(r"^\[([^\]]+)\]$", s)
+    if m:
+        cur = m.group(1)
+        (children if cur.endswith(":children") else ini_groups).setdefault(
+            cur.split(":")[0] if cur.endswith(":children") else cur, [])
+        continue
+    if cur is None or cur.endswith(":vars"):
+        continue
+    host = s.split()[0]
+    if cur.endswith(":children"):
+        children.setdefault(cur.split(":")[0], []).append(host)
+    else:
+        ini_groups.setdefault(cur, []).append(host)
+
+
+def hosts_in(group: str, seen=None) -> set[str]:
+    seen = seen or set()
+    if group in seen:
+        return set()
+    seen.add(group)
+    out = set(ini_groups.get(group, []))
+    for child in children.get(group, []):
+        out |= hosts_in(child, seen)
+    return out
+
+
+dev_hosts: set[str] = set()
+for g in sorted(dev_groups):
+    dev_hosts |= hosts_in(g)
+check(len(dev_hosts) > 0,
+      f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
+expected = {f"{prefix}{i + 1}" for i in range(node_count)}
+check(dev_hosts == expected,
+      f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
+      f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
+
+# ============================================================ STAGE 2 — BASELINE (code)
+BASE_ROLE = ANS / "roles" / "node_baseline"
+check(BASE_ROLE.is_dir(),
+      "no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
+base_tasks = "\n".join(p.read_text(errors="replace")
+                       for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
+base_defaults = "\n".join(p.read_text(errors="replace")
+                          for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
+# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
+# a declarative, idempotent task, not a shell one-liner.
+for pattern, what in (
+    (r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
+    (r"modprobe:", "kernel modules loaded (modprobe module)"),
+    (r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
+    (r"sysctl", "sysctl tuning"),
+    (r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
+    (r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
+    (r"swap", "swap disabled"),
+    (r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
+):
+    check(re.search(pattern, base_tasks) is not None,
+          f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
+          "k3s/Longhorn/KubeVirt prerequisites to be applied as code")
+check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
+      re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
+      "node_baseline's package/module sets are not variables with defaults — a per-host special "
+      "case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
+
+# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
+K3S_ROLE = ANS / "roles" / "k3s_server"
+k3s_text = "\n".join(p.read_text(errors="replace")
+                     for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
+check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
+      "no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
+check("--cluster-init" in k3s_text,
+      "the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
+check(re.search(r"--server\s+https", k3s_text) is not None,
+      "the k3s role never joins secondaries with --server — nodes would not form one cluster")
+check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
+      "the k3s role never supplies a join token — the join could only be completed by hand")
+# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
+# flag that actually disables it.
+for pattern, what in (
+    (r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
+    (r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
+    (r"traefik", "traefik (Gateway API via Cilium)"),
+    (r"servicelb", "servicelb"),
+):
+    check(re.search(pattern, k3s_text) is not None,
+          f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
+          "install, or the bundled component fights the platform's own")
+check((ANS / "roles" / "cilium").is_dir(),
+      "no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
+check((ANS / "roles" / "flux_bootstrap").is_dir(),
+      "no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
+
+# ============================================== STAGE 4 — MANAGE + the chain is REAL
+SITE = PLAYBOOKS / "site.yml"
+check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
+site_txt = SITE.read_text() if SITE.is_file() else ""
+order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
+check(order[:1] == ["node-baseline.yml"],
+      f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
+for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
+    check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
+if "k3s.yml" in order and "cilium.yml" in order:
+    check(order.index("k3s.yml") < order.index("cilium.yml"),
+          "site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
+
+# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
+# --syntax-check and --list-hosts open NO connection to any node: they resolve the
+# playbook, roles, collections and inventory only. Running them is the difference between
+# "the chain is written down" and "the chain resolves".
+if shutil.which("ansible-playbook") is None:
+    note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
+         "chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
+else:
+    # cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
+    # is the one in force — the same resolution a real run gets.
+    def ansible(*args) -> subprocess.CompletedProcess:
+        return subprocess.run(["ansible-playbook", *args], cwd=ANS,
+                              capture_output=True, text=True)
+
+    syn = ansible("--syntax-check", "playbooks/site.yml")
+    syn_out = (syn.stderr or "") + (syn.stdout or "")
+    lh = None
+    if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
+        # NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
+        note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
+             "passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
+             "were not driven; every parse assertion above still ran")
+    else:
+        check(syn.returncode == 0,
+              "`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
+              f"does not even load: {syn_out.strip()[:400]}")
+        lh = ansible("--list-hosts", "playbooks/site.yml")
+        check(lh.returncode == 0,
+              "`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
+              f"resolve against the tracked inventory: "
+              f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
+    if lh is not None and lh.returncode == 0:
+        # Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
+        plays: list[tuple[str, set[str]]] = []
+        name, hosts, in_hosts = None, set(), False
+        for line in lh.stdout.splitlines():
+            m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
+            if m:
+                if name is not None:
+                    plays.append((name, hosts))
+                name, hosts, in_hosts = m.group(1), set(), False
+                continue
+            if re.match(r"\s*hosts \(\d+\):", line):
+                in_hosts = True
+                continue
+            if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
+                hosts.add(line.strip())
+        if name is not None:
+            plays.append((name, hosts))
+        check(len(plays) >= 4,
+              f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
+        # Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
+        # nodes means the third was finished by hand.
+        STAGES = {
+            "baseline": r"baseline",
+            "cluster-join": r"^k3s (primary|secondar)",
+            "cni": r"cilium",
+            "manage (flux hand-off)": r"flux",
+        }
+        for stage, rx in STAGES.items():
+            covered: set[str] = set()
+            matched = 0
+            for pname, phosts in plays:
+                if re.search(rx, pname, re.I):
+                    matched += 1
+                    covered |= phosts
+            check(matched >= 1, f"no play in site.yml implements the {stage} stage")
+            check(covered == dev_hosts,
+                  f"the {stage} stage targets {sorted(covered)}, not every dev node "
+                  f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
+                  "node (R-ENV-01)")
+
+# --------------------------------------------------------------------------- verdict
+# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
+check(CHECKS >= 30,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+for n in NOTES:
+    print(f"  note: {n}")
+print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
+      f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
+      f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
+      "clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
+      "proven here (see the header's gap list; rebuild is L3 node-loss).")
+PY

conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml

@@ -0,0 +1,94 @@
+# =============================================================================
+# L0 STATIC — two egress paths, and the rule set that selects between them  (R-NET-04)  [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
+#          routing rules ... realized as two paths — the local `internet` path and the
+#          in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
+#          routing: two paths, one physical WAN": "the `internet` path is simply the default
+#          route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
+#          the nodes themselves — egress the local/private WAN by default and **never** the
+#          VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
+#          kill-switch blackholes **only** that class."
+#
+# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
+# ------------------------------------------------------------------------
+# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
+# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
+# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
+# that select among them, which nothing else asserts:
+#   * exactly TWO paths, the same closed set in the schema, the admission policy and the
+#     renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
+#   * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
+#     Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
+#     `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
+#     absence of a matching rule IS the default path;
+#   * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
+#     vpn-class pod's cluster/LAN traffic still takes the local path;
+#   * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
+#     table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
+#   * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
+#     DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
+#     (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
+#
+# WHY L0: every clause above is a statement about ALL rules — including rules that must be
+# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
+# route in the main table). A cluster can show which path two pods took; it cannot show that
+# no third rule exists or that the vocabulary is closed.
+#
+# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
+# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
+# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
+# against its own values.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
+#       routing / OpenVPN / nftables templates, and every environment's overlay addresses are
+#       parsed and cross-checked,
+# THEN  there are exactly two paths with `internet` as the default; exactly one steering rule
+#       selects exactly the vpn class by label; the gateway is selected by node label with a
+#       named egress interface; all RFC1918 space and every address any environment assigns
+#       itself is excluded from the tunnel; the tunnel's default route and its fail-closed
+#       blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
+#       cannot take the main table; and no nftables DROP is unscoped.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
+# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
+# Chainsaw Test wraps the SAME script.
+#
+# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
+#   * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
+#     tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
+#     classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
+#   * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
+#     that generate them; a hand-added rule on the box is invisible here.
+#   * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
+#     honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
+#   * IPv6 egress.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: egress-paths-selected-by-rules
+  labels:
+    conformance.home.k3s/R-NET-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-NET-04"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
+      node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
+      exactly two egress paths with the WAN as the default, exactly one rule selects exactly
+      the vpn class, local destinations stay on the local path in every environment, the
+      tunnel never becomes the default route, and nothing unscoped can blackhole the default
+      path.
+spec:
+  steps:
+    - name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
+        conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh

@@ -0,0 +1,513 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
+# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
+#         rules. Per OQ-1 the environment has **one physical commercial WAN and no
+#         multi-ISP plans**, so this is realized as two paths — the local `internet` path and
+#         the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
+#         uplinks) is **out of scope** unless a future ADR reopens it."
+#         — spec docs/00-vision-and-scope.md).
+#
+# The elaborating spec text this check quotes and asserts against — docs/03
+# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
+#     | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
+#                                                    router -> internet |
+#     | `vpn`      | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
+#                                                    egress gateway (ADR-0013) |
+#     "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
+#      to policy-route across; the `internet` path is simply the default route out."
+#     "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
+#      themselves — egress the local/private WAN by default and **never** the VPN. Only
+#      services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
+#      blackholes **only** that class."
+#     "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
+#      route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
+#      only what Cilium points at `tun0` uses the VPN."
+#
+# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
+# ---------------------------------------------------------------------------------
+# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
+# and the class can be flipped via GitOps. Those are proven by
+# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
+# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
+# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
+#
+# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
+# distinctive content — none of which any existing test asserts — is:
+#
+#   (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
+#       one closed set of two values agreed by the schema, the admission policy and the
+#       renderer. A third value appearing in one of them but not the others would be a path
+#       nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
+#       class would contradict OQ-1's resolution.
+#   (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
+#       path is entered only via rules that select the vpn class — a Cilium
+#       CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
+#       to the tunnel source address. Nothing selects the `internet` class, because
+#       "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
+#       the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
+#   (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
+#       rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
+#       pod's cluster/LAN traffic still takes the local path at the same time as its
+#       internet traffic takes the tunnel.
+#   (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
+#       lives in a DEDICATED routing table reached only through the scoped rule, the
+#       provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
+#       from touching the main table. Otherwise there would be one path (the VPN), not two.
+#   (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
+#       dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
+#       by the vpn class's own source set / fwmark. The `internet` path therefore fails
+#       OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
+#       CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
+#       asserts the OTHER direction, which that test does not.)
+#
+# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
+# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
+# exactly one class"), and scoping ("no unqualified drop, no default route in the main
+# table") are statements about ALL rules, including the ones that would have to be absent.
+# A running cluster can show you which path two pods took; it cannot show you that no third
+# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
+# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
+# tests/l2-behavioral/egress-default-path (same requirement label).
+#
+# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
+# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
+# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
+# LAN is checked against its own addresses, and the check keeps working when they change.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
+#   - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
+#     pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
+#     classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
+#     (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
+#   - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
+#     generate them, driven by no node. A hand-added `ip rule` on the gateway node is
+#     invisible here (and to every other test); drift on the box is an Ansible check-mode
+#     exercise for the operator.
+#   - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
+#     honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
+#     because nobody tried" — and inventing one would be a hollow assertion.
+#   - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
+#     If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
+#     this check would NOT catch its absence.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import ipaddress
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+LBL = "homelab.keithjackson.dev"
+CLASS_LABEL = f"{LBL}/egress-class"
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def die():
+    print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+
+
+def docs(path: Path):
+    """Every YAML document in a file, skipping empties."""
+    try:
+        return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
+                      if ".git" not in p.parts and "_artifacts" not in p.parts)
+SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
+RENDER = ROOT / "catalog" / "render.py"
+VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
+CLUSTERS = ROOT / "clusters"
+
+for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
+    check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
+                      "read at all (R-NET-04)")
+if FAILURES:
+    die()
+
+
+# ======================================================= (1) EXACTLY TWO PATHS, AGREED
+import json  # noqa: E402  — only needed for the schema
+
+schema = json.loads(SCHEMA.read_text())
+egress_prop = ((schema.get("properties") or {}).get("egress") or {})
+enum = egress_prop.get("enum")
+check(isinstance(enum, list) and len(enum) == 2,
+      f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
+      "realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
+      "nothing routes; one value is not 'multiple egress paths'")
+PATHS = set(enum or [])
+check(PATHS == {"internet", "vpn"},
+      f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
+      "`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
+check(egress_prop.get("default") == "internet",
+      f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
+      "'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
+      "the default route out'. A service that says nothing must get the WAN")
+
+# The admission policy must constrain the pod label to the SAME closed set, or a workload
+# could carry a class the schema never allowed (and that no rule routes).
+pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
+check(len(pol_files) >= 1,
+      "no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
+      "carrying an egress class outside the declared path set")
+pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
+for cls in sorted(PATHS):
+    check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
+          f"the egress admission policy never names the {cls!r} path — a class the policy "
+          "does not know cannot be validated against the path set")
+# No tracked object may carry a class outside the closed set — with ONE self-policing
+# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
+# admission test can assert the policy REJECTS it; that pod is exempt only while a
+# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
+# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
+def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
+    tests_root = fixture.parent
+    while tests_root != ROOT and tests_root.name != "tests":
+        tests_root = tests_root.parent
+    if tests_root == ROOT:
+        return False
+    for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
+        for d in docs(tf):
+            for case in (d.get("results") or []):
+                if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
+                    return True
+    return False
+
+
+stray: list[str] = []
+_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
+for f in TRACKED_YAML:
+    for d in docs(f):
+        labels = ((d.get("metadata") or {}).get("labels") or {})
+        cls = labels.get(CLASS_LABEL)
+        # Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
+        tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
+        for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
+            if value is None or value in PATHS or value == "CLASS":
+                continue
+            name = ((owner.get("metadata") or {}).get("name") or "?")
+            if rejected_by_a_policy_test(name, f):
+                continue
+            stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
+    # Values written as raw text (templates, docs-as-code) are caught too.
+    for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
+        if value in PATHS or value == "CLASS":
+            continue
+        if any(value in s for s in stray):
+            continue
+        # Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
+        # carries this value is a policy-test fixture proven to be rejected.
+        owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
+                  if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
+        if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
+            continue
+        stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
+check(not stray,
+      f"tracked YAML assigns egress-class value(s) outside the declared path set "
+      f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
+      "policy allows is a workload on no path at all (the only exemption is a Kyverno "
+      "policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
+
+# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
+# what the routing rule selects on, so a renderer that rewrote it would break the coupling
+# between the declaration and the path.
+rtext = RENDER.read_text()
+check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
+      or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
+      "catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
+      f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
+      "two independent facts that could disagree")
+
+
+# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
+policies: list[tuple[Path, dict]] = []
+for p in TRACKED_YAML:
+    for d in docs(p):
+        if d.get("kind") == "CiliumEgressGatewayPolicy":
+            policies.append((p, d))
+check(len(policies) == 1,
+      f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
+      f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
+      "rule. Two rules can overlap, and which one wins for an overlapping pod is not "
+      "something the repository states")
+
+for path, pol in policies:
+    rel = str(path.relative_to(ROOT))
+    spec = pol.get("spec") or {}
+    selectors = spec.get("selectors") or []
+    check(len(selectors) == 1,
+          f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
+          "selector widens the set of pods steered off the default path, and R-NET-04's "
+          "selection must be exactly 'the vpn class'")
+    for sel in selectors:
+        pod_sel = sel.get("podSelector")
+        check(isinstance(pod_sel, dict) and pod_sel,
+              f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
+              "EVERY pod, which would put the whole cluster on the VPN path and leave no "
+              "default path at all")
+        ml = (pod_sel or {}).get("matchLabels") or {}
+        me = (pod_sel or {}).get("matchExpressions") or []
+        check(not me,
+              f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
+              "single label equality; expressions (In/NotIn/Exists) can select pods that "
+              "never declared a class, so the rule set would no longer be disjoint from the "
+              "default path")
+        check(set(ml) == {CLASS_LABEL},
+              f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
+              f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
+              "than the declared class (and a namespace/app key would silently exempt pods)")
+        check(ml.get(CLASS_LABEL) == "vpn",
+              f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
+              "the `vpn` class may be steered: docs/03 — 'Only services labeled "
+              "egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
+              "route the default path through the tunnel, i.e. leave one path, not two")
+
+    # The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
+    # rule, so the gateway can move (or gain a standby) with no rule rewrite.
+    gw = spec.get("egressGateway") or {}
+    node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
+    check(bool(node_sel),
+          f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
+          "named host means the vpn path cannot fail over (docs/03 requires two gateway "
+          "nodes + Egress Gateway HA)")
+    check("nodeName" not in gw,
+          f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
+          "label so the path survives losing that node")
+    check(isinstance(gw.get("interface"), str) and gw.get("interface"),
+          f"{rel}: the egressGateway names no egress interface — without it the steered class "
+          "is not actually sent down a different path from the default")
+
+    # ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
+    dests = spec.get("destinationCIDRs") or []
+    excl = spec.get("excludedCIDRs") or []
+    check([str(c) for c in dests] == ["0.0.0.0/0"],
+          f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
+          "('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
+          "minus the exclusions below — narrowing it here would silently leave public "
+          "destinations on the WAN for a vpn-class service")
+    try:
+        excluded = [ipaddress.ip_network(str(c)) for c in excl]
+    except ValueError as e:
+        excluded = []
+        check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
+    check(len(excluded) >= 3,
+          f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
+          "pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
+
+    def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
+        return any(net.subnet_of(e) for e in excluded if e.version == net.version)
+
+    # RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
+    for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
+        net = ipaddress.ip_network(rfc)
+        check(covered(net),
+              f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
+              "pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
+              "coexist for the same pod and east-west traffic would break")
+
+    # Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
+    # is what makes 'local stays local' true per environment rather than by coincidence.
+    def overlay_addresses() -> list[tuple[str, str]]:
+        found: list[tuple[str, str]] = []
+        base = CLUSTERS / "base" / "cluster-vars.yaml"
+        data = {}
+        if base.is_file():
+            for d in docs(base):
+                if d.get("kind") == "ConfigMap":
+                    data = {k: str(v) for k, v in (d.get("data") or {}).items()}
+        for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
+            env = {**data}
+            for m in re.finditer(
+                    r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
+                    r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
+                env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
+            label = f.parent.name
+            for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
+                        "CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
+                v = env.get(key, "")
+                if v and not v.startswith("TBD"):
+                    found.append((f"{label}/{key}", v))
+        return found
+
+    addrs = overlay_addresses()
+    check(len(addrs) >= 5,
+          f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
+          "the 'local traffic stays local' assertion would be checking almost nothing")
+    for label, value in addrs:
+        try:
+            net = ipaddress.ip_network(value, strict=False)
+        except ValueError:
+            continue
+        check(covered(net),
+              f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
+              "NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
+              "would try to reach its own cluster/LAN through the VPN")
+
+
+# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
+TPL = VPN_ROLE / "templates"
+up = (TPL / "vpn-egress-up.sh.j2")
+down = (TPL / "vpn-egress-down.sh.j2")
+conf = (TPL / "vpn-egress.conf.j2")
+nft = (TPL / "vpn-egress-killswitch.nft.j2")
+for f in (up, down, conf, nft):
+    check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
+                       "rules are not authored")
+if FAILURES:
+    die()
+
+defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
+
+
+def render(text: str) -> str:
+    """Substitute `{{ var }}` with the role's own default value, for every default.
+
+    The templates reference their knobs by NAME, so an analysis that looked for the VALUE
+    (e.g. the nft set name) would silently match nothing and pass vacuously — which is
+    exactly what an earlier draft of this check did. Rendering first means the assertions
+    below examine the text the node actually gets. Only simple `{{ name }}` expressions are
+    substituted; anything else is left as-is (and the assertions are written so an
+    unsubstituted expression fails rather than passes).
+    """
+    out = text
+    for k, v in defaults.items():
+        if isinstance(v, (str, int)):
+            out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
+    return out
+
+
+up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
+table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
+check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
+      f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
+      "DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
+      "putting the tunnel's default route in any of them makes the VPN the node's default "
+      "route instead of a selectable second path")
+check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
+      "no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
+      "needs a rule, at a defined precedence")
+
+# Every `ip route ... default` the up-script installs must name the dedicated table.
+route_lines = [l.strip() for l in up_t.splitlines()
+               if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
+check(len(route_lines) >= 2,
+      f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
+      "default route via the tunnel AND the fail-closed blackhole behind it")
+for l in route_lines:
+    check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
+          f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
+          "landing in the main table changes the path every other pod and the node itself "
+          "uses — the VPN would stop being opt-in")
+    check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
+          f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
+          "kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
+          "be blackholed")
+
+# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
+# everything into the tunnel table, which would leave one path instead of two.
+rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
+check(len(rule_lines) >= 1,
+      "the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
+      "routing table would never be consulted (R-NET-04's 'selected by routing rules')")
+for l in rule_lines:
+    check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
+          f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
+          "EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
+          "VPN is no longer opt-in")
+    check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
+          f"an ip rule looks up a table other than the tunnel's: {l!r}")
+check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
+      "the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
+      "address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
+      "distinguishes the steered class from every other flow on the gateway node")
+check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
+      "the up-script does not refuse to run when the tunnel address is unknown — installing "
+      "a partial rule set is how a path ends up half-selected")
+
+# The provider must not be able to make the tunnel the default route either.
+check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
+      "the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
+      "routes itself, in the MAIN table, and the VPN would become the node's default route "
+      "(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
+      "routing)')")
+check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
+      "the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
+      "provider could redirect the node's whole default route into the tunnel, collapsing "
+      "the two paths into one")
+check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
+      "the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
+      "default route")
+
+
+# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
+src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
+fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
+check(bool(src_set),
+      "the role declares no nftables source set — the kill-switch would have no way to name "
+      "'the steered class' and could only match broadly")
+policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
+check(len(policies_lines) >= 1,
+      "the kill-switch nft template declares no base chain — nothing would be enforced")
+for l in policies_lines:
+    check(re.search(r"policy\s+accept\s*;", l) is not None,
+          f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
+          "blackholes the `internet` path too, and docs/03 requires the two classes to fail "
+          "in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
+drop_lines = [l.strip() for l in nft_t.splitlines()
+              if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
+check(len(drop_lines) >= 1,
+      "the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
+      "and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
+for l in drop_lines:
+    scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
+    check(scoped,
+          f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
+          f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
+          "the default `internet` path, which must never be blackholed")
+
+
+# --------------------------------------------------------------------------- verdict
+# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
+check(CHECKS >= 35,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    die()
+print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
+      f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
+      f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
+      f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
+      f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
+      "is L2 (egress-class for the two labelled classes, egress-default-path for the "
+      "unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
+PY

conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml

@@ -0,0 +1,88 @@
+# =============================================================================
+# L0 STATIC — the HA control plane is authored, for every environment  (R-PLAT-01)  [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
+#          production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
+#          exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
+#          embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
+#          "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
+#
+# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
+# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
+# from one code path, with exactly one --cluster-init founder, no external datastore, and a
+# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
+# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
+# cluster cannot tell you what the NEXT environment brought up from this repo would look
+# like. The observed half is asserted separately at L2 by
+# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
+#
+# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
+# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
+# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
+# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the inventory, the k3s_server and kube_vip roles, their plays and each
+#       environment's Flux overlay are parsed and cross-checked against each other,
+# THEN  k3s is the pinned distro and nothing is installed as an agent; every environment
+#       resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
+#       ONE --cluster-init founder and every other server joining it with --server;
+#       the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
+#       the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
+#       DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
+#       in the apiserver serving cert's SANs together with every server's own address, and
+#       it is neither a node address nor the ingress Gateway VIP nor inside that
+#       environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
+#       roles and plays retargeted by variables only.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
+# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
+# Chainsaw Test wraps the SAME script.
+#
+# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
+#   * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
+#     API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
+#   * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
+#     is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
+#     drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
+#   * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
+#     infra/PROD-PROVISIONING.md).
+#
+# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
+#   * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
+#     environments rather than the floating VIP, contradicting the prod overlay's own comment.
+#     It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
+#     The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
+#     contract and per environment.
+#   * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
+#     the order is asserted — not as a tidiness preference but because the CNI's endpoint is
+#     the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
+#     API server.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: ha-control-plane-as-code
+  labels:
+    conformance.home.k3s/R-PLAT-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-PLAT-01"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
+      environment's Flux overlay are parsed and cross-checked, then every environment is
+      authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
+      embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
+      VIP that is in the serving cert's SANs and collides with no other allocator — built by
+      one code path for both environments.
+spec:
+  steps:
+    - name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
+        conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/ha-control-plane-as-code/check.sh

@@ -0,0 +1,729 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — ha-control-plane-as-code
+# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
+#         HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
+#         same multi-node topology." — spec docs/00-vision-and-scope.md).
+#
+# The elaborating spec text this check quotes and asserts against:
+#   docs/01-architecture-overview.md  "Kubernetes distro | **k3s** (HA, embedded etcd)"
+#   docs/02-environments-and-provisioning.md, table:
+#       "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
+#        (k3s server on all 3)"   (the SAME cell for dev/test and prod)
+#       "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
+#   docs/02 §"k3s topology decisions":
+#       "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
+#       "Nodes are both control-plane and workload (3-node converged cluster); with only 3
+#        nodes we do not isolate control-plane nodes."
+#       "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
+#        step differs"
+#
+# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
+# ---------------------------------------------------------------------
+# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
+#
+#   AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
+#       plane in BOTH environments, from ONE code path": provable with no cluster, and
+#       ONLY provable here. A live cluster cannot tell you whether the *next* environment
+#       brought up from this repo will also be HA, whether a second `--cluster-init`
+#       founder could exist, or whether an external datastore is configured anywhere. That
+#       is this check.
+#   OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
+#       etcd members, with the API answering through the floating VIP": that is cluster
+#       state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
+#
+# Neither half is sufficient alone, and this file claims only the first.
+#
+# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
+# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
+# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
+# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
+# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
+# its group name. Add a fourth prod node, rename a group, or add a third environment and
+# the assertions follow; nothing here restates an address, a hostname or a node count.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
+#       Flux overlay are parsed and cross-checked against each other,
+# THEN  DISTRO:   k3s is the distro, pinned, installed by the one role, and no node is
+#                 installed as an `agent` (an agent is not a control-plane member);
+#       SERVERS:  every environment's server group resolves to a quorum-capable, ODD
+#                 number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
+#                 EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
+#                 every other server joins it with `--server` (so a second founder — the
+#                 split-brain shape — cannot be authored);
+#       ETCD:     the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
+#                 `--disable-etcd` anywhere in the provisioning code, so every server is
+#                 an etcd member (docs/02's "no external datastore", "on all 3");
+#       ENDPOINT: the API has a floating endpoint that survives losing the founder — the
+#                 kube-vip VIP is control-plane-only (never Service LB, or it would fight
+#                 Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
+#                 baked into the apiserver serving cert's SAN list ALONG WITH every
+#                 server's own address (so TLS validates whichever node answers);
+#       ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
+#                 is not that environment's ingress Gateway VIP, and lies OUTSIDE that
+#                 environment's Cilium LoadBalancer pool — the two allocators can never be
+#                 handed the same address;
+#       CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
+#                 two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
+#                 the cold `--server` join, k3s_apiserver_endpoint derives from
+#                 kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
+#                 K3S_API_HOST equals that environment's VIP and is none of its node
+#                 addresses ([parity], both halves), the HelmRelease keeps a per-environment
+#                 token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
+#                 Cilium with roles/kube_vip blocking until it answers;
+#       ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
+#                 retargeted by variables only (docs/02 "the *same* Ansible bootstrap
+#                 runs"), so "exercise the same multi-node topology" is a property of the
+#                 code rather than a coincidence of two hand-written inventories.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
+#   - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
+#     a healthy quorum, that kube-vip is actually floating the VIP, and that the API
+#     answers through it. All of that is asserted at L2 by
+#     tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
+#   - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
+#     the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
+#     coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
+#     data-availability claim and does NOT assert control-plane survival. A dedicated
+#     "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
+#     is NOT claimed here or at L2.
+#   - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
+#     mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
+#     decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
+#     check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
+#   - IPv6 / dual-stack, and any claim about the edge network.
+#
+# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
+#   * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
+#     (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
+#     environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
+#     overlay's own comment. Making the check fail over it would have turned the merge gate
+#     red on a design question this workstream may not decide (R-TEST-06), so it was reported
+#     instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
+#     repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
+#     VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
+#     is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
+#   * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
+#     k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
+#     and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
+#     for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
+#     unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
+#     ordering is no longer a preference about tidiness, it is a dependency of the control
+#     plane coming up at all.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import ipaddress
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+NOTES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def note(msg):
+    NOTES.append(msg)
+
+
+def die():
+    print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+
+
+ANS = ROOT / "infra" / "ansible"
+INV = ANS / "inventory" / "hosts.ini"
+GV = ANS / "inventory" / "group_vars"
+K3S_ROLE = ANS / "roles" / "k3s_server"
+VIP_ROLE = ANS / "roles" / "kube_vip"
+PLAYBOOKS = ANS / "playbooks"
+CLUSTERS = ROOT / "clusters"
+
+for p, why in ((INV, "the inventory that defines the node set"),
+               (K3S_ROLE, "the role that installs the k3s control plane"),
+               (VIP_ROLE, "the role that floats the control-plane VIP"),
+               (CLUSTERS, "the Flux overlays that identify each environment")):
+    check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
+                      "control plane is authored at all (R-PLAT-01)")
+if FAILURES:
+    die()
+
+K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
+VIP_TEXT = "\n".join(p.read_text(errors="replace")
+                     for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
+
+
+# ===================================================================== inventory parse
+# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
+# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
+# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
+ini_hosts: dict[str, list[str]] = {}
+ini_children: dict[str, list[str]] = {}
+host_vars: dict[str, dict[str, str]] = {}
+cur = None
+for line in INV.read_text().splitlines():
+    s = line.strip()
+    if not s or s.startswith("#"):
+        continue
+    m = re.match(r"^\[([^\]]+)\]$", s)
+    if m:
+        cur = m.group(1)
+        continue
+    if cur is None or cur.endswith(":vars"):
+        continue
+    host = s.split()[0]
+    if cur.endswith(":children"):
+        ini_children.setdefault(cur.split(":")[0], []).append(host)
+    else:
+        ini_hosts.setdefault(cur, []).append(host)
+        hv = host_vars.setdefault(host, {})
+        for kv in s.split()[1:]:
+            if "=" in kv:
+                k, v = kv.split("=", 1)
+                hv[k] = v
+
+
+def hosts_in(group: str, seen=None) -> set[str]:
+    seen = seen or set()
+    if group in seen:
+        return set()
+    seen.add(group)
+    out = set(ini_hosts.get(group, []))
+    for child in ini_children.get(group, []):
+        out |= hosts_in(child, seen)
+    return out
+
+
+def load_yaml(p: Path) -> dict:
+    try:
+        d = yaml.safe_load(p.read_text()) or {}
+    except yaml.YAMLError:
+        return {}
+    return d if isinstance(d, dict) else {}
+
+
+ALL_VARS = load_yaml(GV / "all.yml")
+
+
+# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
+# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
+# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
+# another environment. Nothing below knows the words "dev" or "prod" as group names.
+class Env:
+    def __init__(self, label, vars_, path):
+        self.label = label
+        self.vars = vars_
+        self.path = path
+
+    def var(self, name, default=None):
+        # group-scoped value wins, else the cluster-wide default from all.yml.
+        if name in self.vars:
+            return self.vars[name]
+        return ALL_VARS.get(name, default)
+
+
+envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
+for vf in sorted(GV.glob("*/vars.yml")):
+    v = load_yaml(vf)
+    if "k3s_primary_group" in v or "k3s_servers_group" in v:
+        envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
+
+check(len(envs) >= 2,
+      "only ONE environment is derivable from the inventory's R-ENV-02 seam "
+      "(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
+      "dev/test to exercise the same multi-node topology, so two must be authored")
+
+# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
+# roles/flux_bootstrap's own default supplies the value for an environment that does not
+# override it, so this is read from code, never guessed from a group name.
+fb_default = None
+fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
+if fb_defaults_file.is_file():
+    fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
+check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
+      "roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
+      "an inventory environment to its Flux overlay is not expressed in code, so this check "
+      "cannot tell which environment is prod")
+
+prod_envs = []
+for e in envs:
+    overlay = e.var("flux_bootstrap_env_path", fb_default)
+    e.overlay = overlay if isinstance(overlay, str) else ""
+    if e.overlay.rstrip("/").endswith("/prod"):
+        prod_envs.append(e)
+check(len(prod_envs) == 1,
+      f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
+      f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
+      "R-PLAT-01 has no unambiguous referent")
+
+
+# ============================================== overlay knob values (per environment)
+# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
+# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
+#     - op: replace
+#       path: /data/<KEY>
+#       value: "<v>"
+# (verified shape across both overlays). A key not patched keeps the base value.
+BASE_VARS: dict[str, str] = {}
+base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
+if base_cm.is_file():
+    d = load_yaml(base_cm)
+    BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
+check(len(BASE_VARS) > 5,
+      "clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
+      "cannot be read, so the VIP/pool separation cannot be checked")
+
+_PATCH_RE = re.compile(
+    r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
+    re.M)
+
+
+def overlay_vars(overlay: str) -> dict[str, str]:
+    out = dict(BASE_VARS)
+    f = ROOT / overlay / "kustomization.yaml"
+    if not f.is_file():
+        return out
+    for m in _PATCH_RE.finditer(f.read_text()):
+        out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
+    return out
+
+
+# ============================================================ PER-ENVIRONMENT TOPOLOGY
+def as_ip(s):
+    try:
+        return ipaddress.ip_address(str(s).strip())
+    except ValueError:
+        return None
+
+
+for e in envs:
+    L = e.label
+    pg = e.var("k3s_primary_group")
+    sg = e.var("k3s_servers_group")
+    check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
+          f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
+          "resolve 'my environment's primary/servers' and would only ever build dev")
+    if not (isinstance(pg, str) and isinstance(sg, str)):
+        continue
+    primaries = hosts_in(pg)
+    servers = hosts_in(sg)
+
+    # --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
+    check(len(primaries) == 1,
+          f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
+          "`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
+          "cluster at all")
+    check(primaries and primaries <= servers,
+          f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
+          f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
+          "it is not part of the HA control plane it founds")
+    n = len(servers)
+    check(n >= 3,
+          f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
+          "'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
+          "cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
+    check(n % 2 == 1,
+          f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
+          f"member count adds a member without adding fault tolerance and introduces a tie: "
+          f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
+    if n:
+        tolerance = n - (n // 2 + 1)
+        check(tolerance >= 1,
+              f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
+              f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
+    if e in prod_envs:
+        check(n == 3,
+              f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
+              "'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
+    else:
+        check(n >= 2,
+              f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
+              "nodes than prod but requires dev/test to 'exercise the same multi-node "
+              "topology' — one node exercises no topology")
+
+    # --- ADDRESSES: the VIP can collide with nothing ----------------------------------
+    vip_raw = e.var("kube_vip_address")
+    vip = as_ip(vip_raw)
+    check(vip is not None,
+          f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
+          "no floating endpoint, so losing the founder takes the API endpoint with it")
+    node_ips = {}
+    for h in sorted(servers):
+        ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
+        check(ip is not None,
+              f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
+              "target and the serving-cert SAN list all derive from it")
+        if ip is not None:
+            node_ips[h] = ip
+    if vip is not None:
+        check(vip not in set(node_ips.values()),
+              f"{L}: kube_vip_address {vip} is ALSO a node address "
+              f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
+              "node's own address cannot float off that node")
+        ov = overlay_vars(getattr(e, "overlay", ""))
+        gw = as_ip(ov.get("GATEWAY_VIP", ""))
+        lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
+        hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
+        check(gw is not None and lo is not None and hi is not None,
+              f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
+              "CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
+              "the Service-LB allocator cannot be checked")
+        if gw is not None:
+            check(vip != gw,
+                  f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
+                  f"address ({vip}) — kube-vip and Cilium would both ARP for it")
+        if lo is not None and hi is not None:
+            check(not (lo <= vip <= hi),
+                  f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
+                  f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
+                  "Service, and the two allocators would fight over ARP for it")
+            check(lo <= hi,
+                  f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
+                  "used to prove the VIP is outside it")
+
+# ==================================================================== DISTRO: k3s only
+check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
+      is not None,
+      "roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
+      "version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
+check("get.k3s.io" in K3S_TEXT,
+      "roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
+      "distro (ADR-0001) and no other distro may stand in for it")
+check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
+      "roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
+      "AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
+for bad, why in (
+    (r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
+                             "nodes (no external datastore)')"),
+    (r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
+    (r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
+                        "an etcd member (docs/02: etcd 'across all 3 nodes')"),
+    (r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
+):
+    hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
+            if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
+            and re.search(bad, p.read_text(errors="replace"))]
+    check(not hits,
+          f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
+          f"mode, and {bad!r} contradicts it")
+
+# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
+check(re.search(r"--cluster-init", K3S_TEXT) is not None,
+      "roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
+      "founded, so there is no HA datastore to join")
+init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
+                      K3S_TEXT)
+check(init_when is not None,
+      "the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
+      "ungated founder flag would re-found the cluster on every server (split brain)")
+join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
+check(join is not None,
+      "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
+      "— without a join target the servers are separate single-node clusters")
+join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
+                      K3S_TEXT, re.S)
+check(join_when is not None,
+      "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
+      "would also try to join itself")
+k3s_play = PLAYBOOKS / "k3s.yml"
+check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
+if k3s_play.is_file():
+    play_txt = k3s_play.read_text()
+    modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
+    check(sorted(modes) == ["primary", "secondary"],
+          f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
+          "one secondary play must exist, or the founder/joiner split is not authored")
+    check(re.search(r"serial:\s*1", play_txt) is not None,
+          "playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
+          "converging two of three servers at once drops embedded etcd below quorum, which "
+          "is exactly the outage an HA control plane exists to avoid")
+    hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
+    check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
+          f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
+          "group name means this play can only ever build ONE environment, so prod cannot "
+          "be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
+
+# =========================================== SERVING-CERT SANs: VIP + every server
+sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
+check("kube_vip_address" in sans,
+      "roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
+      "serving cert would carry no SAN for the floating VIP, so every client reaching the "
+      "API through it fails TLS validation and the HA endpoint is unusable")
+check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
+      "k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
+      "cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
+      "resolution of that set is what keeps one role serving both environments (R-ENV-02)")
+check(re.search(r"tls-san:", K3S_TEXT) is not None,
+      "the SAN list is passed only as an install flag and never persisted to "
+      "/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
+      "no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
+
+# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
+# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
+# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
+# plane the CNI's control-plane path depended on one machine. Measured on live prod
+# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
+# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
+# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
+# policy update lands at all.
+#
+# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
+# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
+# the configuration in git WOULD build, in BOTH environments, including the next one — a live
+# cluster can only tell you about itself.
+#
+# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
+# "Cilium uses the HA address" requires an HA address to name; before the split there was
+# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
+# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
+# the Flux half agrees with the Ansible half per environment.
+CIL_ROLE = ANS / "roles" / "cilium"
+cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
+check(cil_values_tpl.is_file(),
+      "roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
+      "API endpoint its kube-proxy replacement dials, are not authored")
+if cil_values_tpl.is_file():
+    # Parse the rendered-shape template, never its prose: strip comment lines first, exactly
+    # as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
+    cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
+                         if not l.lstrip().startswith("#"))
+    m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
+    check(m is not None,
+          "roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
+          "(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
+    if m is not None:
+        host_expr = m.group(1).strip()
+        check("k3s_apiserver_endpoint" in host_expr,
+              f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
+              "variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
+              "expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
+              "control-plane path depend on ONE server of an HA control plane, and the failure "
+              "is silent: the datapath keeps running from Cilium's cache while no service or "
+              "policy update ever lands (measured on prod 2026-09-06)")
+        check("k3s_api_host" not in host_expr,
+              f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
+              "k3s_api_host — the FOUNDER's own address. That variable exists only for the "
+              "cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
+              "failure by construction and must never be a steady-state endpoint")
+
+# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
+# than restated: an environment overrides one line (kube_vip_address) and both the floating
+# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
+# same environment actually floats, which is the two-places-one-fact bug this repo keeps
+# hitting.
+endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
+check("kube_vip_address" in endpoint_def,
+      f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
+      "from kube_vip_address, so that overriding the VIP for an environment also moves what "
+      "that environment's CNI dials. A restated literal can silently disagree with the VIP "
+      "kube-vip is actually floating")
+check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
+      "group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
+      "addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
+      "build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
+
+# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
+# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
+# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
+# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
+for e in envs:
+    L = e.label
+    ov = overlay_vars(getattr(e, "overlay", ""))
+    k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
+    vip = as_ip(e.var("kube_vip_address"))
+    check(k3s_api_host_knob is not None,
+          f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
+          "token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
+          "kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
+          "fails the whole platform Kustomization")
+    if k3s_api_host_knob is not None and vip is not None:
+        check(k3s_api_host_knob == vip,
+              f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
+              f"but this environment's control-plane VIP (kube_vip_address, which is what "
+              f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
+              f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
+              "endpoints into the same CNI and fight over it. If they differ because the "
+              "overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
+              "an HA control plane must not dial one server")
+        check(k3s_api_host_knob not in set(
+                  as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
+                  for h in hosts_in(str(e.var("k3s_servers_group")))),
+              f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
+              "environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
+              "single machine on a control plane built specifically to survive losing one")
+
+# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
+# would ship one environment's control-plane address to every environment (R-ENV-02), and an
+# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
+# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
+# being replaced could translate: deadlocked on 2 of 3 nodes).
+hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
+check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
+                    "config after the bootstrap, so the parity contract has only one side")
+if hr.is_file():
+    hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
+                        if not l.lstrip().startswith("#"))
+    hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
+    check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
+          f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
+          f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
+          "\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
+          "address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
+          "incident verbatim")
+
+# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
+# one-command bring-up, so this is a property of the authored sequence, not of an operator's
+# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
+# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
+site_play = PLAYBOOKS / "site.yml"
+check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
+                           "whose ORDER can be checked")
+if site_play.is_file():
+    imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
+    for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
+        check(needed in imports,
+              f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
+              "k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
+              "CNI, not an optional extra step — leaving it out of the bring-up means a cold "
+              "build installs a CNI pointed at an address nothing announces")
+    if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
+        check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
+              f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
+              "kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
+              "unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
+              "on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
+              "ships as a k3s auto-deploy manifest and needs a cluster to apply it")
+check(re.search(r"wait_for", VIP_TEXT) is not None
+      and re.search(r"kube_vip_address", VIP_TEXT) is not None,
+      "roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
+      "same as having a floating endpoint (k3s still has to apply it, the image still has to "
+      "pull, leader election still has to settle). Since the CNI is installed immediately "
+      "afterward against this exact address, an unverified VIP turns a missing endpoint into "
+      "unreachable cilium-agents diagnosed from the far side")
+
+# ============================================ FLOATING ENDPOINT: kube-vip on every server
+# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
+# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
+# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
+# thing that ships.) Parser contract: the template writes each env var as an adjacent
+# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
+VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
+check(len(VIP_TEMPLATES) >= 1,
+      "roles/kube_vip ships no manifest template — nothing deploys the API VIP")
+vip_manifest = "\n".join(
+    "\n".join(l for l in p.read_text(errors="replace").splitlines()
+              if not l.lstrip().startswith("#"))
+    for p in VIP_TEMPLATES)
+vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
+                          vip_manifest))
+check(len(vip_env) >= 6,
+      f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
+      "parser lost its grip on the file, so the assertions below would assert nothing")
+check(vip_env.get("cp_enable") == "true",
+      f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
+      "mode must be ON or it floats no API VIP at all")
+check(vip_env.get("svc_enable") == "false",
+      f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
+      "false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
+      "for the same addresses (ADR-0002 gives Service LB to Cilium)")
+check(vip_env.get("vip_leaderelection") == "true",
+      f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
+      "— without leader election the VIP has no defined owner to fail over FROM, which is "
+      "the whole HA mechanism")
+check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
+      f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
+      "environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
+      "prod (R-ENV-02) and would not match the serving-cert SAN list")
+check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
+      "the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
+      "node, which is the opposite of a floating endpoint")
+vip_play = PLAYBOOKS / "kube-vip.yml"
+check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
+if vip_play.is_file():
+    vp = vip_play.read_text()
+    vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
+    check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
+          f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
+          "group through the same environment-scoped variable the other plays use, so the "
+          "manifest exists on EVERY server and the VIP survives losing any one of them")
+check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
+      and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
+      "roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
+      "<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
+      "manifest somewhere k3s never reads and come up with no VIP at all")
+
+# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
+# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
+# step differs". A prod-only role or playbook would be a fork of the control-plane path.
+role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
+play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
+env_words = re.compile(r"prod|dev", re.I)
+forked = sorted(n for n in (role_names | play_names)
+                if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
+check(not forked,
+      f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
+      "the SAME code retargeted by variables (docs/02), not by a per-environment fork")
+for e in envs:
+    if e.path.name != "vars.yml":
+        continue
+    # Every key a non-default environment sets must be a variable something already reads,
+    # or it is an invented knob nothing consumes (a silent no-op that looks like config).
+    consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
+                          if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
+    for k in sorted(e.vars):
+        if k.startswith("ansible_"):
+            continue
+        check(consumers.count(k) > 1,
+              f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
+              "environment override that nothing reads is not a scoped override, it is dead "
+              "configuration (and would make prod silently inherit a dev default)")
+
+# --------------------------------------------------------------------------- verdict
+# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
+check(CHECKS >= 40,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    die()
+for n in NOTES:
+    print(f"  note: {n}")
+summary = "; ".join(
+    f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
+    f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
+    for e in envs)
+print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
+      "R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
+      "embedded etcd on every server with no external datastore, a leader-elected "
+      "control-plane-only VIP on every server that is SAN'd into the serving cert and "
+      "collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
+      "dials that VIP, not the founder, in both environments and on both sides of the "
+      "[parity] contract), from one code path for both environments. The LIVE "
+      "cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
+      "SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
+PY

conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml

@@ -0,0 +1,68 @@
+# =============================================================================
+# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
+# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
+# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
+# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
+# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
+# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
+# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
+# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
+# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
+# COUPLING was untested, which is what makes the class reintroducible.
+#
+# GIVEN the repository as authored,
+# WHEN  the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
+#       (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
+# THEN  (1) every authored value of a guarded knob that is placeholder-shaped by an
+#           INDEPENDENT detector is actually refused by that guard, in base and in every
+#           environment overlay;
+#       (2) no environment overlay carries a placeholder-shaped value for a knob no guard
+#           refuses;
+#       (3) every function in standup.sh that reads a guarded knob applies the guard to it
+#           (the "three guards died at once" regression);
+#       (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
+#           — the exact anchoring defect above — and still accepts a real endpoint;
+#   AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
+#           testing a copy if the guard stops being a self-contained predicate.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
+# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
+# wraps the SAME script.
+#
+# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
+# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
+# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
+# and that the guard runs at all in a real standup (the dispatch path is shell, executed
+# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
+# authored backup target cannot be a placeholder that the guard fails to recognise.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: placeholder-guard-coupling
+  labels:
+    conformance.home.k3s/R-ENV-02: "true"
+    conformance.home.k3s/R-STATE-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every placeholder-shaped value authored in cluster-vars, for any environment
+      overlay, is actually recognised as a placeholder by the real bootstrap guard that
+      is supposed to refuse it — so a value change in one file cannot silently disarm a
+      guard in another and enable velero against an endpoint that does not exist.
+spec:
+  steps:
+    - name: couple-authored-placeholders-to-the-real-guard
+      description: >-
+        Extract the real placeholder predicate from bootstrap/standup.sh, run it against
+        every authored cluster-vars value the repo actually carries, and assert the guard
+        refuses each one an independent detector calls a placeholder.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/placeholder-guard-coupling/check.sh

@@ -0,0 +1,516 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — placeholder-guard-coupling
+# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
+#                   — spec docs/02-environments-and-provisioning.md)
+#         R-STATE-03 ("Scheduled off-cluster backups + restore (DR)."
+#                   — spec docs/04-storage-and-state.md)
+#
+# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE
+# --------------------------------------------------------------------
+# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a
+# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
+# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
+# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
+# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match —
+# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped
+# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname
+# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and
+# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident.
+#
+# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was
+# never tested: a guard in one file keying on a value in another, with nothing tying them
+# together. This check is that tie, and it is derived from BOTH real files — no current
+# string is hardcoded here, so it does not rot when the placeholder or the guard changes.
+#
+# HOW THE REAL GUARD IS EXERCISED (and why not a copy)
+# ---------------------------------------------------
+# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
+# --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
+# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
+# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
+# those, then calls the real function. That is the real bytes, not a reimplementation:
+#   * every function whose name ends `_is_placeholder` is discovered, not named here;
+#   * a duplicate definition of one is a failure (which of the two would run?);
+#   * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
+#     else, no command substitution, no call to another function. If the guard ever grows
+#     a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
+#     remedy (move the predicate into a small sourceable library that standup.sh sources)
+#     rather than quietly falling back to testing a copy of the logic;
+#   * a CANARY pins the extraction itself: the extracted function must refuse the empty
+#     string and accept a realistic endpoint. An extraction that grabbed the wrong lines,
+#     or a predicate degraded to always-true / always-false, fails there — this repo has
+#     already been bitten by a check that passed when perturbed because it asked a question
+#     that was always satisfiable.
+# There is deliberately NO second implementation of the matching logic anywhere in this file.
+#
+# THE INDEPENDENT DETECTOR (the other half of the comparison)
+# ----------------------------------------------------------
+# Asking "does the guard agree with the guard?" proves nothing, so the set of
+# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the
+# UNION of two independent sources:
+#   * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
+#     CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates and the empty
+#     string. These are NOT derived from the guard, so DELETING a guard arm cannot make a
+#     value stop being placeholder-shaped — it makes assertion (1) fail, which is the point.
+#     A value using an idiom the guard does not know is a finding either way: fix it by
+#     giving the knob a real value, or by teaching the guard the idiom.
+#   * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED.
+#     Stripping the anchors is what makes the comparison non-circular in the dimension that
+#     actually broke: the detector matches a word ANYWHERE, case-insensitively, and the
+#     guard must agree. A prefix-anchored guard cannot pass.
+#
+# GIVEN the repository as authored,
+# WHEN  the real predicate(s) are extracted from bootstrap/standup.sh and executed,
+# THEN  (1) VALUES: for every knob a guard is responsible for, every authored value — the
+#           base default AND each environment overlay's value — that the independent
+#           detector calls placeholder-shaped is REFUSED by the real guard, and every value
+#           it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero
+#           suspended forever, which is the same class of silent failure);
+#       (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a
+#           placeholder-shaped value for a knob that no guard refuses. (Base alone is
+#           exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` —
+#           and its loud sentinels exist precisely to be overridden. A base placeholder an
+#           overlay forgets to override IS caught, because it becomes that overlay's
+#           effective value.)
+#       (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`,
+#           following one hop of variable aliasing) applies a placeholder guard to it. This
+#           is the "all three guards died at once" regression stated as a property: the
+#           guard being correct is worthless if a consumer stops calling it.
+#       (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
+#           value carrying that word in the MIDDLE (not at the start) — the precise defect
+#           of 2026-09-05 — while still accepting a realistic endpoint.
+#       (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim
+#           from the file and self-contained, the canary holds, no divergent second copy of
+#           the matching logic exists elsewhere in standup.sh, and a floor on the number of
+#           assertions run keeps a pass from degenerating into a no-op.
+#   AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
+#           ci/gate.sh sets the merge block when that stage fails.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps)
+#   - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
+#     that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
+#     run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
+#   - It does not execute standup.sh's dispatch path, so it cannot prove the guard is
+#     REACHED in a real run — only that, when reached, it refuses what it must. The
+#     consumer property (3) is a static approximation of reachability: it is a text-level
+#     taint trace over one file, not an interpreter, and a guard whose result is computed
+#     and then ignored would satisfy it.
+#   - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
+#     EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
+#     outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
+#     asserted by the node_baseline role itself, at play time).
+#   - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit
+#     to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so
+#     the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh
+#     load_touched_requirements) — this check runs. An edit to clusters/* maps to
+#     R-ENV-02, which this check carries, so a cluster-vars value change also selects it.
+#     A change that touches standup.sh ALONGSIDE some unrelated mapped path (say
+#     platform/cilium/) would scope the run away from this check; closing that needs a
+#     `bootstrap/*` entry in the path map, which is not this check's file to edit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}"
+
+cd "${ROOT}"
+
+if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then
+  echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2
+  exit 3
+fi
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT
+
+# ---------------------------------------------------------------------------------------
+# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files,
+# and run every assertion that does not need a shell. Writes:
+#   ${WORK}/guards.sh     the predicate source lines, verbatim, and nothing else
+#   ${WORK}/corpus.tsv    <guard>\t<expect: placeholder|usable>\t<origin>\t<value>
+#   ${WORK}/findings.txt  one line per failed assertion
+#   ${WORK}/counts.txt    <assertions-run>
+# ---------------------------------------------------------------------------------------
+"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY'
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def bail(msg):
+    """A failure that makes the rest of the check meaningless: report and stop."""
+    print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr)
+    (WORK / "findings.txt").write_text(msg + "\n")
+    (WORK / "counts.txt").write_text("0\n")
+    (WORK / "corpus.tsv").write_text("")
+    (WORK / "guards.sh").write_text("")
+    raise SystemExit(1)
+
+
+TEXT = (ROOT / STANDUP_REL).read_text()
+LINES = TEXT.splitlines()
+
+# ------------------------------------------------------------------ (5) extract the guard
+DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
+defs: dict[str, list[int]] = {}
+for i, line in enumerate(LINES):
+    m = DEF_RE.match(line)
+    if m:
+        defs.setdefault(m.group(1), []).append(i)
+
+if not defs:
+    bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed "
+         "(update this check's discovery pattern in the SAME change) or it was DELETED, which "
+         "is the 2026-09-05 failure with the guard removed instead of disarmed.")
+
+# An "argument-shaped" call: <name> "<something>"
+CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
+
+guard_src: list[str] = []
+guard_words: dict[str, list[str]] = {}
+ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
+ALLOWED = (
+    re.compile(r"^$"),
+    re.compile(r"^#"),
+    re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
+    re.compile(r"^esac$"),
+    re.compile(r"^return\s+[01]$"),
+    ARM_RE,
+)
+
+for name, at in sorted(defs.items()):
+    check(len(at) == 1,
+          f"{STANDUP_REL} defines {name}() {len(at)} times (lines "
+          f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
+          "this check can no longer say which one it extracted. Keep exactly one definition.")
+    if len(at) != 1:
+        continue
+    start = at[0]
+    end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
+    if end is None:
+        bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the "
+             "extraction boundary this check relies on is gone. Restore the conventional "
+             "layout, or move the predicate into a sourceable library and point this check at it.")
+    body = LINES[start:end + 1]
+    block = "\n".join(body)
+    check(block in TEXT,
+          f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; "
+          "the harness would be testing something the script does not contain")
+
+    words: list[str] = []
+    for raw in body[1:-1]:
+        s = raw.strip()
+        if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
+            bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED "
+                 f"case/return predicate (offending line: {s!r}). This check exercises the REAL "
+                 "function by sourcing its own lines; a body that calls out to anything else "
+                 "cannot be extracted faithfully, and silently testing a COPY of the logic is "
+                 "exactly the failure mode this check exists to prevent. REMEDY: move the "
+                 "predicate into a tiny library that standup.sh sources (e.g. "
+                 "bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do "
+                 "NOT reimplement the matching here.")
+        m = ARM_RE.match(s)
+        if not m:
+            continue
+        for pat in m.group("pats").split("|"):
+            pat = pat.strip()
+            if pat in ('""', "''", ""):
+                continue                      # the empty-value arm: covered by the canary
+            words.append(pat.strip("'\"").strip("*"))
+    check(len(words) >= 1,
+          f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
+          "loud placeholder string has been removed, so the guard now only refuses the empty "
+          "value and any 'TBD'-style endpoint would sail straight through it")
+    guard_words[name] = words
+    guard_src.append(block)
+
+(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n")
+
+# ------------------------------------------------- no divergent second copy of the logic
+# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means
+# a second, independently-drifting implementation of the same test (which is how three
+# guards came to disagree with one value in the first place).
+guard_line_nums = {j for name, at in defs.items() if len(at) == 1
+                   for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
+                                              if LINES[k] == "}") + 1)}
+all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
+divergent: list[str] = []
+for j, line in enumerate(LINES):
+    if j in guard_line_nums or line.strip().startswith("#"):
+        continue
+    code = line.split("#", 1)[0]
+    hit = [w for w in all_words if w in code.upper()]
+    if hit:
+        divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
+check(not divergent,
+      "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
+      f"predicate — a SECOND implementation of the same test, which is precisely how a value "
+      f"change disarmed three guards at once. Call the predicate instead. Sites: "
+      + " | ".join(divergent))
+
+# ------------------------------------------------------------- cluster-vars, as authored
+CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+if not CV.is_file():
+    bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is "
+         "missing and this check cannot mean anything")
+
+base: dict[str, str] = {}
+for d in yaml.safe_load_all(CV.read_text()):
+    if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
+            and (d.get("metadata") or {}).get("name") == "cluster-vars":
+        base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()}
+check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
+                     "ConfigMap parse failed, so every value assertion below would be vacuous")
+
+overlays: dict[str, dict[str, str]] = {}
+for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+    env = overlay.parent.name
+    if env == "base":
+        continue
+    eff = dict(base)
+    try:
+        k = yaml.safe_load(overlay.read_text()) or {}
+    except Exception as exc:                                   # noqa: BLE001
+        check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this "
+                     "check cannot compute that overlay's effective values")
+        continue
+    for p in (k.get("patches") or []):
+        tgt = p.get("target") or {}
+        if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
+            continue
+        try:
+            ops = yaml.safe_load(p.get("patch") or "") or []
+        except Exception:                                      # noqa: BLE001
+            continue
+        for op in ops if isinstance(ops, list) else []:
+            if not isinstance(op, dict):
+                continue
+            m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
+            if m and str(op.get("op", "")) in ("replace", "add"):
+                eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
+    overlays[env] = eff
+check(len(overlays) >= 1,
+      "no clusters/<env>/kustomization.yaml overlay was parsed — assertion (2) would be vacuous")
+
+# ------------------------------------------------ which knobs is a guard responsible for?
+# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a
+# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here.
+FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
+funcs: dict[str, list[str]] = {}
+i = 0
+while i < len(LINES):
+    m = FUNC_RE.match(LINES[i])
+    if m:
+        end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
+        funcs[m.group(1)] = LINES[i + 1:end]
+        i = end + 1
+        continue
+    i += 1
+check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the "
+                      "consumer assertion (3) would be vacuous")
+
+CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
+ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
+
+guarded_keys: set[str] = set()
+consumers: list[tuple[str, str, bool]] = []      # (func, key, guarded-in-this-func)
+for fname, body in funcs.items():
+    text = "\n".join(body)
+    keys_of_var: dict[str, set[str]] = {}
+    for m in CVAR_RE.finditer(text):
+        keys_of_var.setdefault(m.group(1), set()).add(m.group(2))
+    # one hop of aliasing: ep="${live}" carries live's taint
+    for _ in range(3):
+        for m in ALIAS_RE.finditer(text):
+            src = keys_of_var.get(m.group(2))
+            if src:
+                keys_of_var.setdefault(m.group(1), set()).update(src)
+    called_on = {m.group(2) for m in CALL_RE.finditer(text)}
+    called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
+    # Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its
+    # value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts.
+    guarded_here: dict[str, bool] = {}
+    for var, keys in keys_of_var.items():
+        for key in keys:
+            guarded_here[key] = guarded_here.get(key, False) or (var in called_vars)
+    for key, g in guarded_here.items():
+        consumers.append((fname, key, g))
+        if g:
+            guarded_keys.add(key)
+
+if not guarded_keys:
+    bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. "
+         "Either the guards were removed, or the way the script reads cluster-vars changed and "
+         "this check's derivation must be updated in the SAME change — it must never silently "
+         "verify an empty set.")
+
+# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it.
+for fname, key, guarded in sorted(consumers):
+    if key not in guarded_keys:
+        continue                       # nothing claims responsibility for this knob at all
+    check(guarded,
+          f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
+          "*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
+          "of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
+          "the guard is that failure in one function.")
+
+# ---------------------------------------------------------------- independent detector
+# Deliberately NOT derived from the guard alone (see the header): SEED is universal
+# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose
+# ANCHORING is wrong still has its vocabulary tested against real values.
+SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
+        "CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
+VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
+ANGLE = re.compile(r"<[^>\s]+>")
+
+
+def placeholder_shaped(value: str) -> bool:
+    if value.strip() == "":
+        return True                                   # an unset endpoint is not a target
+    up = value.upper()
+    return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
+
+
+rows: list[tuple[str, str, str, str]] = []            # guard, expect, origin, value
+GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None
+
+# (1) VALUES: base default and every overlay's value, for every guarded knob.
+for key in sorted(guarded_keys):
+    seen: list[tuple[str, str]] = []
+    if key in base:
+        seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key]))
+    for env, eff in sorted(overlays.items()):
+        seen.append((f"clusters/{env} effective:{key}", eff.get(key, "")))
+    check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
+                      "protects a knob nothing authors")
+    for origin, value in seen:
+        expect = "placeholder" if placeholder_shaped(value) else "usable"
+        for name in sorted(guard_words):
+            rows.append((name, expect, origin, value))
+
+# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob.
+for env, eff in sorted(overlays.items()):
+    for key, value in sorted(eff.items()):
+        if key in guarded_keys or not placeholder_shaped(value):
+            continue
+        check(False,
+              f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
+              f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it "
+              "reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay "
+              "that ships a placeholder no guard recognises is the 2026-09-05 failure with a "
+              "different knob's name on it.")
+
+# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
+#     and a realistic endpoint must still be accepted.
+for name, words in sorted(guard_words.items()):
+    for w in sorted(set(words)):
+        rows.append((name, "placeholder",
+                     f"anchoring:{name}:mid-string({w})",
+                     f"https://account-{w}-id.r2.cloudflarestorage.com"))
+        rows.append((name, "placeholder", f"anchoring:{name}:tail({w})",
+                     f"https://r2-endpoint.example.invalid/{w}"))
+    rows.append((name, "placeholder", f"canary:{name}:empty", ""))
+    rows.append((name, "usable", f"canary:{name}:real-r2",
+                 "https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"))
+    rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000"))
+
+# Every guarded knob must reach the oracle once per authored source (base + each overlay),
+# so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
+authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
+expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
+check(len(authored) == expected_rows,
+      f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
+      f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
+      f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from "
+      "an overlay's effective values, and the coupling is only partly exercised")
+
+# ------------------------------------------------------- (6) the check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+check("--layer l0" in L0,
+      "ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate")
+check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# --------------------------------------------------------------------------- handoff
+with (WORK / "corpus.tsv").open("w") as fh:
+    for name, expect, origin, value in rows:
+        fh.write(f"{name}\t{expect}\t{origin}\t{value}\n")
+(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
+(WORK / "counts.txt").write_text(f"{CHECKS}\n")
+
+print(f"guards: {', '.join(sorted(guard_words))}  "
+      f"vocabulary: {', '.join(VOCAB)}")
+print(f"guarded knobs: {', '.join(sorted(guarded_keys))}  "
+      f"consumers traced: {len(consumers)}  overlays: {', '.join(sorted(overlays))}")
+for name, expect, origin, value in rows:
+    if not origin.startswith(("anchoring:", "canary:")):
+        print(f"  authored  {origin} = {value!r} -> expect {expect}")
+PY
+
+# ---------------------------------------------------------------------------------------
+# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
+# ---------------------------------------------------------------------------------------
+# shellcheck disable=SC1090,SC1091
+source "${WORK}/guards.sh"
+
+oracle_failures=0
+oracle_rows=0
+while IFS=$'\t' read -r guard expect origin value; do
+  [ -n "${guard}" ] || continue
+  oracle_rows=$((oracle_rows + 1))
+  if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi
+  if [ "${verdict}" != "${expect}" ]; then
+    oracle_failures=$((oracle_failures + 1))
+    if [ "${expect}" = placeholder ]; then
+      printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
+    else
+      printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
+    fi
+  fi
+done < "${WORK}/corpus.tsv"
+
+# ------------------------------------------------------------------------------ verdict
+py_checks="$(cat "${WORK}/counts.txt")"
+total=$((py_checks + oracle_rows))
+findings="$(cat "${WORK}/findings.txt")"
+
+# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it
+# asked a question that was always satisfiable. The floors below are the minimum evidence
+# for the verdict to mean anything, not a style rule.
+if [ "${oracle_rows}" -lt 6 ]; then
+  findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised
+"
+fi
+if [ "${total}" -lt 20 ]; then
+  findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless
+"
+fi
+
+if [ -n "${findings}" ]; then
+  echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2
+  printf '%s' "${findings}" | sed 's/^/  - /' >&2
+  exit 1
+fi
+
+echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."

conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml

@@ -0,0 +1,77 @@
+# =============================================================================
+# L0 STATIC — prod is bare metal, and is not a development surface  (R-ENV-03) [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
+#          storage; it MUST NOT be an active development surface until the roadmap
+#          declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
+#          Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
+#          differs", "Never develop against prod"); roadmap/phases.md Phase 7.
+#
+# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
+# governance rule is verified through the property that ENFORCES it, never by observing
+# that nobody misbehaved today. Here that property is structural: prod receives the base
+# configuration plus environment-scoped overrides and nothing else, from the same branch
+# and the same Ansible code as dev, so there is no place to put prod-only work at all.
+#
+# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
+# group_vars retargets the k3s roles away from the global default), the prod overlay is
+# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
+# node count that inventory resolves to. No hostname, address or environment name is
+# written into the test.
+#
+# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
+# WHEN  the prod inventory, overlay, IaC module and readiness record are inspected — and
+#       the shared provisioning chain is actually retargeted at prod with --list-hosts,
+# THEN  prod resolves to exactly three nodes reached directly on their own sshd; no IaC
+#       resource creates any of them (bare metal is imaged, not instantiated); prod's
+#       Longhorn path is node-local with one replica per node and the default
+#       StorageClass is the local driver; prod runs the same playbooks/roles as dev,
+#       retargeted only by variables the shared code already defines; the prod overlay
+#       carries only the base plus environment-scoped patches on the same Git ref dev
+#       tracks, with no prod-only manifest anywhere in the shared configuration; and prod
+#       artifacts exist only alongside the operator's recorded opening of the prod phase.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
+# where present — ansible-playbook in --list-hosts mode, which opens no connection);
+# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
+# wraps the SAME script.
+#
+# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
+#   * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
+#     SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
+#     node-health/BIOS surveys that establish it write to gitignored artifacts, and the
+#     Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
+#   * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
+#     disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
+#     HA test; asserting it here would also make this test environment-specific, since
+#     the same suite must pass against dev.
+#   * Git HISTORY: this is the working tree at HEAD.
+#   * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
+#     the recorded declaration.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: prod-baremetal-not-dev-surface
+  labels:
+    conformance.home.k3s/R-ENV-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-03"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
+      inspected, then prod is authored as three bare-metal nodes with node-local storage,
+      bootstrapped by dev's own code retargeted by variables, receiving only
+      environment-scoped overrides on the same branch — so there is nowhere to develop
+      against prod — and it exists only because the roadmap phase was opened.
+spec:
+  steps:
+    - name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
+        conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/prod-baremetal-not-dev-surface/check.sh

@@ -0,0 +1,528 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
+# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
+#         MUST NOT be an active development surface until the roadmap declares it ready."
+#         — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
+#         bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
+#         "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
+#         step differs", and "**Never develop against prod.** Prod hardware is a target, not
+#         a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
+#         "operator declares prod hardware ready"; note "Do not begin building on prod
+#         hardware before this phase is explicitly opened".)
+#
+# HOW A "MUST NOT" IS MADE TESTABLE
+# --------------------------------
+# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
+# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
+# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
+# be. In this repo that property is structural — prod receives the base configuration plus
+# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
+# even if someone wanted to. This check asserts that, together with the first clause's
+# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
+# bootstrap code as dev).
+#
+# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
+# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
+# see the gap list. What IS checkable is that the repository describes prod that way and
+# that no IaC creates prod nodes as virtual machines.
+#
+# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
+# WHEN  the prod inventory, the prod overlay, the IaC module and the roadmap record are
+#       inspected — and, where offline execution is possible, the provisioning chain is
+#       actually retargeted at prod with `--list-hosts`,
+# THEN  (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
+#           (no NAT indirection, i.e. real machines on the LAN);
+#       (2) NO IaC resource creates any prod node: the VM module's generated node names are
+#           disjoint from the prod host set and no prod hostname appears anywhere under
+#           infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
+#       (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
+#           local path, and the replica count equals the node count (one replica per node's
+#           own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
+#           is not counted against this;
+#       (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
+#           by variables only — every prod group_vars key is an existing role/global
+#           variable (an invented prod-only knob would be a fork), and no prod-only role
+#           or playbook exists;
+#       (5) prod is NOT a development surface: clusters/prod carries only the base plus
+#           sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
+#           (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
+#           prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
+#           and no prod-only manifest exists anywhere in the rendered configuration;
+#       (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
+#           carries the phase that opens prod build (checked when the spec repo is mounted),
+#           and the repository records that the operator opened it.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
+#   - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
+#     SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
+#     node-health and BIOS surveys that establish it write to gitignored artifacts, so
+#     nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
+#     "am I virtual" fact either — a bare-metal check would need a privileged host-path
+#     probe of DMI data, which is a heavier and more invasive test than the claim warrants.
+#   - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
+#     disks at the local path). That is cluster state and belongs to R-PLAT-01's
+#     (currently uncovered) HA test; asserting it here would also make this test
+#     environment-specific, since the same suite must pass against dev.
+#   - That prod's HISTORY contains no development. This is the working tree at HEAD.
+#   - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
+#     prod work is accompanied by the operator's recorded declaration.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
+export SPEC_REPO
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import os
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+NOTES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def note(msg):
+    NOTES.append(msg)
+
+
+# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
+# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
+# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
+# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
+# narrow: every other non-zero exit is a real failure. The parse assertions always run.
+ENV_ONLY_FAILURE = re.compile(
+    r"vault password|vault-id|Attempting to decrypt"
+    r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
+
+ANS = ROOT / "infra" / "ansible"
+INV = ANS / "inventory" / "hosts.ini"
+GV = ANS / "inventory" / "group_vars"
+TOFU = ROOT / "infra" / "tofu"
+check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
+if not INV.is_file():
+    print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
+    raise SystemExit(1)
+
+# --------------------------------------------------------------- inventory parsing
+ini_groups: dict[str, list[dict]] = {}
+children: dict[str, list[str]] = {}
+cur = None
+for line in INV.read_text().splitlines():
+    s = line.strip()
+    if not s or s.startswith("#"):
+        continue
+    m = re.match(r"^\[([^\]]+)\]$", s)
+    if m:
+        cur = m.group(1)
+        continue
+    if cur is None or cur.endswith(":vars"):
+        continue
+    if cur.endswith(":children"):
+        children.setdefault(cur.split(":")[0], []).append(s.split()[0])
+        continue
+    parts = s.split()
+    host = {"name": parts[0]}
+    for kv in parts[1:]:
+        if "=" in kv:
+            k, v = kv.split("=", 1)
+            host[k] = v
+    ini_groups.setdefault(cur, []).append(host)
+
+
+def hosts_in(group: str, seen=None) -> list[dict]:
+    seen = seen or set()
+    if group in seen:
+        return []
+    seen.add(group)
+    out = list(ini_groups.get(group, []))
+    for child in children.get(group, []):
+        out += hosts_in(child, seen)
+    return out
+
+
+def load_gv(group: str) -> dict:
+    d: dict = {}
+    for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
+        if p.is_file():
+            try:
+                d.update(yaml.safe_load(p.read_text()) or {})
+            except Exception:
+                pass
+    return d
+
+
+# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
+# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
+# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
+ALL_VARS = load_gv("all")
+dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
+candidates = []
+for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
+    g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
+    if g == "all":
+        continue
+    v = load_gv(g)
+    if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
+        candidates.append((g, v))
+check(len(candidates) == 1,
+      f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
+      f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
+      "the same roles as dev")
+if not candidates:
+    print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+PROD_GROUP, PROD_VARS = candidates[0]
+PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
+prod_hosts = hosts_in(PROD_GROUP)
+prod_names = sorted(h["name"] for h in prod_hosts)
+
+# ------------------------------------------------- (1) exactly three, directly reachable
+check(len(prod_names) == 3,
+      f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
+      "says prod is a 3× mini-PC cluster")
+dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
+for h in prod_hosts:
+    # Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
+    # NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
+    # indirection a hypervisor-hosted estate needs and bare metal does not.
+    check(str(h.get("ansible_port", "22")) == "22",
+          f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
+          "implies an indirection (NAT to a virtual estate), not a machine on the LAN")
+    check(h.get("k3s_node_ip") == h.get("ansible_host"),
+          f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
+          f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
+          "bare-metal node's SSH address IS its cluster address")
+
+# ------------------------------------------------------ (2) nothing creates prod as a VM
+TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
+    if TOFU.is_dir() else ""
+check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
+
+
+def tf_default(var: str):
+    m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
+    if not m:
+        return None
+    d = re.search(r"default\s*=\s*(.+)", m.group(1))
+    return d.group(1).strip() if d else None
+
+
+prefix = (tf_default("node_name_prefix") or "").strip('"')
+try:
+    node_count = int((tf_default("node_count") or "0").strip())
+except ValueError:
+    node_count = 0
+iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
+check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
+check(iac_names.isdisjoint(set(prod_names)),
+      f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
+      "— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
+      "hypervisor module")
+for name in prod_names:
+    check(name not in TF,
+          f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
+          "the VM module at all")
+# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
+# still a VM).
+tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
+prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
+check(tf_ips.isdisjoint(prod_ips),
+      f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
+
+# --------------------------------------------------------------- (3) node-local storage
+# Every environment's value for a cluster-vars knob (base default + each overlay patch).
+def env_values(key: str) -> dict[str, str]:
+    vals: dict[str, str] = {}
+    base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+    if base.is_file():
+        for d in yaml.safe_load_all(base.read_text()):
+            if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
+                    and (d.get("metadata") or {}).get("name") == "cluster-vars":
+                v = (d.get("data") or {}).get(key)
+                if v is not None:
+                    vals["base"] = str(v)
+    for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+        env = overlay.parent.name
+        if env == "base":
+            continue
+        try:
+            k = yaml.safe_load(overlay.read_text()) or {}
+        except Exception:
+            continue
+        for p in (k.get("patches") or []):
+            tgt = p.get("target") or {}
+            if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
+                continue
+            try:
+                ops = yaml.safe_load(p.get("patch") or "") or []
+            except Exception:
+                continue
+            for op in ops if isinstance(ops, list) else []:
+                if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
+                    vals[env] = str(op.get("value"))
+    return vals
+
+
+# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
+cluster_envs = env_values("CLUSTER_ENV")
+prod_overlays = [e for e, v in cluster_envs.items()
+                 if e != "base" and v != cluster_envs.get("base")]
+check(len(prod_overlays) >= 1,
+      "no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
+prod_overlay = None
+for e in prod_overlays:
+    if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
+        prod_overlay = e
+check(prod_overlay is not None,
+      f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
+      "cluster config and the machine inventory disagree about which estate is prod")
+
+data_paths = env_values("LONGHORN_DATA_PATH")
+replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
+if prod_overlay:
+    p = data_paths.get(prod_overlay, data_paths.get("base", ""))
+    check(p.startswith("/"),
+          f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
+          "node-local filesystem path")
+    check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
+          f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
+    r = replicas.get(prod_overlay, replicas.get("base", ""))
+    check(r.isdigit() and int(r) == len(prod_names),
+          f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
+          "'local storage' on a 3-node estate means one replica per node's own disk")
+# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
+# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
+# counted here.
+default_scs = []
+for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
+                        capture_output=True, text=True, check=True).stdout.split("\0"):
+    if not f:
+        continue
+    try:
+        for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
+            if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
+                    str(((d.get("metadata") or {}).get("annotations") or {})
+                        .get("storageclass.kubernetes.io/is-default-class", "")) == "true":
+                default_scs.append((f, str(d.get("provisioner", ""))))
+    except Exception:
+        continue
+check(len(default_scs) == 1,
+      f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
+      "land on by default must be unambiguous")
+for f, prov in default_scs:
+    check(prov == "driver.longhorn.io",
+          f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
+          "tier would not be its own local disks")
+
+# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
+PLAYBOOKS = ANS / "playbooks"
+ROLES = ANS / "roles"
+prod_token = PROD_GROUP.replace("k3s_", "")
+for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
+    for entry in sorted(d.iterdir()) if d.is_dir() else []:
+        check(prod_token not in entry.name.lower(),
+              f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
+              "the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
+# Every prod group_vars key must be a variable the shared code already knows: a role default,
+# a global, or an ansible built-in. An invented key is a fork of the role by another name.
+known: set[str] = set(ALL_VARS)
+for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
+    for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
+        try:
+            known |= set((yaml.safe_load(p.read_text()) or {}).keys())
+        except Exception:
+            pass
+check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
+for k in sorted(PROD_VARS):
+    check(k.startswith("ansible_") or k in known,
+          f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
+          "configuration that the dev path never exercises is developing against prod")
+# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
+check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
+      f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
+
+# ---------------------------------------- (5) prod is not a development surface (overlay)
+if prod_overlay:
+    odir = ROOT / "clusters" / prod_overlay
+    files = sorted(p.name for p in odir.iterdir() if p.is_file())
+    check(files == ["kustomization.yaml", "sync.yaml"],
+          f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
+          "kustomization and its own root sync; anything else is configuration authored FOR PROD")
+    k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
+    check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
+          f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
+          "plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
+    # Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
+    ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
+    for p in (k.get("patches") or []):
+        tgt = p.get("target") or {}
+        body = p.get("patch") or ""
+        kind = tgt.get("kind")
+        if kind is None:                       # inline patch: kind comes from the body
+            try:
+                doc = yaml.safe_load(body) or {}
+            except Exception:
+                doc = {}
+            kind = doc.get("kind")
+            name = (doc.get("metadata") or {}).get("name")
+            if kind == "Kustomization":
+                # Flux Kustomization patches are allowed ONLY to wire sops decryption —
+                # anything else (path, interval, images, values) changes what prod RUNS.
+                keys = set((doc.get("spec") or {}).keys())
+                check(keys <= {"decryption"},
+                      f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
+                      "— only `decryption` (the per-cluster sops key) is an environment-scoped "
+                      "override; changing anything else is developing against prod")
+                continue
+            check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
+                  f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
+                  "environment-scoped object — prod must not carry configuration of its own")
+        else:
+            check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
+                  f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
+                  "an environment-scoped object — prod must not carry configuration of its own")
+
+    # Same Git ref as the other environment(s): a prod-only branch is a development line
+    # that only prod runs, which is exactly what R-ENV-03 forbids.
+    def git_ref(overlay: str):
+        kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
+        for p in (kk.get("patches") or []):
+            body = p.get("patch") or ""
+            try:
+                doc = yaml.safe_load(body) or {}
+            except Exception:
+                continue
+            if doc.get("kind") == "GitRepository":
+                return (doc.get("spec") or {}).get("ref")
+        return None
+
+    others = [o.name for o in (ROOT / "clusters").iterdir()
+              if o.is_dir() and o.name not in ("base", prod_overlay)]
+    prod_ref = git_ref(prod_overlay)
+    check(prod_ref is not None,
+          f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
+          "points at, with no record of what it runs")
+    for o in others:
+        check(git_ref(o) == prod_ref,
+              f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
+              "development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
+
+# No prod-only manifest anywhere in the rendered configuration.
+tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                                     capture_output=True, text=True, check=True
+                                     ).stdout.split("\0") if f]
+RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
+for f in tracked:
+    if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
+                                                 f, re.I):
+        check(False,
+              f"{f} is a prod-specific file inside the shared configuration — every environment "
+              "renders these paths, so prod-only content here is development against prod")
+check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
+
+# --------------------------- (6) prod work exists only with a declaration of readiness
+# In-repo: the operator's recorded decision to open the prod phase.
+log = ROOT / "DECISIONS-LOG.md"
+check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
+      "DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
+      "inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
+# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
+spec = Path(os.environ.get("SPEC_REPO", ""))
+phases = spec / "roadmap" / "phases.md"
+if phases.is_file():
+    txt = phases.read_text(errors="replace")
+    check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
+          re.search(r"prod", txt, re.I) is not None,
+          "the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
+          "to does not exist")
+    check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
+          is not None,
+          "the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
+          "names would be unenforceable")
+else:
+    note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
+         "(the in-repo declaration still was)")
+
+# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
+if shutil.which("ansible-playbook") is None:
+    note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
+         "prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
+else:
+    sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
+    args = ["--list-hosts", "playbooks/site.yml",
+            "-e", f"k3s_hosts={PROD_GROUP}",
+            "-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
+    if sec_groups:
+        args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
+    lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
+    out = (lh.stderr or "") + (lh.stdout or "")
+    if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
+        # See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
+        note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
+             "passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
+             "driven; every parse assertion above still ran")
+    else:
+        check(lh.returncode == 0,
+              f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
+              "the same Ansible bootstrap to run for prod")
+        seen: set[str] = set()
+        in_hosts = False
+        for line in lh.stdout.splitlines():
+            if re.match(r"\s*hosts \(\d+\):", line):
+                in_hosts = True
+                continue
+            if re.search(r"play #\d+", line):
+                in_hosts = False
+                continue
+            if in_hosts and line.strip():
+                seen.add(line.strip())
+        check(seen == set(prod_names),
+              f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
+              f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
+
+# --------------------------------------------------------------------------- verdict
+check(CHECKS >= 25,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+for n in NOTES:
+    print(f"  note: {n}")
+print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
+      f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
+      f"plus environment-scoped patches only, local Longhorn path "
+      f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
+      f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
+      "only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
+      "live cluster's shape is NOT asserted here (see the header's gap list).")
+PY

conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml

@@ -0,0 +1,69 @@
+# =============================================================================
+# L0 STATIC — certificate lifecycle needs no manual step  (R-TLS-02)   [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+# Spec:    docs/00 R-TLS-02 ("Certificates MUST be automatically issued, renewed, and
+#          revoked without manual steps"); mechanism in docs/08 + ADR-0006.
+#
+# R-TLS-02 is one adverbial clause plus three runtime verbs. This test asserts the
+# clause — "without manual steps" — which in a GitOps platform is a property of the
+# REPOSITORY: a human step in the certificate path leaves evidence in git (committed
+# cert material, a Gateway serving a Secret nothing declares, an issuer with no
+# automated solver, a credential never delivered, auto-approval switched off, or CRs
+# nothing reconciles). The three verbs are observed by the companion L2 test.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the issuer a Certificate names is `${ACME_CLUSTER_ISSUER}`,
+# so the check resolves it against the BASE cluster-vars default AND every overlay's value and
+# requires each resolved issuer to exist. dev's issuer and prod's issuer are both verified from
+# one authored assertion, with no hostname, address or environment name written down.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every serving-TLS reference, Certificate, Issuer and the cert-manager release
+#       is inspected offline,
+# THEN  every Gateway listener that terminates TLS serves a Secret a tracked Certificate
+#       produces; no certificate material is committed anywhere; every Certificate's
+#       issuer resolves — in EVERY environment — to a tracked ACME issuer with an
+#       automated dns01/http01 solver whose credential is SOPS-delivered from git;
+#       cert-manager's automatic request approval is not disabled; and every one of
+#       those objects is rendered by a Flux Kustomization, so nothing in the chain
+#       depends on a human running kubectl.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML, no
+# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and
+# this Chainsaw Test wraps the SAME script.
+#
+# KNOWN, DELIBERATE GAPS (named in check.sh's header too — this test claims NONE of them):
+#   * ISSUED / RENEWED as runtime events: L2 (l2-behavioral/cert-lifecycle-automated).
+#   * REVOKED: not proven at any layer today. cert-manager performs no ACME revocation,
+#     and the spec's own reading (docs/08: "removing/rotating a certificate is a
+#     declarative change") needs `enableCertificateOwnerRef`, which the tracked
+#     cert-manager HelmRelease does not set — so a removed Certificate leaves its
+#     serving Secret behind. The L2 test asserts the withdrawal behaviour and is
+#     EXPECTED TO FAIL until that knob is set; escalated, not softened (R-TEST-06).
+#   * That the live cluster is serving the tracked Certificate (cluster state).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: tls-lifecycle-automated
+  labels:
+    conformance.home.k3s/R-TLS-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-TLS-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the tracked tree, when the certificate path is inspected offline, then no
+      manual step exists anywhere in it: serving Secrets are cert-manager-produced, no
+      cert material is committed, every environment's issuer is a tracked ACME issuer
+      with an automated solver and a git-delivered credential, auto-approval is on, and
+      the whole chain is Flux-reconciled.
+spec:
+  steps:
+    - name: then-no-manual-step-exists-in-the-certificate-path
+      description: >-
+        Run the offline assertion script over the tracked tree (same script ci/l0.sh
+        and conformance/run.sh --layer l0 execute).
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/tls-lifecycle-automated/check.sh

@@ -0,0 +1,478 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — tls-lifecycle-automated
+# Serves: R-TLS-02 ("Certificates MUST be automatically **issued, renewed, and revoked**
+#         without manual steps." — spec docs/00-vision-and-scope.md; mechanism in
+#         docs/08-tls-and-certificates.md, ADR-0006.)
+#
+# WHICH CLAUSE THIS LAYER CAN PROVE — AND WHICH IT CANNOT
+# ------------------------------------------------------
+# R-TLS-02 has one adverbial clause ("without manual steps") and three verbs
+# ("issued", "renewed", "revoked"). In a GitOps platform the *without manual steps*
+# clause IS a repository property, and it is the half that can be proven with no
+# cluster: if a human had to mint, install, or approve a serving certificate, the
+# evidence of that human step would be IN THE REPO — a committed `tls.crt`, a Gateway
+# pointing at a Secret nothing declares, an issuer that no automated solver can
+# satisfy, a per-cert credential handed over out of band, or an admission path that
+# waits for a manual CertificateRequest approval. This check asserts that none of
+# those exist and that the automated path is completely wired instead.
+#
+# The three VERBS are runtime events. They are asserted by the companion L2 test
+# (conformance/tests/l2-behavioral/cert-lifecycle-automated), which observes an
+# issuance, an unattended renewal, and the withdrawal of a removed certificate. This
+# check does NOT claim them; see "WHAT THIS DOES *NOT* PROVE" below.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every serving-TLS reference, Certificate, Issuer and the cert-manager release
+#       are inspected offline,
+# THEN  (1) at least one cert-manager `Certificate` is authored (vacuity floor: a repo
+#           with no Certificate cannot pass this check by having nothing to check);
+#       (2) every Gateway HTTPS listener that TERMINATES TLS references ONLY Secrets
+#           that a tracked Certificate declares as its `secretName` — so the serving
+#           credential is produced by the controller, never dropped in by hand;
+#       (3) NO certificate material is tracked anywhere: no Secret manifest of type
+#           kubernetes.io/tls, no tls.crt/tls.key/ca.crt key in any committed Secret
+#           (plain OR SOPS-encrypted), and no PEM certificate body in any tracked
+#           YAML — i.e. issuance is not, and cannot silently become, a human step;
+#       (4) every Certificate's `issuerRef` resolves to a TRACKED issuer IN EVERY
+#           ENVIRONMENT — the `${ACME_CLUSTER_ISSUER}` token is resolved against the
+#           base cluster-vars default AND every overlay's value, so an environment
+#           whose issuer does not exist is caught here rather than at prod bring-up;
+#       (5) every issuer so referenced is an ACME issuer with a NON-EMPTY solver list
+#           whose every solver is an automated challenge type (dns01 / http01) — the
+#           same code path serves issuance AND renewal, so a solver a human must
+#           complete would defeat both;
+#       (6) each dns01 solver's provider credential is referenced BY NAME and that
+#           Secret is delivered SOPS-encrypted from git via a kustomization — the
+#           operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
+#       (7) cert-manager's automatic CertificateRequest approval is NOT disabled
+#           (`disableAutoApproval`), and any `approveSignerNames` narrowing still
+#           covers the ClusterIssuers actually used — a request that waits for a
+#           human approver is the definition of a manual step;
+#       (8) the whole chain is reconciled from git — cert-manager's HelmRelease, the
+#           ClusterIssuers and the Certificate are each included in a kustomization
+#           that a Flux Kustomization renders — so a deleted Certificate is
+#           re-created and re-issued with no human action;
+#       (9) no Certificate defeats renewal arithmetically (`renewBefore` >= `duration`
+#           would mean the renewal instant never arrives).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
+#   - ISSUED: that an issuance actually happens. Asserted at L2 (step 2 of
+#     l2-behavioral/cert-lifecycle-automated) and, for the real ACME/DNS-01 path,
+#     evidenced by l2-behavioral/trusted-tls serving a publicly-trusted cert.
+#   - RENEWED: that a renewal actually happens unattended. docs/08's acceptance gate
+#     is explicit that this must be "observed across a renewal" — L2 step 3 does that
+#     by fast-forwarding a certificate's renewal window; nothing static can.
+#   - REVOKED: NOT PROVEN AT ANY LAYER TODAY, and not asserted here. Two distinct
+#     readings, neither established: (a) ACME revocation — cert-manager does not
+#     revoke certificates at all, so no repo state could show it; (b) the spec's own
+#     reading in docs/08 ("removing/rotating a certificate is a declarative change")
+#     — withdrawal of a REMOVED certificate's serving material requires cert-manager's
+#     `enableCertificateOwnerRef`, which the tracked HelmRelease does NOT set (chart
+#     default false). L2 step 4 asserts the withdrawal behaviour and is expected to
+#     FAIL until that knob is set; escalated to the PM rather than softened (R-TEST-06).
+#   - That the tracked Certificate is the one the live Gateway is actually serving
+#     (cluster state), or that the ACME account/DNS credential is still valid.
+#   - Certificate EXPIRY ALERTING, which docs/08 delegates to R-OBS-04 (see
+#     l3-fault-injection/alerting).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def tracked() -> list[str]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                         capture_output=True, text=True, check=True).stdout
+    return [p for p in out.split("\0") if p]
+
+
+FILES = tracked()
+check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
+YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)$", f)]
+
+
+def docs_of(rel: str):
+    """Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
+    try:
+        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
+                if isinstance(d, dict)]
+    except Exception:
+        return []
+
+
+DOCS: list[tuple[str, dict]] = [(rel, d) for rel in YAML_FILES for d in docs_of(rel)]
+
+
+def of_kind(kind: str, group_prefix: str = "") -> list[tuple[str, dict]]:
+    out = []
+    for rel, d in DOCS:
+        if d.get("kind") != kind:
+            continue
+        if group_prefix and not str(d.get("apiVersion", "")).startswith(group_prefix):
+            continue
+        out.append((rel, d))
+    return out
+
+
+def ns_of(d: dict, default: str = "default") -> str:
+    return (d.get("metadata") or {}).get("namespace") or default
+
+
+def name_of(d: dict) -> str:
+    return (d.get("metadata") or {}).get("name") or "<unnamed>"
+
+
+# ------------------------------------------------------------ cluster-vars resolution
+# Every environment's value for a ${TOKEN}: the base default plus each overlay's JSON6902
+# `- op: replace / path: /data/KEY / value: ...` patch on the cluster-vars ConfigMap. This
+# is the same mechanism flux-postbuild-substitution audits; here we need the VALUES, so an
+# issuer name can be resolved per environment instead of being asserted for dev only.
+def env_values(key: str) -> dict[str, str]:
+    vals: dict[str, str] = {}
+    base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+    if base.is_file():
+        for d in yaml.safe_load_all(base.read_text()):
+            if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
+                    and name_of(d) == "cluster-vars":
+                v = (d.get("data") or {}).get(key)
+                if v is not None:
+                    vals["base"] = str(v)
+    for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+        env = overlay.parent.name
+        if env == "base":
+            continue
+        try:
+            k = yaml.safe_load(overlay.read_text()) or {}
+        except Exception:
+            continue
+        for p in (k.get("patches") or []):
+            tgt = p.get("target") or {}
+            if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
+                continue
+            try:
+                ops = yaml.safe_load(p.get("patch") or "") or []
+            except Exception:
+                continue
+            for op in ops if isinstance(ops, list) else []:
+                if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
+                    vals[env] = str(op.get("value"))
+    return vals
+
+
+TOKEN_RE = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
+
+
+def resolve_per_env(value: str) -> dict[str, str]:
+    """A literal resolves to itself in every environment; a ${TOKEN} to each env's value."""
+    m = TOKEN_RE.match(value.strip())
+    if not m:
+        return {"(literal)": value.strip()}
+    return env_values(m.group(1))
+
+
+# --------------------------------------------- what Flux actually renders (reachability)
+# A manifest is DELIVERED only if it is reachable from a Flux Kustomization's `path` through
+# the kustomize resource graph: <path>/kustomization.yaml -> resources -> (file | directory
+# -> that directory's kustomization.yaml -> ...). Membership is deliberately computed by
+# walking that graph rather than by asking "is this file's DIRECTORY mentioned somewhere":
+# the loose form silently passes when an entry is deleted from the very kustomization.yaml
+# that the directory reference renders (verified — it made two assertions here no-ops).
+KUST_RESOURCES: dict[str, list[str]] = {}
+for rel in [f for f in YAML_FILES if f.endswith("kustomization.yaml")]:
+    refs: list[str] = []
+    for d in docs_of(rel):
+        for r in (d.get("resources") or []) + (d.get("components") or []):
+            if not isinstance(r, str) or "://" in r:      # skip remote bases
+                continue
+            try:
+                p = (ROOT / rel).parent.joinpath(r).resolve()
+                refs.append(str(p.relative_to(ROOT)))
+            except Exception:
+                continue
+    KUST_RESOURCES[rel] = refs
+
+FLUX_PATHS: set[str] = set()
+for _rel, _d in DOCS:
+    if _d.get("kind") == "Kustomization" and \
+            str(_d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io/"):
+        p = str((_d.get("spec") or {}).get("path", "")).lstrip("./").rstrip("/")
+        if p:
+            FLUX_PATHS.add(p)
+
+
+def _walk(entry: str, seen: set[str], out: set[str]) -> None:
+    if entry in seen:
+        return
+    seen.add(entry)
+    kust = entry if entry.endswith("kustomization.yaml") else f"{entry}/kustomization.yaml"
+    if kust in KUST_RESOURCES:                            # a directory (or a kustomization)
+        for r in KUST_RESOURCES[kust]:
+            _walk(r, seen, out)
+        return
+    out.add(entry)                                        # a leaf manifest
+
+
+RENDERED: set[str] = set()
+for _p in FLUX_PATHS:
+    _walk(_p, set(), RENDERED)
+
+
+def delivered(rel: str) -> bool:
+    """True iff Flux renders this file from one of its Kustomization paths."""
+    return rel in RENDERED
+
+
+# --------------------------------------------------- (1) there is something to check
+CERTS = of_kind("Certificate", "cert-manager.io/")
+check(len(CERTS) >= 1,
+      "no cert-manager.io Certificate is authored anywhere in the tracked tree — every "
+      "assertion below would be vacuous, so this is a FAILURE, not a pass (R-TLS-02)")
+
+ISSUERS = of_kind("ClusterIssuer", "cert-manager.io/") + of_kind("Issuer", "cert-manager.io/")
+check(len(ISSUERS) >= 1,
+      "no cert-manager ClusterIssuer/Issuer is authored — nothing could issue automatically")
+
+# ---------------------------- (2) every terminating listener serves a controller cert
+CERT_SECRETS: dict[tuple[str, str], str] = {           # (namespace, secretName) -> file
+    (ns_of(d, "default"), str((d.get("spec") or {}).get("secretName", ""))): rel
+    for rel, d in CERTS
+}
+listeners_checked = 0
+for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
+    gw_ns = ns_of(gw)
+    for lis in ((gw.get("spec") or {}).get("listeners") or []):
+        tls = lis.get("tls") or {}
+        if str(tls.get("mode", "")) != "Terminate":
+            continue
+        listeners_checked += 1
+        refs = tls.get("certificateRefs") or []
+        check(len(refs) > 0,
+              f"{rel}: Gateway {gw_ns}/{name_of(gw)} listener {lis.get('name')!r} terminates "
+              "TLS with NO certificateRefs — it can only serve a certificate someone puts "
+              "there by hand")
+        for r in refs:
+            kind = str(r.get("kind", "Secret"))
+            key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
+            check(kind == "Secret",
+                  f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
+                  "Secret refs can be traced to a cert-manager Certificate")
+            check(key in CERT_SECRETS,
+                  f"{rel}: listener {lis.get('name')!r} serves Secret {key[0]}/{key[1]}, which "
+                  "NO tracked cert-manager Certificate declares as its secretName — that "
+                  "credential would have to be created by hand (R-TLS-02 'without manual steps')")
+check(listeners_checked > 0,
+      "no Gateway listener with tls.mode=Terminate found in the tracked tree — there is no "
+      "serving-TLS reference to trace, so (2) proved nothing")
+
+# --------------------------------- (3) no certificate material is tracked, anywhere
+TLS_KEYS = {"tls.crt", "tls.key", "ca.crt", "tls-combined.pem"}
+for rel, d in DOCS:
+    if d.get("kind") != "Secret":
+        continue
+    stype = str(d.get("type", ""))
+    check(stype != "kubernetes.io/tls",
+          f"{rel}: a Secret of type kubernetes.io/tls is COMMITTED (metadata.name="
+          f"{name_of(d)}) — a serving certificate authored by a human, not issued")
+    for field in ("data", "stringData"):
+        for k in (d.get(field) or {}):
+            check(str(k) not in TLS_KEYS,
+                  f"{rel}: committed Secret {name_of(d)} carries {field}.{k} — certificate "
+                  "material in git means a manual issuance step exists (R-TLS-02)")
+
+PEM_RE = re.compile(r"-----BEGIN (?:[A-Z ]+ )?CERTIFICATE-----")
+for rel in YAML_FILES:
+    body = (ROOT / rel).read_text(errors="replace")
+    # Scoped to YAML manifests deliberately: prose/docs may legitimately SHOW a PEM header
+    # in an example, and a .md file is not applied to any cluster. A manifest cannot.
+    check(not PEM_RE.search(body),
+          f"{rel}: contains a PEM CERTIFICATE body — a hand-minted certificate in a manifest")
+
+# --------------------------- (4)+(5)+(6) the issuer chain is automated in EVERY env
+# The cert-manager release's namespace IS the controller's `--cluster-resource-namespace`
+# (chart default), i.e. where a ClusterIssuer's solver credential must live. Derived from the
+# HelmRelease, never written down here, so this holds if the release namespace ever changes.
+cm_releases = [(rel, d) for rel, d in of_kind("HelmRelease") if name_of(d) == "cert-manager"]
+CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manager"
+
+ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
+solver_creds: list[tuple[str, str, str, str]] = []     # (issuer, ns, secretName, ref field)
+issuers_verified: set[str] = set()
+for rel, cert in CERTS:
+    spec = cert.get("spec") or {}
+    ref = spec.get("issuerRef") or {}
+    kind = str(ref.get("kind", "Issuer"))
+    per_env = resolve_per_env(str(ref.get("name", "")))
+    check(len(per_env) > 0,
+          f"{rel}: Certificate {name_of(cert)} issuerRef.name is a token with NO value in "
+          "clusters/base/cluster-vars.yaml or any overlay — it would render empty and never issue")
+    for env, iname in sorted(per_env.items()):
+        hit = ISSUER_INDEX.get((kind, iname))
+        check(hit is not None,
+              f"{rel}: Certificate {name_of(cert)} resolves issuerRef to {kind}/{iname} in "
+              f"environment {env!r}, but no such issuer is tracked — that environment cannot "
+              "issue anything without a hand-created issuer (R-TLS-02, R-ENV-02)")
+        if hit is None:
+            continue
+        irel, issuer = hit
+        issuers_verified.add(iname)
+        ispec = issuer.get("spec") or {}
+        acme = ispec.get("acme")
+        check(isinstance(acme, dict),
+              f"{irel}: {kind}/{iname} (used by {name_of(cert)} in env {env!r}) is not an ACME "
+              f"issuer — its issuance/renewal path is {sorted(ispec)}; docs/08/ADR-0006 mandate "
+              "ACME so both are unattended")
+        if not isinstance(acme, dict):
+            continue
+        check(str(acme.get("server", "")).startswith("https://"),
+              f"{irel}: {kind}/{iname} has no https ACME directory endpoint")
+        check(bool((acme.get("privateKeySecretRef") or {}).get("name")),
+              f"{irel}: {kind}/{iname} declares no privateKeySecretRef — the ACME account key "
+              "would have to be provisioned by hand")
+        solvers = acme.get("solvers") or []
+        check(len(solvers) > 0,
+              f"{irel}: {kind}/{iname} declares NO solvers — every order would sit unsolved "
+              "until a human completed the challenge")
+        for s in solvers:
+            types = [t for t in ("dns01", "http01") if t in s]
+            check(len(types) == 1,
+                  f"{irel}: {kind}/{iname} has a solver with challenge types {sorted(s)} — only "
+                  "the automated dns01/http01 solvers satisfy 'without manual steps'")
+            for provider, cfg in (s.get("dns01") or {}).items():
+                if not isinstance(cfg, dict):
+                    continue
+                for field, val in cfg.items():
+                    if isinstance(val, dict) and str(field).endswith(("SecretRef", "secretRef")):
+                        n = str(val.get("name", ""))
+                        check(bool(n),
+                              f"{irel}: {kind}/{iname} dns01 {provider}.{field} has no secret name")
+                        if n:
+                            # A ClusterIssuer resolves secret refs in the controller's
+                            # cluster-resource-namespace; a namespaced Issuer in its own.
+                            want_ns = CM_NS if kind == "ClusterIssuer" else ns_of(issuer, CM_NS)
+                            solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
+
+check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
+
+# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
+check(len(solver_creds) > 0,
+      "no dns01 solver credential reference found — docs/08 (OQ-4) requires the scoped "
+      "Cloudflare token to drive DNS-01, so this check would otherwise be vacuous")
+# Keyed by (namespace, name): a same-named Secret in ANOTHER namespace is NOT this credential.
+# (Verified: keying by name alone let the external-dns copy of `cloudflare-api-token` satisfy
+# the cert-manager reference, so deleting cert-manager's own copy passed — a silent no-op.)
+SECRET_DOCS = {(ns_of(d, CM_NS), name_of(d)): (rel, d)
+               for rel, d in DOCS if d.get("kind") == "Secret"}
+for iname, sns, sname, field in solver_creds:
+    hit = SECRET_DOCS.get((sns, sname))
+    check(hit is not None,
+          f"{iname}: dns01 {field} references Secret {sname!r}, which no tracked manifest "
+          f"delivers into namespace {sns!r} (the issuer's secret-resolution namespace) — the "
+          "credential would be created by hand in-cluster (R-TLS-02)")
+    if hit is None:
+        continue
+    srel, sdoc = hit
+    check("sops" in sdoc,
+          f"{srel}: Secret {sname} is the DNS-01 credential but is not SOPS-encrypted (R-SEC-01)")
+    check(delivered(srel),
+          f"{srel}: Secret {sname} is not rendered by any Flux Kustomization path — Flux would "
+          "never deliver it, so issuance would wait on a manual kubectl apply")
+
+# ------------------------------- (7) automatic CertificateRequest approval is intact
+check(len(cm_releases) == 1,
+      f"expected exactly one cert-manager HelmRelease, found {[r for r, _ in cm_releases]} — "
+      "the controller that issues and renews must itself be GitOps-managed")
+for rel, hr in cm_releases:
+    values = ((hr.get("spec") or {}).get("values") or {})
+    check(values.get("disableAutoApproval") in (None, False),
+          f"{rel}: cert-manager values set disableAutoApproval={values.get('disableAutoApproval')!r} "
+          "— every CertificateRequest would then wait for an approver, which is a manual step")
+    signers = values.get("approveSignerNames")
+    if signers is not None:
+        pats = [str(s) for s in (signers if isinstance(signers, list) else [signers])]
+        for iname in sorted(issuers_verified):
+            ok = any(p in ("*", "*/*") or "clusterissuers.cert-manager.io" in p and
+                     (p.endswith("/*") or p.endswith(f"/{iname}")) for p in pats)
+            check(ok,
+                  f"{rel}: approveSignerNames={pats} does not cover ClusterIssuer {iname} — "
+                  "requests for it would never be auto-approved")
+    # The chart's own CRD install must be on, else the CRs below cannot even be applied.
+    check((values.get("crds") or {}).get("enabled") is True,
+          f"{rel}: cert-manager values do not enable CRD installation — the Certificate CRD "
+          "would have to be installed by hand")
+
+# ------------------------------- (8) the whole chain is continuously reconciled from git
+check(len(FLUX_PATHS) > 0 and len(RENDERED) > 20,
+      f"the kustomize resource graph resolved {len(RENDERED)} manifest(s) from {len(FLUX_PATHS)} "
+      "Flux Kustomization path(s) — too few to be a real graph, so (8) would prove nothing")
+for rel, _ in cm_releases + CERTS + ISSUERS:
+    check(delivered(rel),
+          f"{rel} is not rendered by any Flux Kustomization path — nothing reconciles it, so a "
+          "deleted Certificate/issuer/controller would stay deleted until a human re-applied it")
+
+# --------------------------------------------- (9) renewal is not defeated arithmetically
+def to_seconds(v) -> int | None:
+    m = re.fullmatch(r"\s*(\d+)\s*([smh])\s*", str(v))
+    if not m:
+        return None
+    return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)]
+
+
+renew_pairs = 0
+for rel, cert in CERTS:
+    spec = cert.get("spec") or {}
+    dur, rb = to_seconds(spec.get("duration")), to_seconds(spec.get("renewBefore"))
+    if dur is None or rb is None:
+        continue
+    renew_pairs += 1
+    check(rb < dur,
+          f"{rel}: Certificate {name_of(cert)} sets renewBefore={spec.get('renewBefore')} >= "
+          f"duration={spec.get('duration')} — the renewal instant never arrives")
+
+# --------------------------------------------------------------------------- verdict
+# NO-OP GUARD: this repo has been bitten by checks that degrade to a successful no-op.
+# The floor below is a count of assertions that MUST have run for the verdict to mean
+# anything; it is not a style rule.
+check(CHECKS >= 20,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
+      "would be meaningless")
+
+if FAILURES:
+    print(f"CHECK FAIL (tls-lifecycle-automated): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: tls-lifecycle-automated — {CHECKS} assertions; {len(CERTS)} Certificate(s), "
+      f"{len(ISSUERS)} issuer(s), {len(issuers_verified)} issuer(s) reached from a Certificate "
+      f"across every environment, {listeners_checked} terminating listener(s), "
+      f"{len(solver_creds)} dns01 credential ref(s), {renew_pairs} duration/renewBefore pair(s). "
+      "R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs "
+      "(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer).")
+PY

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -68,7 +68,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l2-behavioral/cert-lifecycle-automated/chainsaw-test.yaml

@@ -0,0 +1,391 @@
+# =============================================================================
+# L2 BEHAVIORAL — certificate lifecycle is automatic  (R-TLS-02)
+#                 [RUNNABLE / NOT YET EXECUTED — see STATUS]
+# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
+# Spec:    docs/00 R-TLS-02 — "Certificates MUST be automatically **issued, renewed,
+#          and revoked** without manual steps."
+#          docs/08 §"Renewal & revocation (R-TLS-02)" — "cert-manager renews well
+#          before expiry automatically; no human action." / "Revocation: removing/
+#          rotating a certificate is a declarative change; cert-manager handles
+#          issuance of replacements."
+#          docs/08 acceptance gate — "Certificates renew automatically ahead of expiry
+#          with no manual step (**observed across a renewal**)."
+# Impl:    platform/cert-manager/**, platform/gateway/wildcard-certificate.yaml.
+# Sibling: tests/l0-static/tls-lifecycle-automated proves the "without manual steps"
+#          clause statically (in the merge gate). This test observes the three VERBS.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): NOTHING about either environment is written down. The
+# serving certificate is DISCOVERED — find the Gateway whose listener terminates TLS, follow
+# its certificateRefs Secret, and read that Secret's `cert-manager.io/certificate-name`
+# annotation to reach the Certificate. No hostname, no namespace list, no issuer name, no
+# address: dev proves dev's cert and prod proves prod's from the same authored assertions.
+#
+# GIVEN cert-manager reconciled by Flux and a Gateway terminating TLS,
+# WHEN  the serving Certificate's provenance and renewal schedule are read, and a NEW
+#       Certificate is then requested, left to renew itself, and removed,
+# THEN  1. the serving credential was produced by the CONTROLLER, not a human: its
+#          Secret carries cert-manager's certificate-name annotation, its
+#          CertificateRequest was submitted by cert-manager's own ServiceAccount and
+#          was APPROVED automatically, and its issuer is a Ready ACME ClusterIssuer;
+#       2. an unattended RENEWAL IS SCHEDULED: status.renewalTime is in the future and
+#          strictly before status.notAfter, so no human is in the renewal path;
+#       3. ISSUED — a newly declared Certificate becomes Ready and its Secret appears
+#          with certificate material, with no step between declaring and having it;
+#       4. RENEWED — that certificate, given a renewal window that opens one minute
+#          after issuance, REISSUES ITSELF at least twice (status.revision >= 3) and
+#          the serving bytes actually change, with no human action. This is docs/08's
+#          "observed across a renewal", fast-forwarded;
+#       5. REMOVED/REVOKED — deleting the Certificate withdraws its serving material.
+#
+# WHY A HERMETIC SELF-SIGNED ISSUER FOR STEPS 3-5, AND WHAT THAT COSTS
+# -------------------------------------------------------------------
+# Steps 3-5 need a certificate lifecycle that completes in minutes and repeats on every
+# run. Driving that through the real ACME issuer would fire a public Let's Encrypt order
+# (and a Cloudflare _acme-challenge write) per conformance run, burning issuance rate
+# limits for evidence that is not about ACME: cert-manager's renewal scheduler and
+# ownership logic are issuer-independent. So steps 3-5 use a `selfSigned` Issuer created
+# in this test's OWN namespace, and the ACME/DNS-01 path is proven elsewhere — step 1-2
+# read the REAL wildcard certificate the Gateway serves, and l2-behavioral/trusted-tls
+# proves a publicly-trusted LE certificate is actually served (R-TLS-01/03).
+# STATED PLAINLY: steps 3-5 do NOT prove the ACME order path renews; they prove the
+# CONTROLLER's issue/renew/withdraw automation runs unattended in this cluster, while
+# steps 1-2 prove the real serving certificate is on exactly that automated path.
+#
+# STATUS: RUNNABLE, NOT YET EXECUTED. Steps 1-2 are read-only. Steps 3-5 CREATE an
+# Issuer/Certificate (and DELETE the Certificate they created) inside chainsaw's
+# per-test namespace — the same class of mutation every other L2 test here performs,
+# self-cleaning via namespace teardown, touching no platform object. WS-TEST authored
+# them against live dev/prod read-only inspection under an explicit instruction not to
+# mutate either cluster, so THE MUTATING STEPS HAVE NOT BEEN RUN. Nothing about the
+# suite gates on them: ci/conformance.sh only runs L2/L3 when CLUSTER_AVAILABLE=true.
+#
+# EXPECTED FAILURE, DELIBERATELY NOT SOFTENED (R-TEST-06)
+# ------------------------------------------------------
+# Step 5 WILL FAIL as the platform is authored today. cert-manager only garbage-collects
+# a Certificate's Secret when the controller runs with `enableCertificateOwnerRef: true`;
+# platform/cert-manager/helmrelease.yaml does not set it (chart default false), and the
+# live wildcard Secret on both clusters has NO ownerReferences (verified read-only,
+# 2026-09-05). So a removed Certificate leaves its credential in place, and "revoked
+# without manual steps" does not hold. That is a finding about the platform, not a
+# defect in this test: the assertion stays as the requirement reads and is escalated to
+# the PM. (ACME revocation proper is not implemented by cert-manager at all.)
+#
+# KNOWN, DELIBERATE GAPS (this test claims none of these):
+#   * Not proven: that the CA would refuse a revoked certificate — there is no
+#     revocation request to make, so nothing observable exists to assert.
+#   * Not proven: renewal of the REAL wildcard cert (its window opens 30 days before a
+#     90-day expiry; a conformance run cannot wait, and forcing it would rotate the
+#     platform's serving credential). Step 2 asserts the schedule; step 4 asserts the
+#     mechanism that will act on it.
+#   * Not proven: certificate expiry ALERTING — docs/08 delegates that to R-OBS-04
+#     (l3-fault-injection/alerting).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: cert-lifecycle-automated
+  labels:
+    conformance.home.k3s/R-TLS-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-TLS-02"
+    conformance.home.k3s/layer: "L2"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given cert-manager and a TLS-terminating Gateway, when the serving certificate's
+      provenance and renewal schedule are read and a new certificate is requested, left
+      to renew, and removed, then issuance, renewal and withdrawal all happen with no
+      human step — the serving cert was controller-issued and auto-approved, its
+      renewal is scheduled ahead of expiry, and a declared certificate issues and
+      reissues itself unattended.
+spec:
+  timeouts:
+    apply: 60s
+    assert: 3m
+    exec: 10m
+  steps:
+    # ---------------------------------------------------------------- READ-ONLY HALF
+    - name: given-cert-manager-is-reconciled-and-healthy
+      description: >-
+        The controller that issues and renews must be running, and must have been
+        installed by Flux rather than by hand — otherwise "automatic" has no engine.
+      try:
+        - script:
+            content: |
+              set -u
+              # Discover cert-manager's Deployments by chart label, in whatever namespace
+              # the release lives (nothing about the environment is assumed).
+              sel='app.kubernetes.io/instance=cert-manager'
+              rows="$(kubectl get deploy -A -l "$sel" \
+                        -o custom-columns='NS:.metadata.namespace,N:.metadata.name,RDY:.status.readyReplicas' \
+                        --no-headers 2>/dev/null || true)"
+              echo "$rows"
+              [ -n "$rows" ] || { echo "FAIL: no cert-manager Deployment found ($sel)"; exit 1; }
+              n=0
+              echo "$rows" | while read -r ns name rdy; do
+                [ "${rdy:-0}" -ge 1 ] || { echo "FAIL: $ns/$name has no ready replica"; exit 1; }
+              done || exit 1
+              n="$(echo "$rows" | wc -l)"
+              # controller + webhook + cainjector: without the webhook a Certificate is
+              # rejected, without cainjector the webhook has no CA — all three or nothing.
+              [ "$n" -ge 3 ] || { echo "FAIL: expected >=3 cert-manager Deployments, saw $n"; exit 1; }
+              # GitOps-managed, not hand-installed: Flux labels what it applies.
+              ns="$(echo "$rows" | head -1 | awk '{print $1}')"
+              lbl="$(kubectl get deploy -n "$ns" -l "$sel" \
+                       -o jsonpath='{.items[0].metadata.labels.helm\.toolkit\.fluxcd\.io/name}' 2>/dev/null || true)"
+              [ -n "$lbl" ] || { echo "FAIL: cert-manager is not owned by a Flux HelmRelease"; exit 1; }
+              echo "CERT_MANAGER_HEALTHY_OK ($n deployments, flux HelmRelease $lbl)"
+            check:
+              (contains($stdout, 'CERT_MANAGER_HEALTHY_OK')): true
+
+    - name: then-the-serving-certificate-was-issued-by-the-controller
+      description: >-
+        Follow the Gateway's terminating listener to the Secret it serves, and prove
+        that Secret's provenance is cert-manager — annotation, CertificateRequest
+        requester, automatic approval, and a Ready ACME issuer.
+      try:
+        - script:
+            content: |
+              set -u
+              # --- discover the serving Secret from the Gateway itself -------------------
+              gw="$(kubectl get gateway -A \
+                      -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
+                      2>/dev/null | head -1)"
+              [ -n "$gw" ] || { echo "FAIL: no Gateway found — nothing terminates TLS"; exit 1; }
+              gns="${gw% *}"; gname="${gw#* }"
+              sec="$(kubectl -n "$gns" get gateway "$gname" \
+                       -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' \
+                       2>/dev/null | awk '{print $1}')"
+              [ -n "$sec" ] || { echo "FAIL: Gateway $gns/$gname terminates no TLS listener"; exit 1; }
+              echo "gateway=$gns/$gname serving-secret=$sec"
+
+              # --- provenance (1): only the controller writes this annotation ------------
+              cert="$(kubectl -n "$gns" get secret "$sec" \
+                        -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}' 2>/dev/null || true)"
+              [ -n "$cert" ] || { echo "FAIL: Secret $gns/$sec carries no cert-manager.io/certificate-name annotation — it was not produced by cert-manager"; exit 1; }
+              crt="$(kubectl -n "$gns" get secret "$sec" -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
+              [ -n "$crt" ] || { echo "FAIL: Secret $gns/$sec holds no tls.crt"; exit 1; }
+
+              # --- provenance (2): the Certificate is Ready and has a revision -----------
+              ready="$(kubectl -n "$gns" get certificate "$cert" \
+                         -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
+              rev="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.revision}' 2>/dev/null || true)"
+              echo "certificate=$gns/$cert ready=$ready revision=$rev"
+              [ "$ready" = "True" ] || { echo "FAIL: Certificate $cert is not Ready"; exit 1; }
+              [ -n "$rev" ] && [ "$rev" -ge 1 ] || { echo "FAIL: Certificate $cert has no issued revision"; exit 1; }
+
+              # --- provenance (3): the request was made BY THE CONTROLLER and auto-approved
+              # spec.username is the identity that created the CertificateRequest. A human
+              # operator would appear as a user/OIDC identity, not a ServiceAccount.
+              cr="$(kubectl -n "$gns" get certificaterequest \
+                      -o jsonpath="{range .items[?(@.metadata.annotations.cert-manager\.io/certificate-name=='${cert}')]}{.metadata.name}{' '}{.spec.username}{'\n'}{end}" \
+                      2>/dev/null | tail -1)"
+              [ -n "$cr" ] || { echo "FAIL: no CertificateRequest for $cert — cannot show who requested it"; exit 1; }
+              crname="${cr% *}"; requester="${cr#* }"
+              echo "certificaterequest=$crname requester=$requester"
+              case "$requester" in
+                system:serviceaccount:*) : ;;
+                *) echo "FAIL: CertificateRequest $crname was submitted by '$requester', not a controller ServiceAccount"; exit 1 ;;
+              esac
+              appr="$(kubectl -n "$gns" get certificaterequest "$crname" \
+                        -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
+              [ "$appr" = "True" ] || { echo "FAIL: CertificateRequest $crname is not Approved=True (approval would be a manual step)"; exit 1; }
+
+              # --- provenance (4): an automated ACME issuer, and it is Ready -------------
+              ikind="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.kind}')"
+              iname="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.spec.issuerRef.name}')"
+              echo "issuer=$ikind/$iname"
+              if [ "$ikind" = "ClusterIssuer" ]; then
+                iready="$(kubectl get clusterissuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
+                iacme="$(kubectl get clusterissuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
+              else
+                iready="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)"
+                iacme="$(kubectl -n "$gns" get issuer "$iname" -o jsonpath='{.spec.acme.server}' 2>/dev/null || true)"
+              fi
+              [ "$iready" = "True" ] || { echo "FAIL: issuer $iname is not Ready — renewal would need a human to fix it"; exit 1; }
+              [ -n "$iacme" ] || { echo "FAIL: issuer $iname is not an ACME issuer"; exit 1; }
+              echo "SERVING_CERT_CONTROLLER_ISSUED_OK"
+            check:
+              (contains($stdout, 'SERVING_CERT_CONTROLLER_ISSUED_OK')): true
+
+    - name: then-an-unattended-renewal-is-already-scheduled
+      description: >-
+        cert-manager writes status.renewalTime only after computing when it will renew.
+        Asserting it is in the future and before expiry proves the renewal is scheduled
+        and that nothing is waiting on a human (docs/08 "renews well before expiry").
+      try:
+        - script:
+            content: |
+              set -u
+              gw="$(kubectl get gateway -A \
+                      -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | head -1)"
+              gns="${gw% *}"; gname="${gw#* }"
+              sec="$(kubectl -n "$gns" get gateway "$gname" \
+                       -o jsonpath='{.spec.listeners[?(@.tls.mode=="Terminate")].tls.certificateRefs[0].name}' | awk '{print $1}')"
+              cert="$(kubectl -n "$gns" get secret "$sec" \
+                        -o jsonpath='{.metadata.annotations.cert-manager\.io/certificate-name}')"
+              renew="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.renewalTime}')"
+              after="$(kubectl -n "$gns" get certificate "$cert" -o jsonpath='{.status.notAfter}')"
+              [ -n "$renew" ] || { echo "FAIL: Certificate $cert has no status.renewalTime — no renewal is scheduled"; exit 1; }
+              [ -n "$after" ] || { echo "FAIL: Certificate $cert has no status.notAfter"; exit 1; }
+              now_s="$(date -u +%s)"
+              r_s="$(date -u -d "$renew" +%s)"
+              a_s="$(date -u -d "$after" +%s)"
+              echo "now=$(date -u -d "@$now_s" +%FT%TZ) renewalTime=$renew notAfter=$after"
+              echo "renewal is in $(( (r_s - now_s) / 86400 ))d; expiry in $(( (a_s - now_s) / 86400 ))d"
+              [ "$r_s" -gt "$now_s" ] || { echo "FAIL: renewalTime is in the PAST — the scheduled renewal did not happen"; exit 1; }
+              [ "$r_s" -lt "$a_s" ] || { echo "FAIL: renewalTime is not before notAfter — renewal would come too late"; exit 1; }
+              echo "RENEWAL_SCHEDULED_OK"
+            check:
+              (contains($stdout, 'RENEWAL_SCHEDULED_OK')): true
+
+    # ------------------------------------------------- MUTATING HALF (hermetic namespace)
+    - name: when-a-certificate-is-declared-then-it-is-issued-with-no-human-step
+      description: >-
+        Declare an Issuer and a Certificate in this test's own namespace, with a renewal
+        window that opens one minute after issuance. Nothing else is done to them for the
+        rest of the test — every later transition is the controller acting on its own.
+      try:
+        - apply:
+            resource:
+              # selfSigned: no external CA, no rate limit, no DNS write. See the header for
+              # what this proves (controller automation) and what it does not (the ACME path).
+              apiVersion: cert-manager.io/v1
+              kind: Issuer
+              metadata:
+                name: conformance-lifecycle
+              spec:
+                selfSigned: {}
+        - apply:
+            resource:
+              apiVersion: cert-manager.io/v1
+              kind: Certificate
+              metadata:
+                name: conformance-lifecycle
+              spec:
+                secretName: conformance-lifecycle-tls
+                # 1h is cert-manager's minimum duration; renewing 59m before expiry puts the
+                # renewal instant ~1 minute after each issuance, so an unattended renewal is
+                # observable inside a test run instead of 30 days from now.
+                duration: 1h
+                renewBefore: 3540s
+                commonName: conformance-lifecycle.invalid
+                dnsNames:
+                  - conformance-lifecycle.invalid
+                privateKey:
+                  rotationPolicy: Always
+                issuerRef:
+                  kind: Issuer
+                  name: conformance-lifecycle
+        - assert:
+            resource:
+              apiVersion: cert-manager.io/v1
+              kind: Certificate
+              metadata:
+                name: conformance-lifecycle
+              status:
+                conditions:
+                  - type: Ready
+                    status: "True"
+                revision: 1
+        - script:
+            content: |
+              set -u
+              crt="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
+                       -o jsonpath='{.data.tls\.crt}' 2>/dev/null || true)"
+              key="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
+                       -o jsonpath='{.data.tls\.key}' 2>/dev/null || true)"
+              [ -n "$crt" ] && [ -n "$key" ] || { echo "FAIL: no certificate material was issued into conformance-lifecycle-tls"; exit 1; }
+              # Record the serving bytes so the renewal step can prove they CHANGED.
+              echo "$crt" | cksum > /tmp/conformance-cert-r1.cksum
+              echo "revision-1 cert fingerprint: $(cat /tmp/conformance-cert-r1.cksum)"
+              echo "CERT_ISSUED_OK"
+            check:
+              (contains($stdout, 'CERT_ISSUED_OK')): true
+
+    - name: then-it-renews-itself-before-expiry-with-no-human-step
+      description: >-
+        Nothing touches the Certificate here. Poll until the controller has reissued it
+        at least twice on its own and the serving bytes differ from revision 1 — docs/08's
+        "observed across a renewal", and every request auto-approved.
+      try:
+        - script:
+            content: |
+              set -u
+              want=3          # revision 1 = issuance; >=3 means TWO unattended renewals
+              for i in $(seq 1 30); do
+                rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
+                         -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
+                echo "poll $i: revision=${rev:-0}"
+                [ "${rev:-0}" -ge "$want" ] && break
+                sleep 20
+              done
+              rev="$(kubectl -n "$NAMESPACE" get certificate conformance-lifecycle \
+                       -o jsonpath='{.status.revision}' 2>/dev/null || echo 0)"
+              [ "${rev:-0}" -ge "$want" ] || { echo "FAIL: revision reached only ${rev:-0} — cert-manager did not renew unattended"; exit 1; }
+
+              # The serving material must actually have rotated, not just the revision count.
+              now="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
+                       -o jsonpath='{.data.tls\.crt}' | cksum)"
+              was="$(cat /tmp/conformance-cert-r1.cksum)"
+              echo "r1=$was  now=$now"
+              [ "$now" != "$was" ] || { echo "FAIL: the Secret still holds the revision-1 certificate — nothing was actually reissued"; exit 1; }
+
+              # Every renewal request was approved by the controller, never by a person.
+              bad=0
+              for cr in $(kubectl -n "$NAMESPACE" get certificaterequest -o name 2>/dev/null); do
+                st="$(kubectl -n "$NAMESPACE" get "$cr" \
+                        -o jsonpath='{.status.conditions[?(@.type=="Approved")].status}' 2>/dev/null || true)"
+                who="$(kubectl -n "$NAMESPACE" get "$cr" -o jsonpath='{.spec.username}' 2>/dev/null || true)"
+                echo "  $cr approved=$st requester=$who"
+                [ "$st" = "True" ] || bad=1
+                case "$who" in system:serviceaccount:*) : ;; *) bad=1 ;; esac
+              done
+              [ "$bad" = 0 ] || { echo "FAIL: a CertificateRequest was not auto-approved, or was not submitted by a controller"; exit 1; }
+              echo "CERT_RENEWED_UNATTENDED_OK (revision=$rev)"
+            check:
+              (contains($stdout, 'CERT_RENEWED_UNATTENDED_OK')): true
+
+    - name: then-removing-the-certificate-withdraws-the-serving-material
+      description: >-
+        docs/08 reads revocation as "removing/rotating a certificate is a declarative
+        change". The declarative change is the deletion; withdrawal of the credential
+        must follow with no manual cleanup. EXPECTED TO FAIL TODAY — cert-manager needs
+        `enableCertificateOwnerRef: true` for the Secret to be garbage-collected, and the
+        tracked HelmRelease does not set it. Kept as the requirement reads (R-TEST-06).
+      try:
+        - delete:
+            ref:
+              apiVersion: cert-manager.io/v1
+              kind: Certificate
+              name: conformance-lifecycle
+        - script:
+            content: |
+              set -u
+              for i in $(seq 1 15); do
+                if ! kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls >/dev/null 2>&1; then
+                  echo "CERT_WITHDRAWN_OK (serving material gone after $((i * 4))s)"
+                  exit 0
+                fi
+                sleep 4
+              done
+              owners="$(kubectl -n "$NAMESPACE" get secret conformance-lifecycle-tls \
+                          -o jsonpath='{.metadata.ownerReferences}' 2>/dev/null || true)"
+              echo "FAIL: the Certificate was deleted but Secret conformance-lifecycle-tls still"
+              echo "      holds usable certificate material (ownerReferences=${owners:-none})."
+              echo "      Removing a certificate therefore does NOT withdraw the credential, so"
+              echo "      R-TLS-02's 'revoked ... without manual steps' does not hold. FIX:"
+              echo "      set enableCertificateOwnerRef: true in platform/cert-manager values"
+              echo "      (owner decision — WS-TEST does not weaken the assertion instead)."
+              exit 1
+            check:
+              (contains($stdout, 'CERT_WITHDRAWN_OK')): true
+      finally:
+        # Namespace teardown removes both objects anyway; explicit so a leftover Secret
+        # from the expected failure above is never left holding key material.
+        - script:
+            content: |
+              kubectl -n "$NAMESPACE" delete secret conformance-lifecycle-tls --ignore-not-found --wait=false || true
+              kubectl -n "$NAMESPACE" delete certificate conformance-lifecycle --ignore-not-found --wait=false || true
+              kubectl -n "$NAMESPACE" delete issuer conformance-lifecycle --ignore-not-found --wait=false || true
+              rm -f /tmp/conformance-cert-r1.cksum || true

conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml

@@ -153,3 +153,20 @@ spec:
       finally:
         - script:
             content: 'echo "cleanup handled by chainsaw namespace teardown"'
+    # -----------------------------------------------------------------------------------
+    # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
+    # chainsaw already fails the test before reaching this step if any of them fails — but a
+    # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
+    # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
+    # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
+    # forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list
+    # — which is precisely what docs/07 asks for ("These proofs are part of the platform
+    # acceptance suite, not one-time checks"). This step exists only to emit that token, and it
+    # is reached only when everything above has passed. Asserted by
+    # tests/l0-static/auditability-proofs-producible.
+    - name: then-emit-the-proof-of-pass-sentinel
+      try:
+        - script:
+            content: echo DEFAULT_DENY_PROVEN_OK
+            check:
+              (contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true

conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml

@@ -3,11 +3,18 @@
 # Catalog: docs/14 "DNS/exposure (R-DNS-*)"
 # Spec:    docs/09 (ExternalDNS auto-manages records; split-horizon foundation)
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER
+# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the
+# expected answer is the LB IP that THIS Service was actually assigned (read back from its own
+# status) — so neither the name nor the address is authored per environment, and no run-time flag
+# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two
+# environments from ever describing the same public name (the 2026-09-04 incident precondition).
+#
 # GIVEN a LoadBalancer Service annotated with an external-dns hostname,
 # WHEN  ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync),
 # THEN  a PUBLIC DNS record for that hostname is auto-created, pointing at the
-#       Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the
+#       Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that
-#       cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint.
+#       Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint.
 #
 # CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown;
 # ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass.
@@ -38,6 +45,13 @@ spec:
     apply: 90s
     assert: 5m
     exec: 6m
+  # -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST.
+  #    flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this
+  #    binding is the same source of truth the platform is rendered from — dev asks dev, prod
+  #    asks prod, and there is no per-environment fork and no flag to forget.
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     - name: given-a-service-requesting-a-dns-name
       try:
@@ -48,7 +62,8 @@ spec:
               metadata:
                 name: dns-probe-svc
                 annotations:
-                  external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev
+                  # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
+                  external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
               spec:
                 type: LoadBalancer
                 # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -59,7 +74,27 @@ spec:
                   - name: http
                     port: 80
                     targetPort: 80
+        # Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will
+        # publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal
+        # keeps the expression nil-safe while the address is still absent, so the assert POLLS
+        # (fails and retries) instead of erroring out on the first evaluation.
+        - assert:
+            resource:
+              apiVersion: v1
+              kind: Service
+              metadata:
+                name: dns-probe-svc
+              status:
+                (length(loadBalancer.ingress || `[]`)): 1
     - name: then-externaldns-creates-the-public-record
+      # Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's
+      # assert guarantees it exists by now — step bindings are evaluated when the step runs).
+      # Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60
+      # range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is
+      # 192.168.2.20-.30, and neither is written down here.
+      bindings:
+        - name: lbIp
+          value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip)
       try:
         - apply:
             resource:
@@ -76,20 +111,27 @@ spec:
                     containers:
                       - name: probe
                         image: busybox:1.36.1
+                        # Both values come from the cluster under test: the suffix from
+                        # flux-system/cluster-vars, the address from this Service's own status.
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
+                          - name: LB_IP
+                            value: ($lbIp)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="conformance-dns.home.keithjackson.dev"
+                            host="conformance-dns.${BASE_DOMAIN}"
                             # Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and
-                            # points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it.
+                            # points at THIS Service's assigned LB IP. Proves ExternalDNS created it.
                             for i in $(seq 1 28); do
-                              ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)"
+                              ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)"
                               if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi
-                              echo "poll $i: record not yet published"; sleep 10
+                              echo "poll $i: record not yet published as ${LB_IP}"; sleep 10
                             done
-                            echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1
+                            echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1
         - assert:
             resource:
               apiVersion: batch/v1

conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml

@@ -2,17 +2,24 @@
 # L2 BEHAVIORAL — split-horizon INTERNAL resolution  (R-DNS-01 / R-DNS-03)   [RUNNABLE]
 # Catalog: docs/14 "DNS/exposure (R-DNS-*)"
 # Spec:    docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
-#          <svc>.home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin.
+#          <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
 # Impl:    platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
-#          answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}).
+#          answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
+# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
+# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
+# per-environment and no flag is passed at run time, so the suite cannot be run against the
+# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
 #
 # GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
 # WHEN  an in-cluster pod resolves a platform hostname over its PUBLIC name
-#       (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10),
+#       (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
-# THEN  the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a
+# THEN  the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
-#       public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the
+#       templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
-#       Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates
+#       IP. This is what lets pod->public-hostname traffic hairpin through the
-#       (R-DNS-01 "services reachable at <svc>.home..."; R-DNS-03 split-horizon).
+#       Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
+#       (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
 #
 # Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
 # split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
@@ -34,13 +41,23 @@ metadata:
     conformance.home.k3s/safety-critical: "false"
     conformance.home.k3s/executable: "true"
     conformance.home.k3s/intent: >-
-      An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal
+      An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
-      Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution.
+      internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
 spec:
   timeouts:
     apply: 60s
     assert: 2m
     exec: 2m
+  # -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
+  #    here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
+  #    (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
+  #    same source of truth the platform is rendered from. Test-level => in scope for every step
+  #    (unlike step `outputs`, which are step-scoped).
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
+    - name: gatewayVip
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
   steps:
     - name: then-cluster-dns-answers-with-the-internal-gateway-vip
       try:
@@ -59,23 +76,32 @@ spec:
                     containers:
                       - name: probe
                         image: busybox:1.36.1
+                        # The cluster's OWN identity, resolved from flux-system/cluster-vars by the
+                        # test-level bindings — never a literal (R-ENV-02).
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
+                          - name: GATEWAY_VIP
+                            value: ($gatewayVip)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="keycloak.home.keithjackson.dev"
+                            host="keycloak.${BASE_DOMAIN}"
                             # Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
-                            # 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP
+                            # 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
-                            # (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway.
+                            # (the address coredns-custom is templated with), proving pods hairpin
+                            # through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
+                            # old hardcoded 192.168.1.53-.60 range AND correct in both environments.
                             out="$(nslookup "$host" 2>/dev/null)"
                             echo "$out"
-                            ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)"
+                            ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
                             if [ -n "$ip" ]; then
                               echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
                               exit 0
                             fi
-                            echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);"
+                            echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
                             echo "      split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
                             exit 1
         - assert:

conformance/tests/l2-behavioral/egress-default-path/chainsaw-test.yaml

@@ -0,0 +1,199 @@
+# =============================================================================
+# L2 BEHAVIORAL — the default egress path is the WAN; the VPN is opt-in  (R-NET-04)  [RUNNABLE]
+# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
+# Spec:    docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
+#          routing rules ... realized as two paths — the local `internet` path and the
+#          in-cluster `vpn` path").
+#          docs/03 §"Egress routing: two paths, one physical WAN": "the `internet` path is
+#          simply the default route out".
+#          docs/03 §"VPN egress model": "**Default is the WAN; the VPN is strictly opt-in.**
+#          Every pod — and the nodes themselves — egress the local/private WAN by default and
+#          **never** the VPN. Only services labeled `egress-class=vpn` are steered to the
+#          tunnel"; and the tunnel "is **split-tunnel**: ... only what Cilium points at `tun0`
+#          uses the VPN."
+#
+# WHAT THIS PROVES THAT tests/l2-behavioral/egress-class DOES NOT
+# --------------------------------------------------------------
+# That test (R-NET-05/07) compares two LABELLED pods — `internet` vs `vpn` — and shows their
+# public IPs differ, then flips a label. Both of its pods declared a class, so it says nothing
+# about the case R-NET-04 actually turns on: a pod that NO routing rule selects. This test adds
+# that pod, and asserts the two claims that make a rule-selected path set complete:
+#
+#   TOTALITY   — an UNLABELLED pod egresses the WAN, i.e. the same public IP as the
+#                explicitly-`internet` pod. Absence of a matching rule IS a path, and it is
+#                the local one. If the steering rule ever widened (an empty selector, a
+#                matchExpression, a second rule), this pod's IP would move to the tunnel and
+#                this assertion is what would notice.
+#   COEXISTENCE — the WAN path and the VPN path are live AT THE SAME TIME, observed in one
+#                pass: three pods, two distinct public IPs. "Multiple egress paths" is a
+#                simultaneity claim, not a claim that one switch has two positions.
+#   PER-DESTINATION SELECTION — the vpn-class pod still reaches an in-cluster/RFC1918
+#                destination over the LOCAL path while its public traffic is tunnelled. The
+#                tunnel is destination-scoped (excludedCIDRs), not a default route, so ONE pod
+#                uses BOTH paths at once.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname or range appears. Every assertion is a
+# RELATION between observed values (equal / not-equal), so it holds against dev and prod alike
+# and survives the WAN or the VPN provider rotating addresses.
+#
+# GIVEN one unlabelled, one `internet`-class and one `vpn`-class pod — the same workload three
+#       times, differing only in the class they declare (or do not),
+# WHEN  each asks an external echo for the public IP it egressed from, and the vpn-class pod
+#       also resolves a name through the in-cluster resolver,
+# THEN  the unlabelled pod's public IP EQUALS the internet-class pod's (the default route out
+#       is the WAN, and the VPN is strictly opt-in);
+# AND   the vpn-class pod's public IP DIFFERS from both (a second, simultaneously-live path);
+# AND   the vpn-class pod still resolves in-cluster (its private-destination traffic was not
+#       swallowed by the tunnel), so the same pod is using both paths at once.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
+#   - THAT NO OTHER RULE EXISTS. Three pods cannot enumerate a rule set. That exactly one
+#     steering rule is authored, that it selects only the vpn class, that no `ip rule` is
+#     unscoped and no nftables DROP unqualified is the L0 half, in
+#     tests/l0-static/egress-paths-selected-by-rules (same requirement label).
+#   - WHICH INTERFACE the packets left by. The observation is the public IP an external echo
+#     reports; it distinguishes the paths without naming them. Asserting "out tun0" would need
+#     host-level datapath inspection (Hubble/tcpdump on the gateway node), which no test in
+#     this suite does.
+#   - THE KILL-SWITCH. Tunnel-down behaviour is R-NET-06's, at L3 (vpn-killswitch): `vpn`
+#     fails closed. Its mirror — that `internet` is never blackholed — is asserted statically
+#     by the L0 half, not here.
+#   - NODE traffic. docs/03 also says "the nodes themselves" egress the WAN by default. That
+#     is a host-level fact this test cannot see from a pod.
+#   - EXTERNAL DEPENDENCY, stated plainly: the public-IP echo (api.ipify.org) is off-cluster,
+#     so a WAN outage or the echo being down fails this test. That is the same dependency
+#     tests/l2-behavioral/egress-class already carries; the alternative — trusting the
+#     cluster's own account of which path it used — would not be a behavioural test at all.
+#
+# CLEANUP: chainsaw deletes the per-test namespace (all three pods with it). Nothing outside
+# that namespace is created or changed — no label on a platform object, no policy edit.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: egress-default-path
+  labels:
+    conformance.home.k3s/R-NET-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-NET-04"
+    conformance.home.k3s/layer: "L2"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given an unlabelled, an internet-class and a vpn-class pod, when each observes the public
+      IP it egressed from, then the unlabelled pod's IP equals the internet-class pod's (the
+      default route out is the WAN, the VPN is opt-in) while the vpn-class pod's differs — two
+      paths live at once — and the vpn-class pod still reaches in-cluster destinations over the
+      local path, so one pod uses both paths simultaneously.
+spec:
+  timeouts:
+    apply: 60s
+    assert: 2m
+    exec: 3m
+  steps:
+    # -----------------------------------------------------------------------------------
+    - name: given-three-pods-differing-only-in-their-declared-egress-class
+      try:
+        - apply:
+            file: fixtures/three-egress-pods.yaml
+        # Bare pods (restartPolicy: Never) — Ready == Running here.
+        - assert:
+            resource:
+              apiVersion: v1
+              kind: Pod
+              metadata: { name: egress-unlabelled }
+              status: { phase: Running }
+        - assert:
+            resource:
+              apiVersion: v1
+              kind: Pod
+              metadata: { name: egress-internet }
+              status: { phase: Running }
+        - assert:
+            resource:
+              apiVersion: v1
+              kind: Pod
+              metadata: { name: egress-vpn }
+              status: { phase: Running }
+    # -----------------------------------------------------------------------------------
+    - name: then-the-unlabelled-pod-takes-the-wan-and-the-vpn-path-is-live-too
+      try:
+        - script:
+            content: |
+              set -eu
+              ns="$NAMESPACE"
+              get_ip() {
+                # Echo the pod's observed public IP, or empty on failure.
+                kubectl -n "$ns" exec "$1" -c curl -- \
+                  curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
+              }
+
+              # Poll until all three answer AND the vpn class has diverged from the WAN:
+              # Cilium may still be programming the egress datapath just after the pods go
+              # Ready. Bounded by the loop, never a bare sleep-and-hope.
+              none="" inet="" vpn=""
+              for i in $(seq 1 12); do
+                [ -n "$none" ] || none="$(get_ip egress-unlabelled)"
+                [ -n "$inet" ] || inet="$(get_ip egress-internet)"
+                [ -n "$vpn" ]  || vpn="$(get_ip egress-vpn)"
+                if [ -n "$none" ] && [ -n "$inet" ] && [ -n "$vpn" ]; then
+                  [ "$vpn" != "$inet" ] && break
+                  # Steering may not have converged; re-read the vpn pod only.
+                  vpn=""
+                fi
+                sleep 5
+              done
+              echo "unlabelled_ip=$none internet_ip=$inet vpn_ip=$vpn"
+
+              [ -n "$none" ] || { echo "FAIL: the UNLABELLED pod never returned a public IP — a pod that declares no egress class must still reach the internet over the default WAN path (docs/03: 'the internet path is simply the default route out'); no path at all means the default was blackholed or steered into a dead tunnel" >&2; exit 1; }
+              [ -n "$inet" ] || { echo "FAIL: the internet-class pod never returned a public IP — the declared default path is down, so no comparison is possible" >&2; exit 1; }
+              [ -n "$vpn" ]  || { echo "FAIL: the vpn-class pod never returned a public IP — the second egress path is not carrying traffic, so the platform is not offering MULTIPLE paths (R-NET-04)" >&2; exit 1; }
+
+              # TOTALITY: no rule selects the unlabelled pod, so it must land on the WAN —
+              # the same WAN the explicitly-internet pod uses.
+              if [ "$none" != "$inet" ]; then
+                echo "FAIL: the UNLABELLED pod egressed $none but the internet-class pod egressed $inet. They must be the same path: docs/03 — 'Default is the WAN; the VPN is strictly opt-in. Every pod ... egress the local/private WAN by default and never the VPN.' A different IP means some rule is selecting a pod that never opted in." >&2
+                exit 1
+              fi
+              # COEXISTENCE: and the vpn path is a genuinely different path, live right now.
+              if [ "$vpn" = "$inet" ]; then
+                echo "FAIL: the vpn-class pod egressed the same public IP as the WAN ($vpn) — there is only ONE live egress path, so 'multiple egress paths selected by routing rules' (R-NET-04) does not hold" >&2
+                exit 1
+              fi
+              echo "EGRESS_DEFAULT_PATH_IS_WAN_OK"
+            check:
+              (contains($stdout, 'EGRESS_DEFAULT_PATH_IS_WAN_OK')): true
+    # -----------------------------------------------------------------------------------
+    - name: then-the-vpn-class-pod-still-reaches-private-destinations-locally
+      try:
+        - script:
+            # PER-DESTINATION SELECTION. The steering rule tunnels 0.0.0.0/0 MINUS all
+            # RFC1918 space, so a vpn-class pod's private-destination traffic must stay on the
+            # local datapath. The cluster resolver is an in-cluster (RFC1918) address reached
+            # from the pod's own /etc/resolv.conf — no literal is written here — so a
+            # successful lookup from the vpn-class pod proves the tunnel did not swallow its
+            # local traffic, i.e. ONE pod is using BOTH paths at the same time.
+            content: |
+              set -eu
+              ns="$NAMESPACE"
+              resolver="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                sh -c 'awk "/^nameserver/{print \$2; exit}" /etc/resolv.conf' 2>/dev/null || true)"
+              [ -n "$resolver" ] || { echo "FAIL: the vpn-class pod has no nameserver in /etc/resolv.conf, so its local-path reachability cannot be observed" >&2; exit 1; }
+              echo "vpn_pod_resolver=$resolver (in-cluster, RFC1918 — excluded from the tunnel)"
+
+              # A cluster-internal name, resolvable only by the in-cluster resolver: reaching
+              # it proves the private-destination path works from a vpn-class pod.
+              out=""
+              for i in $(seq 1 6); do
+                out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                  nslookup kubernetes.default.svc.cluster.local 2>/dev/null || true)"
+                case "$out" in *"Address"*) break ;; esac
+                sleep 5
+              done
+              case "$out" in
+                *Address*) : ;;
+                *) echo "FAIL: the vpn-class pod could not resolve kubernetes.default.svc.cluster.local via its in-cluster resolver ($resolver). Its RFC1918 traffic is being steered into the tunnel (or blackholed) instead of taking the local path — the two paths are not coexisting per destination, and the tunnel is behaving as a default route (docs/03: split-tunnel, 'only what Cilium points at tun0 uses the VPN')" >&2; exit 1 ;;
+              esac
+              echo "EGRESS_PATHS_COEXIST_PER_DESTINATION_OK"
+            check:
+              (contains($stdout, 'EGRESS_PATHS_COEXIST_PER_DESTINATION_OK')): true

conformance/tests/l2-behavioral/egress-default-path/fixtures/three-egress-pods.yaml

@@ -0,0 +1,67 @@
+# Three pods for the R-NET-04 default-path test — the SAME workload three times, differing
+# ONLY in how (or whether) they declare an egress class:
+#
+#   egress-unlabelled  NO egress-class label at all -> selected by NO routing rule
+#   egress-internet    egress-class: internet       -> the declared default path
+#   egress-vpn         egress-class: vpn            -> the only class any rule steers
+#
+# WHY AN UNLABELLED POD IS THE POINT. docs/03: "Every pod — and the nodes themselves — egress
+# the local/private WAN by default and **never** the VPN." The unlabelled pod is the case no
+# other test covers: nothing selects it, so the DATAPATH's default must carry it, and it must
+# come out the same WAN as the explicitly-`internet` pod. That is the totality half of
+# R-NET-04's "selected by routing rules" — absence of a rule is itself a path.
+#
+# The unlabelled pod is admissible here only because chainsaw's per-test namespace is not an
+# app namespace: Kyverno's require-egress-class-label (R-NET-05) rejects an unlabelled pod in
+# a namespace labelled homelab.keithjackson.dev/app-namespace=true. That admission guarantee
+# is R-NET-05's, and it is exactly why this test must run outside an app namespace to observe
+# the datapath's own default at all.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-unlabelled
+  labels:
+    app: egress-default-path-test
+spec:
+  containers:
+    - name: curl
+      image: curlimages/curl:8.11.1
+      command: ["sleep", "3600"]
+      resources:
+        requests: { cpu: 10m, memory: 16Mi }
+        limits: { memory: 32Mi }
+  restartPolicy: Never
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-internet
+  labels:
+    app: egress-default-path-test
+    homelab.keithjackson.dev/egress-class: internet
+spec:
+  containers:
+    - name: curl
+      image: curlimages/curl:8.11.1
+      command: ["sleep", "3600"]
+      resources:
+        requests: { cpu: 10m, memory: 16Mi }
+        limits: { memory: 32Mi }
+  restartPolicy: Never
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-vpn
+  labels:
+    app: egress-default-path-test
+    homelab.keithjackson.dev/egress-class: vpn
+spec:
+  containers:
+    - name: curl
+      image: curlimages/curl:8.11.1
+      command: ["sleep", "3600"]
+      resources:
+        requests: { cpu: 10m, memory: 16Mi }
+        limits: { memory: 32Mi }
+  restartPolicy: Never

conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml

@@ -68,6 +68,15 @@ spec:
   timeouts:
     apply: 60s
     assert: 180s
+  # -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
+  #    (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
+  #    (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
+  #    authored here. Test-level => in scope for the fixture apply AND every probe step below
+  #    (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
+  #    fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     # -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
     #    Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
@@ -139,12 +148,14 @@ spec:
                         env:
                           - name: GWVIP
                             value: ($gwvip)
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="gw-declared-conformance.home.keithjackson.dev"
+                            host="gw-declared-conformance.${BASE_DOMAIN}"
                             last=000
                             i=0
                             while [ "$i" -lt 30 ]; do
@@ -207,12 +218,14 @@ spec:
                         env:
                           - name: GWVIP
                             value: ($gwvip)
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="gw-undeclared-conformance.home.keithjackson.dev"
+                            host="gw-undeclared-conformance.${BASE_DOMAIN}"
                             reached_gw=no
                             i=0
                             while [ "$i" -lt 8 ]; do
@@ -300,3 +313,20 @@ spec:
               apiVersion: v1
               kind: Namespace
               name: conformance-gw-probe
+    # -----------------------------------------------------------------------------------
+    # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
+    # chainsaw already fails the test before reaching this step if any of them fails — but a
+    # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
+    # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
+    # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
+    # forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
+    # — which is precisely what docs/07 asks for ("These proofs are part of the platform
+    # acceptance suite, not one-time checks"). This step exists only to emit that token, and it
+    # is reached only when everything above has passed. Asserted by
+    # tests/l0-static/auditability-proofs-producible.
+    - name: then-emit-the-proof-of-pass-sentinel
+      try:
+        - script:
+            content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
+            check:
+              (contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true

conformance/tests/l2-behavioral/gateway-ingress-reachable/fixtures/app-and-probe.yaml

@@ -10,6 +10,13 @@
 # The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
 # rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
 # probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
+#
+# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
+# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
+# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
+# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
+# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
+# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
 ---
 apiVersion: v1
 kind: Namespace
@@ -134,7 +141,8 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - gw-declared-conformance.home.keithjackson.dev
+    # gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
+    - (join('.', ['gw-declared-conformance', $baseDomain]))
   rules:
     - matches:
         - path:
@@ -158,7 +166,8 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - gw-undeclared-conformance.home.keithjackson.dev
+    # gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
+    - (join('.', ['gw-undeclared-conformance', $baseDomain]))
   rules:
     - matches:
         - path:

conformance/tests/l2-behavioral/ha-control-plane/chainsaw-test.yaml

@@ -0,0 +1,379 @@
+# =============================================================================
+# L2 BEHAVIORAL — the cluster under test IS an HA k3s control plane  (R-PLAT-01)  [RUNNABLE]
+# Catalog: docs/14 L2 row ("real runtime behavior, probes against a cluster")
+# Spec:    docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
+#          production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
+#          exercise the same multi-node topology.")
+#          docs/01 "Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02"
+#          docs/02 table: "Control plane | HA embedded etcd (k3s server on all 3) | HA
+#                          embedded etcd (k3s server on all 3)"
+#                         "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
+#          docs/02 §"k3s topology decisions": "HA control plane with **embedded etcd** across
+#          all 3 nodes (no external datastore)"; "Nodes are both control-plane and workload
+#          (3-node converged cluster)".
+#
+# WHY L2 — AND WHAT L0 ALREADY DID
+# --------------------------------
+# tests/l0-static/ha-control-plane-as-code (same requirement label) proves the AUTHORED
+# half offline: the repo builds a 3-server embedded-etcd control plane in both environments,
+# one founder, no external datastore, a leader-elected control-plane-only VIP in the serving
+# cert's SANs. It cannot prove that any of it is TRUE OF A RUNNING CLUSTER — an inventory
+# entry is not a node, and a templated DaemonSet is not a floating VIP. That is this test:
+# every assertion below is read from the cluster the suite is pointed at.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): no address, hostname, node count or range is written
+# here. The expected node count comes from the cluster's OWN declared identity
+# (flux-system/cluster-vars CLUSTER_ENV, via a test-level chainsaw binding — the same
+# ConfigMap Flux substitutes from); the control-plane VIP is read out of the kube-vip
+# DaemonSet the cluster is actually running; the server addresses come from the Node
+# objects; and the LoadBalancer pool the VIP must stay outside of is read from cluster-vars
+# too. R-PLAT-01 itself distinguishes prod from dev/test ("3 nodes ... in prod; dev/test MAY
+# run fewer"), so the assertion is necessarily environment-AWARE — but it takes the
+# environment from the cluster, never from a flag or a literal, which is precisely the
+# mechanism the tests/README.md convention prescribes.
+#
+# GIVEN the cluster this suite is pointed at,
+# WHEN  its Nodes, its kube-vip DaemonSet and its kube-apiserver endpoints are inspected,
+# THEN  DISTRO:  every node's kubelet is a k3s build (`+k3s`) — the distro really is k3s;
+#       SERVERS: every Ready node is BOTH control-plane AND etcd (the converged 3-node
+#                topology docs/02 fixes — no agent-only node, no control-plane-only node),
+#                the count is ODD and quorum-capable (tolerates >= 1 loss), and it is
+#                EXACTLY three when the cluster declares itself prod;
+#       ETCD:    every one of those nodes is a REGISTERED embedded-etcd member (k3s stamps
+#                `etcd.k3s.cattle.io/node-name` on a node that joined the etcd cluster), so
+#                the datastore is embedded and replicated across all of them;
+#       VIP:     kube-vip is Ready on EVERY control-plane node, is control-plane-only
+#                (svc_enable=false — it must not contend with Cilium LB-IPAM for ARP), and
+#                the address it floats is neither a node address, nor the ingress Gateway
+#                VIP, nor inside the Cilium LoadBalancer pool;
+#       ENDPOINT: the kube-apiserver answers `/readyz` THROUGH THAT VIP over TLS that
+#                validates against the cluster CA for the VIP itself — i.e. the VIP really
+#                fronts the API and the serving cert really carries it;
+#       EVERY SERVER SERVES: the apiserver also answers `/readyz` on EVERY server's own
+#                address, again with a CA-valid cert for that address. This is the assertion
+#                that separates a real HA control plane from one server plus two joiners:
+#                if only the founder ran an apiserver, the other addresses would refuse.
+#
+# THE PROBE RUNS IN-CLUSTER, DELIBERATELY. An earlier draft dialled the VIP and the node
+# addresses from the RUNNER. That works from a host on the node LAN (verified against prod)
+# and FAILS against dev, whose nodes sit behind a NAT the runner reaches only via a
+# forwarded port — i.e. it would have made the test a property of where it was launched
+# from. A Job on the cluster is on the node network in both environments, so the probe is
+# the same probe everywhere. The Job is created imperatively (kubectl heredoc) rather than
+# as a fixture file because its address list is DISCOVERED from the cluster at run time —
+# writing it into a fixture is exactly the per-environment literal R-ENV-02 forbids.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
+#   - FAILOVER / SURVIVAL. Nothing here removes a node. That etcd keeps quorum, that the VIP
+#     migrates, and that the API stays reachable when a server dies is an L3 fault-injection
+#     claim and is proven NOWHERE in this suite yet (tests/l3-fault-injection/node-loss
+#     drains a node for R-STATE-02's data-availability claim and asserts nothing about the
+#     control plane). This is the largest remaining R-PLAT-01 gap.
+#   - THE NEXT environment. That the repo would build an HA control plane again is the L0
+#     half (tests/l0-static/ha-control-plane-as-code).
+#   - "PRODUCTION-GRADE" as a capacity/performance judgement (operator's, docs/02 + §2 of
+#     infra/PROD-PROVISIONING.md).
+#   - THAT THE RUNNING cilium-agents ACTUALLY DIAL THIS VIP. As of 2026-09-06 the repository
+#     authors `K3S_API_HOST` = the control-plane VIP in both environments (asserted at L0 by
+#     tests/l0-static/ha-control-plane-as-code, on both sides of the [parity] contract), but
+#     this test does not read the agents' live `KUBERNETES_SERVICE_HOST`. Until the operator
+#     applies that change the LIVE agents still carry the PRIMARY NODE's address — measured
+#     2026-09-06: all three prod agents on 192.168.2.10 while kube-vip held 192.168.2.9. An
+#     in-cluster assertion on the agents' endpoint belongs here and is NOT yet written; it is
+#     the obvious follow-up once the change is applied and would catch drift the L0 check
+#     cannot see (a hand-edited DaemonSet, a Helm value overridden out of band).
+#   - etcd health beyond membership registration (raft leader, DB size, alarms) — k3s exposes
+#     it only on the node-local etcd metrics port, which is not reachable through the
+#     Kubernetes API this test is allowed to use.
+#
+# CLEANUP: the probe Job lives in chainsaw's per-test namespace and is deleted with it
+# (config/chainsaw-config.yaml skipDelete: false). Everything else is read-only: this test
+# creates nothing outside its own namespace and mutates no platform object.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: ha-control-plane
+  labels:
+    conformance.home.k3s/R-PLAT-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-PLAT-01"
+    conformance.home.k3s/layer: "L2"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Given the cluster under test, when its Nodes, kube-vip DaemonSet and apiserver
+      endpoints are probed, then every node is a k3s server that is both control-plane and a
+      registered embedded-etcd member, the count is odd and quorum-capable (exactly three
+      when the cluster declares itself prod), kube-vip floats a control-plane-only VIP that
+      collides with no allocator, and the apiserver answers /readyz through that VIP AND on
+      every server's own address with a CA-valid cert.
+spec:
+  timeouts:
+    apply: 60s
+    assert: 2m
+    exec: 3m
+  # -- R-ENV-02: the environment's identity and its address allocations are READ FROM THE
+  #    CLUSTER UNDER TEST. flux-system/cluster-vars is the ConfigMap Flux itself substitutes
+  #    from, so these are the same values the platform was rendered with. Test-level, so
+  #    they are in scope for every step (step outputs/bindings are step-scoped).
+  bindings:
+    - name: clusterEnv
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CLUSTER_ENV)
+    - name: gatewayVip
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
+    - name: lbPoolStart
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_START)
+    - name: lbPoolStop
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.CILIUM_LB_POOL_STOP)
+  steps:
+    # -----------------------------------------------------------------------------------
+    - name: then-the-node-set-is-a-quorum-capable-converged-k3s-control-plane
+      description: >-
+        Read the Node objects: k3s on every node, every node both control-plane and a
+        registered embedded-etcd member, an odd quorum-capable count, exactly three in prod.
+      try:
+        - script:
+            env:
+              - name: CLUSTER_ENV
+                value: ($clusterEnv)
+            content: |
+              set -eu
+              fail() { echo "FAIL: $*" >&2; exit 1; }
+
+              total="$(kubectl get nodes -o name | grep -c . || true)"
+              [ "$total" -ge 1 ] || fail "the cluster reports NO nodes at all"
+
+              # --- k3s, on every node (R-PLAT-01 names the distro; docs/01 ADR-0001) -----
+              # A non-k3s kubelet here would mean the platform is not running on k3s.
+              notk3s="$(kubectl get nodes \
+                -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.status.nodeInfo.kubeletVersion}{"\n"}{end}' \
+                | grep -v '+k3s' | grep -c . || true)"
+              [ "$notk3s" = "0" ] \
+                || fail "$notk3s node(s) do not run a k3s kubelet — R-PLAT-01 requires the platform to run on k3s"
+
+              # --- every node is BOTH control-plane AND etcd (docs/02 converged topology) -
+              # Counted by label EXISTENCE, so it does not matter whether k3s writes "true"
+              # or "". A control-plane node without the etcd role would be a server with an
+              # external/absent datastore; an etcd node without control-plane would be an
+              # isolated datastore member. Both are excluded by requiring all three counts
+              # to be equal.
+              cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
+              etcd="$(kubectl get nodes -l node-role.kubernetes.io/etcd -o name | grep -c . || true)"
+              echo "nodes_total=$total control_plane=$cp etcd=$etcd cluster_env=${CLUSTER_ENV:-<unset>}"
+              [ "$cp" = "$total" ] \
+                || fail "$cp of $total nodes carry node-role.kubernetes.io/control-plane — docs/02 fixes a converged cluster where every node is a control-plane member ('k3s server on all 3')"
+              [ "$etcd" = "$total" ] \
+                || fail "$etcd of $total nodes carry node-role.kubernetes.io/etcd — docs/02 requires embedded etcd 'across all 3 nodes (no external datastore)'"
+
+              # --- every node is a REGISTERED embedded-etcd member -----------------------
+              # k3s stamps etcd.k3s.cattle.io/node-name on a node that actually joined the
+              # embedded-etcd cluster. The role label alone is a label; this annotation is
+              # the datastore's own record of membership.
+              members="$(kubectl get nodes \
+                -o jsonpath='{range .items[*]}{.metadata.annotations.etcd\.k3s\.cattle\.io/node-name}{"\n"}{end}' \
+                | grep -c . || true)"
+              [ "$members" = "$total" ] \
+                || fail "$members of $total nodes are registered embedded-etcd members (etcd.k3s.cattle.io/node-name) — the datastore is not replicated across every server"
+
+              # --- Ready ---------------------------------------------------------------
+              notready="$(kubectl get nodes \
+                -o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \
+                | grep -v '=True$' | grep -c . || true)"
+              [ "$notready" = "0" ] \
+                || fail "$notready node(s) are not Ready — a control plane missing a member is not the HA topology under test"
+
+              # --- quorum arithmetic ---------------------------------------------------
+              # Embedded etcd needs floor(N/2)+1 members to commit. Fault tolerance is
+              # N - quorum; 'production-grade HA' must survive at least one loss, which
+              # rules out 1 and 2 members. An EVEN count buys no extra tolerance and adds
+              # a tie, so it is also rejected.
+              quorum=$(( total / 2 + 1 ))
+              tolerance=$(( total - quorum ))
+              echo "quorum=$quorum tolerance=$tolerance"
+              [ "$tolerance" -ge 1 ] \
+                || fail "$total etcd member(s) give quorum $quorum and tolerate $tolerance loss(es) — an HA control plane must survive losing a server"
+              [ $(( total % 2 )) -eq 1 ] \
+                || fail "$total etcd members is EVEN: quorum is the same as for $(( total - 1 )) members, so the extra member adds a tie instead of tolerance"
+
+              # --- the count R-PLAT-01 itself fixes, per environment --------------------
+              # The environment is the cluster's OWN CLUSTER_ENV (cluster-vars), never a
+              # flag: 'k3s across 3 nodes ... in prod; dev/test MAY run fewer nodes but MUST
+              # exercise the same multi-node topology'.
+              case "${CLUSTER_ENV:-}" in
+                prod)
+                  [ "$total" = "3" ] \
+                    || fail "this cluster declares CLUSTER_ENV=prod and has $total nodes — R-PLAT-01 requires 'k3s across 3 nodes ... in prod' (docs/02: '3 physical nodes')" ;;
+                "")
+                  fail "flux-system/cluster-vars has no CLUSTER_ENV — the cluster does not declare which environment it is, so R-PLAT-01's prod clause has no referent (and no literal may be substituted for it)" ;;
+                *)
+                  [ "$total" -ge 2 ] \
+                    || fail "this cluster declares CLUSTER_ENV=${CLUSTER_ENV} and has $total node(s) — dev/test may run fewer nodes than prod but MUST 'exercise the same multi-node topology'" ;;
+              esac
+              echo "HA_TOPOLOGY_OK"
+            check:
+              (contains($stdout, 'HA_TOPOLOGY_OK')): true
+    # -----------------------------------------------------------------------------------
+    - name: then-a-control-plane-only-vip-is-floated-on-every-server
+      description: >-
+        Read the running kube-vip DaemonSet: Ready on every control-plane node,
+        control-plane-only, and floating an address that collides with no other allocator.
+      try:
+        - script:
+            env:
+              - name: GATEWAY_VIP
+                value: ($gatewayVip)
+              - name: LB_POOL_START
+                value: ($lbPoolStart)
+              - name: LB_POOL_STOP
+                value: ($lbPoolStop)
+            content: |
+              set -eu
+              fail() { echo "FAIL: $*" >&2; exit 1; }
+              # dotted quad -> integer, for range comparison without ipcalc/python.
+              n() { echo "$1" | awk -F. '{printf "%d", ($1*16777216)+($2*65536)+($3*256)+$4}'; }
+
+              # The DaemonSet is found by the label roles/kube_vip stamps on it, not by a
+              # hardcoded name in a namespace this test guessed.
+              ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
+              [ -n "$ds" ] \
+                || fail "no kube-vip DaemonSet is running in kube-system — the kube-apiserver has no floating endpoint, so losing the founder takes the API endpoint with it"
+
+              desired="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.desiredNumberScheduled}')"
+              ready="$(kubectl -n kube-system get "$ds" -o jsonpath='{.status.numberReady}')"
+              cp="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name | grep -c . || true)"
+              vip="$(kubectl -n kube-system get "$ds" \
+                -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
+              cp_enable="$(kubectl -n kube-system get "$ds" \
+                -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="cp_enable")].value}')"
+              svc_enable="$(kubectl -n kube-system get "$ds" \
+                -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="svc_enable")].value}')"
+              echo "kube_vip=$ds ready=$ready/$desired control_plane_nodes=$cp vip=$vip cp_enable=$cp_enable svc_enable=$svc_enable"
+
+              [ -n "$vip" ] \
+                || fail "the kube-vip DaemonSet announces no address — nothing is floated"
+              [ "$cp_enable" = "true" ] \
+                || fail "kube-vip runs with cp_enable=$cp_enable — control-plane mode is off, so it floats no API VIP"
+              [ "$svc_enable" = "false" ] \
+                || fail "kube-vip runs with svc_enable=$svc_enable — it would also serve Service LoadBalancers and contend with Cilium LB-IPAM for ARP on the same addresses (ADR-0002 gives Service LB to Cilium)"
+              [ "$desired" = "$cp" ] \
+                || fail "kube-vip is scheduled on $desired node(s) but there are $cp control-plane nodes — the VIP must be able to move to ANY server, or its failover set is smaller than the control plane"
+              [ "$ready" = "$desired" ] \
+                || fail "kube-vip is Ready on $ready of $desired nodes — a VIP holder that is not running cannot take the address over"
+
+              # --- the VIP collides with nothing --------------------------------------
+              node_ips="$(kubectl get nodes \
+                -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' \
+                | grep -c "^${vip}$" || true)"
+              [ "$node_ips" = "0" ] \
+                || fail "the floated VIP $vip is ALSO a node's own InternalIP — a VIP that is a node's address cannot float off that node"
+              [ "$vip" != "${GATEWAY_VIP:-}" ] \
+                || fail "the control-plane VIP and the ingress Gateway VIP are the same address ($vip) — kube-vip and Cilium would both ARP for it"
+              if [ -n "${LB_POOL_START:-}" ] && [ -n "${LB_POOL_STOP:-}" ]; then
+                v="$(n "$vip")"; lo="$(n "$LB_POOL_START")"; hi="$(n "$LB_POOL_STOP")"
+                [ "$lo" -le "$hi" ] \
+                  || fail "the cluster's Cilium LB pool ${LB_POOL_START}-${LB_POOL_STOP} is inverted, so the VIP cannot be proven to be outside it"
+                if [ "$v" -ge "$lo" ] && [ "$v" -le "$hi" ]; then
+                  fail "the control-plane VIP $vip lies INSIDE this cluster's Cilium LoadBalancer pool ${LB_POOL_START}-${LB_POOL_STOP} — LB-IPAM could hand the kube-API's VIP to a Service"
+                fi
+              else
+                fail "flux-system/cluster-vars declares no CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and the Service-LB allocator cannot be checked, and no literal range may be substituted for it (R-ENV-02)"
+              fi
+              echo "HA_VIP_CONTROL_PLANE_ONLY_OK"
+            check:
+              (contains($stdout, 'HA_VIP_CONTROL_PLANE_ONLY_OK')): true
+    # -----------------------------------------------------------------------------------
+    - name: then-the-apiserver-answers-through-the-vip-and-on-every-server
+      description: >-
+        From inside the cluster, probe /readyz on the floated VIP and on every server's own
+        address, validating TLS against the cluster CA for the address dialled.
+      try:
+        - script:
+            # The address list is DISCOVERED here (Nodes + the kube-vip DaemonSet), so the
+            # Job cannot be a static fixture — writing the addresses down is the
+            # per-environment literal R-ENV-02 forbids. Hence the imperative heredoc, the
+            # same pattern tests/l3-fault-injection/dr-restore uses for run-time-shaped
+            # fixtures. The Job lives in chainsaw's per-test namespace and is torn down with
+            # it; nothing outside that namespace is created or changed.
+            content: |
+              set -eu
+              ns="$NAMESPACE"
+              fail() { echo "FAIL: $*" >&2; exit 1; }
+
+              ds="$(kubectl -n kube-system get ds -l app.kubernetes.io/name=kube-vip-ds -o name | head -1)"
+              vip="$(kubectl -n kube-system get "$ds" \
+                -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="address")].value}')"
+              servers="$(kubectl get nodes -l node-role.kubernetes.io/control-plane \
+                -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{" "}{end}')"
+              [ -n "$vip" ] || fail "no floated VIP to probe"
+              [ -n "$servers" ] || fail "no control-plane node addresses to probe"
+              echo "probing vip=$vip servers=$servers"
+
+              # The probe pod authenticates with its own ServiceAccount token and validates
+              # TLS against the cluster CA, both from the projected SA volume. /readyz is
+              # authorized for any authenticated identity via the built-in
+              # system:public-info-viewer role — anonymous access is NOT relied on (verified:
+              # an unauthenticated /readyz on this platform answers 401).
+              kubectl -n "$ns" apply -f - <<EOF
+              apiVersion: batch/v1
+              kind: Job
+              metadata:
+                name: ha-api-endpoint-probe
+              spec:
+                backoffLimit: 1
+                activeDeadlineSeconds: 150
+                template:
+                  spec:
+                    restartPolicy: Never
+                    containers:
+                      - name: probe
+                        image: curlimages/curl:8.11.1
+                        env:
+                          - name: VIP
+                            value: "${vip}"
+                          - name: SERVERS
+                            value: "${servers}"
+                        command:
+                          - sh
+                          - -c
+                          - |
+                            set -u
+                            SA=/var/run/secrets/kubernetes.io/serviceaccount
+                            probe() {   # \$1 = address to dial
+                              curl -sS --max-time 15 \\
+                                --cacert "\$SA/ca.crt" \\
+                                -H "Authorization: Bearer \$(cat \$SA/token)" \\
+                                "https://\$1:6443/readyz"
+                            }
+                            rc=0
+                            out="\$(probe "\$VIP" 2>&1)" || rc=1
+                            echo "vip \$VIP -> [\$out]"
+                            if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
+                              echo "FAIL: the kube-apiserver did not answer /readyz through the floated VIP \$VIP with a CA-valid cert for that address — the VIP does not front the API (or it is missing from the serving cert's SANs)"
+                              exit 1
+                            fi
+                            for s in \$SERVERS; do
+                              rc=0
+                              out="\$(probe "\$s" 2>&1)" || rc=1
+                              echo "server \$s -> [\$out]"
+                              if [ "\$rc" != "0" ] || [ "\$out" != "ok" ]; then
+                                echo "FAIL: server \$s does not answer /readyz with a CA-valid cert for its own address — it is not running a kube-apiserver, so the control plane is not HA across every node"
+                                exit 1
+                              fi
+                            done
+                            echo "HA_API_ENDPOINT_OK"
+              EOF
+
+              # Bounded wait; on failure surface the probe's own output before failing.
+              if ! kubectl -n "$ns" wait --for=condition=complete --timeout=140s job/ha-api-endpoint-probe; then
+                kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40 || true
+                fail "the in-cluster API-endpoint probe did not complete — see its log above"
+              fi
+              logs="$(kubectl -n "$ns" logs job/ha-api-endpoint-probe --tail=40)"
+              echo "$logs"
+              case "$logs" in
+                *HA_API_ENDPOINT_OK*) echo "HA_API_ENDPOINT_REACHABLE_OK" ;;
+                *) fail "the probe completed without reporting success" ;;
+              esac
+            check:
+              (contains($stdout, 'HA_API_ENDPOINT_REACHABLE_OK')): true

conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml

@@ -3,10 +3,15 @@
 # Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client
 # (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof.
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the
+# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by
+# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which
+# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster.
+#
 # GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm,
 # WHEN  an unauthenticated browser starts the OIDC login (GET /login/generic_oauth)
 #   ->  THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on
-#       keycloak.home.keithjackson.dev/realms/homelab — proving SSO is driven by the one IdP
+#       keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP
 #       (R-IDP-01: central Identity Provider as the single user directory).
 #
 # Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up.
@@ -30,6 +35,11 @@ spec:
     apply: 60s
     assert: 3m
     exec: 3m
+  # -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST
+  #    (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from).
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     - name: given-grafana-is-an-oidc-client
       try:
@@ -59,6 +69,9 @@ spec:
                     containers:
                       - name: probe
                         image: busybox:1.36.1
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
                         command:
                           - sh
                           - -c
@@ -68,13 +81,14 @@ spec:
                             # busybox wget has no --max-redirect; -S dumps every response header
                             # (incl. the initial 302 Location to the IdP) before following, so the
                             # grep below still matches the Keycloak authorize URL. (Grafana verified
-                            # live: 302 -> keycloak.home.keithjackson.dev/realms/homelab/.../auth.)
+                            # live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.)
+                            idp="keycloak.${BASE_DOMAIN}/realms/homelab"
                             out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)"
                             echo "$out"
-                            if echo "$out" | grep -qi 'keycloak.home.keithjackson.dev/realms/homelab'; then
+                            if echo "$out" | grep -qiF "$idp"; then
-                              echo GRAFANA_OIDC_REDIRECT_OK
+                              echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}"
                             else
-                              echo "FAIL: Grafana did not redirect to the Keycloak homelab realm"; exit 1
+                              echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1
                             fi
         - assert:
             resource:

conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml

@@ -3,8 +3,13 @@
 # Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
 # Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014.
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test names NO hostname. It talks to oauth2-proxy over its
+# in-cluster Service DNS and asserts HTTP behaviour, so the same authored test is correct on dev and
+# on prod. The issuer host below is written as ${BASE_DOMAIN} because that is what the platform
+# substitutes; it is prose, not an input — nothing here needs parameterizing.
+#
 # GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available
-#       after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is
+#       after successfully discovering https://keycloak.${BASE_DOMAIN}/realms/homelab — this is
 #       the proxy->Keycloak flow working),
 # WHEN  an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth)
 #   ->  THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal

conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml

@@ -3,8 +3,13 @@
 # Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
 # Spec:    docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
+# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
+# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
+# prod's on prod, with nothing to pass at run time.
+#
 # GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
-# WHEN  an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and
+# WHEN  an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
 #       verifies the chain against the SYSTEM trust store,
 # THEN  the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
 #       chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
@@ -36,6 +41,11 @@ metadata:
       that chains to a publicly-trusted CA root (LE production); the wildcard SAN
       covers internal-only hostnames too.
 spec:
+  # -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
+  #    — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     - name: then-gateway-cert-chains-to-trusted-root
       try:
@@ -54,13 +64,16 @@ spec:
                     containers:
                       - name: probe
                         image: curlimages/curl:8.11.1
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
                         command:
                           - sh
                           - -c
                           - |
                             set -eu
                             # Probe TWO wildcard hostnames — one "public", one "internal-only". Both
-                            # match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA
+                            # match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
                             # bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
                             # R-TLS-03 (the same trusted wildcard covers internal-only names). Names
                             # resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
@@ -68,7 +81,7 @@ spec:
                             # the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
                             # hostname fails. TLS terminates at the gateway before HTTP routing, so no
                             # backend is needed.
-                            for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do
+                            for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
                               curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
                                    -o /dev/null "https://${host}/"
                               echo "TLS_TRUSTED ${host}"
@@ -80,3 +93,20 @@ spec:
               kind: Job
               metadata: { name: tls-probe }
               status: { succeeded: 1 }
+    # -----------------------------------------------------------------------------------
+    # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
+    # chainsaw already fails the test before reaching this step if any of them fails — but a
+    # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
+    # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
+    # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
+    # forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
+    # — which is precisely what docs/07 asks for ("These proofs are part of the platform
+    # acceptance suite, not one-time checks"). This step exists only to emit that token, and it
+    # is reached only when everything above has passed. Asserted by
+    # tests/l0-static/auditability-proofs-producible.
+    - name: then-emit-the-proof-of-pass-sentinel
+      try:
+        - script:
+            content: echo TLS_TRUSTED_PROVEN_OK
+            check:
+              (contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true

conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml

@@ -40,10 +40,17 @@
 #      alternative source for the data.
 #
 # KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them):
-#   * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are
+#   * OFF-HOST is NOT proven, only OFF-CLUSTER — TRUE AS RUN, and being fixed. On the
-#     VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9),
+#     dev estate the k3s nodes are VMs on the same Proxmox host that runs the
-#     so this drill does not survive loss of that hypervisor. Off-site/off-host
+#     versitygw/MinIO S3 target (192.168.1.9), so this drill does not survive loss of
-#     replication is an operator concern outside R-STATE-03's wording.
+#     that hypervisor: one machine takes the cluster AND its backups.
+#     That is exactly why the operator moved off-cluster DR to Cloudflare R2 on
+#     2026-09-05 (commit 32d6766 — S3_BACKUP_ENDPOINT is now an R2 endpoint). Once that
+#     merges and this drill is RE-RUN against R2, off-host stops being a gap and becomes
+#     proven, because the target no longer shares a failure domain with the cluster.
+#     UNTIL THEN THIS TEST'S PASS IS AGAINST MinIO. R-STATE-03 is currently proven only
+#     for the same-host target; re-proving it against R2 is a prerequisite for claiming
+#     off-host DR, and the ledger row that proves R-STATE-03 predates the move.
 #   * The test does NOT read the backup objects out of the bucket with an S3 client.
 #     Doing so would require handing the conformance runner the backup credential,
 #     widening a privileged test identity for weaker evidence than (1)-(4) above.

infra/BACKUP-R2.md

@@ -0,0 +1,315 @@
+# Off-cluster backup on Cloudflare R2 — operator procedure
+
+**Owner:** WS-BACKUP · **Serves:** R-STATE-03 (DR restore), R-IDP-05 (Keycloak DB), R-ENV-02 (one
+environment-agnostic path) · **Status:** authored and staged, **NOT switched on**.
+
+> **Read this first.** Velero-against-R2 and CNPG/barman-against-R2 are **UNPROVEN in this repo**.
+> Nothing here claims they work. What *is* proven is in [§7](#7-what-is-proven-and-what-is-not).
+> R-STATE-03 is currently PROVEN **against MinIO only** and must be re-proven against R2 before this
+> migration can be called done.
+
+## Why R2 at all
+
+The off-cluster target used to be a MinIO on the Proxmox host (`http://192.168.1.9:9000`). Two things
+killed it, both recorded on 2026-09-05:
+
+1. **Prod physically cannot reach it.** Prod is on the SOHO network `192.168.2.0/24`, which is
+   pfSense's **WAN**; the MinIO sits inside pfSense's LAN `192.168.1.0/24`. Measured *from a pod inside
+   prod* — the workstation's own reachability proves nothing about pod egress and has misled this
+   project once already — `curl http://192.168.1.9:9000/minio/health/live` timed out after 15 s.
+   Reaching it would need a WAN→LAN rule that does not exist.
+2. **Dev's "off-cluster" backup was not off-cluster.** That MinIO runs on the *same Proxmox host* as
+   dev's VMs, so it shared a failure domain with the thing it was backing up.
+
+R2 needs no firewall change, is genuinely off-site, and is already trusted with the OpenTofu remote
+state (`infra/tofu/REMOTE-STATE.md`) — the same account, the same kind of token. It is explicitly an
+interim decision: *"that's where backups will go for the time being until a better on-prem backup
+solution is created."*
+
+---
+
+## 1. Cloudflare: buckets, token, account id
+
+All of this is in the Cloudflare dashboard; none of it touches the cluster.
+
+1. **R2 → Create bucket** → `velero`. Location hint: whatever is nearest; jurisdiction default.
+2. **R2 → Create bucket** → `cnpg`.
+   Both names are fixed by the manifests: `bucket: velero` in `platform/velero/helmrelease.yaml`, and
+   `cnpg` via `CNPG_BACKUP_PATH` (`s3://cnpg/dev`, `s3://cnpg/prod`). Use exactly those names or change
+   the manifests — do not improvise.
+3. **R2 → Manage API Tokens → Create API Token**
+   - Permission: **Object Read & Write** (not "Admin Read & Write" — the cluster never needs to create
+     or delete buckets, and a token that can is a token that can delete your backups).
+   - **Specify bucket(s):** `velero` **and** `cnpg`. One token scoped to both — the manifests already
+     assume one shared key, and `s3check` proves both buckets with it.
+   - Copy the **Access Key ID** and **Secret Access Key**. They are shown **once**. Put them in the
+     password manager before you do anything else.
+4. **Account id** — 32 hex characters, visible in the dashboard URL and in the R2 S3 endpoint the
+   token page shows you.
+
+   **This repo already knows it.** `infra/tofu/backend.r2.hcl` commits
+   `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com` for the OpenTofu state bucket,
+   with the (correct) note that an account id is not a credential. **If the backup buckets live in the
+   same Cloudflare account as the tofu state — they should; there is no reason for a second account —
+   then the endpoint is already known and you do not need to look anything up:**
+
+   ```
+   https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+   ```
+
+   Confirm it matches the endpoint on the token page. If it does not, you have two accounts and the
+   `velero`/`cnpg` buckets are in the other one — use that one's id, and say so in the DECISIONS-LOG.
+
+The endpoint form is always `https://<account-id>.r2.cloudflarestorage.com` — **no bucket in the
+hostname** (path-style; see [§6](#6-r2-specific-gotchas)).
+
+---
+
+## 2. Repo values to set
+
+Only **one** non-secret value is missing, and it is not in a secret file:
+
+| File | Key | From | To |
+|---|---|---|---|
+| `clusters/base/cluster-vars.yaml` | `S3_BACKUP_ENDPOINT` | `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` | `https://<account-id>.r2.cloudflarestorage.com` |
+
+Already authored, **do not change**: `S3_BACKUP_REGION: "auto"`, `VELERO_BACKUP_PREFIX` (dev `dev` /
+prod `prod`), `CNPG_BACKUP_PATH` (`s3://cnpg/dev` / `s3://cnpg/prod`), and `checksumAlgorithm: ""` on
+the Velero BackupStorageLocation.
+
+The placeholder is deliberately loud, and the guards match it as a **substring** — `bootstrap/standup.sh`
+refuses to probe it and refuses to resume `velero` while it is there.
+
+---
+
+## 3. Credentials: four Secrets, one command
+
+The R2 key is read by **four** committed SOPS Secrets, in **two** different shapes. No manifest changes
+are needed — only the encrypted values change:
+
+| File | Consumer | Shape |
+|---|---|---|
+| `platform/velero/velero-cloud.sops.yaml` | Velero (`velero-cloud-credentials`, ns `velero`) | `stringData.cloud` — an **AWS INI file**: `[default]` + `aws_access_key_id=` / `aws_secret_access_key=` |
+| `platform/keycloak/keycloak-db-backup-credentials.sops.yaml` | CNPG `keycloak-db` (ns `keycloak`) | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+| `apps/wave-0/nextcloud-db-backup-credentials.sops.yaml` | CNPG `nextcloud-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+| `apps/wave-0/semaphore-db-backup-credentials.sops.yaml` | CNPG `semaphore-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+
+A Secret is namespace-scoped and a Pod reads only its own namespace's, which is why there are four
+copies of one key rather than one.
+
+### The one command
+
+```bash
+./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
+```
+
+It prompts for the two values (never echoed, never in shell history, never written to a file), writes
+them into all four Secrets with `sops set` — which encrypts in place, so **no decrypted copy of any
+Secret is ever created** — then decrypts each file back and proves all four now hold the same key by
+comparing 32-bit fingerprints. It never prints a credential.
+
+There is deliberately **no plaintext template to fill in and encrypt**. The
+`cp x.example x.yaml && $EDITOR && sops -e -i` pattern leaves a plaintext Kubernetes Secret sitting in
+the working tree, one `git add -A` away from being in git forever; `sops set` removes that window
+entirely. If you would rather edit by hand, `sops platform/velero/velero-cloud.sops.yaml` decrypts to
+`$EDITOR` and re-encrypts on save — also with no plaintext on disk — but then you must repeat it four
+times and keep them identical yourself.
+
+To check the four files agree without changing anything:
+
+```bash
+./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
+```
+
+### What is NOT changed
+
+`platform/backup/minio-credentials.sops.yaml` keeps the **MinIO** key. That is not an oversight — it
+is what you need to read the pre-migration backups ([§8](#8-going-back-to-minio)). Its
+`.yaml.example` sibling documents the MinIO shape and stays accurate for the MinIO target.
+
+Review the diff before committing: you should see **only** `ENC[...]` blobs and `lastmodified`/`mac`
+change in four files. If a key name or a `metadata.name` moved, stop.
+
+---
+
+## 4. Prove the target BEFORE merging
+
+```bash
+K=infra/ansible/.cache/dev.kubeconfig        # or prod.kubeconfig
+./infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig $K --expect-server https://192.168.2.50:6443 \
+  --s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
+```
+
+`--s3-endpoint` probes a **candidate** endpoint instead of the live `cluster-vars` value, so the target
+is proven before it is authored. Run it against **both** clusters — prod's egress is the whole reason
+this migration exists, and it is a different network position from dev's.
+
+`s3check` runs a throwaway pod per credential and, for each, proves:
+
+1. the committed credential **loads and authenticates**, in the exact shape its consumer reads it
+   (Velero's INI file mounted from the `cloud` key; CNPG's discrete `ACCESS_KEY_ID`/`ACCESS_SECRET_KEY`);
+2. the bucket **exists and is listable** under this environment's own prefix (`ListObjectsV2`, the
+   operation Velero's BSL sync and Barman's WAL scan both perform);
+3. **PUT → GET → byte-compare (sha256 + length) → DELETE → confirm-gone** of one throwaway object at
+   `<prefix>/.s3check/<pod>.probe` — a genuine write, not a HEAD. A HEAD passes with a read-only token
+   and with R2 rejecting every write;
+4. whether the **R2 checksum trap** ([§6](#6-r2-specific-gotchas)) is present, probed on purpose by
+   repeating the PUT with checksums *enabled*, and named explicitly either way.
+
+It **fails closed**: a pod that cannot be created, an empty log, a missing verdict line, a short
+assertion count, an OK-line count that disagrees with the claimed count, or any single failed
+assertion all exit non-zero. There is no "reachability unknown, carrying on". Each failure names its
+own cause and fix — wrong key, unscoped token, missing bucket, no pod egress, checksum rejection.
+
+It writes and deletes exactly one object per bucket and leaves no pod behind. It applies nothing and
+does not enable `velero`.
+
+---
+
+## 5. The verification sequence — and it ends in a RESTORE
+
+**A backup that has never been restored is not a backup.** R-STATE-03 is PROVEN against MinIO; that
+proof does not transfer to a different object store with a different S3 implementation.
+
+1. **`s3check` on dev, then on prod.** Both must pass. (§4)
+2. Merge the endpoint + the four re-encrypted Secrets. Let Flux reconcile.
+3. **Resume `velero`** — `bootstrap/standup.sh … resume` does it once the endpoint is no longer a
+   placeholder, or `flux resume kustomization velero`.
+4. `velero backup-location get` → `default` **Available**. *Available only means the BSL could be
+   read;* it is not a backup.
+5. `velero backup create --from-schedule daily` → wait for **Completed**, then
+   `velero backup logs <name>`. Watch specifically for `PodVolumeBackup` failures: the PVC-data path
+   is kopia, a **different S3 client** from the one the BSL uses (see §7), so metadata can succeed
+   while volume data fails.
+6. **Restore.** `velero restore create --from-backup <name> --include-namespaces <scratch-ns>` into a
+   *scratch* namespace, and verify the PVC contents actually arrived — not just that the Restore says
+   Completed. This is the R-STATE-03 re-proof.
+7. **CNPG PITR.** Confirm WAL is arriving:
+   `kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.conditions}'` (look for
+   `ContinuousArchiving`), and that objects exist under `s3://cnpg/<env>/keycloak-db/`. Then do the
+   restore that matters: a `Cluster` with `bootstrap.recovery` from the barman object store into a
+   scratch namespace, and check the recovered database actually contains data. The existing
+   `conformance/tests/l2-behavioral/keycloak-db-dr` and `l3-fault-injection/dr-restore` tests are the
+   harness for this; they must be re-run **against R2**, and their previous PROVEN state records MinIO,
+   not R2.
+8. Only after 6 and 7 pass may this migration be described as done, and the ledger updated. Until
+   then the honest status is "R2 accepts objects; recovery from R2 is untested".
+
+---
+
+## 6. R2-specific gotchas
+
+**`region: auto`.** R2 accepts nothing else, and S3 clients require *some* region. Authored as
+`S3_BACKUP_REGION: "auto"`. This repo already learned it — `infra/tofu/backend.tf` carries
+`region = "auto"  # R2 ignores region, but the S3 backend requires a value.`
+
+**`checksumAlgorithm: ""` — do not delete this line.** `velero-plugin-for-aws`'s own
+`backupstoragelocation.md` for **v1.14.2** (the pinned version) documents:
+
+> `checksumAlgorithm` — The checksum algorithm to use for uploading objects to S3. Supported values
+> "CRC32", "CRC32C", "SHA1", "SHA256". If the value is set as empty string "", no checksum will be
+> calculated and attached to the request headers. **Optional (defaults to "CRC32")**
+
+So *omitting* the key means CRC32 **is** sent: `x-amz-sdk-checksum-algorithm` +
+`x-amz-checksum-crc32`. This repo has already measured that R2 rejects AWS checksums — that is exactly
+why `infra/tofu/backend.tf` needs `skip_s3_checksum = true  # R2 rejects AWS trailing (streaming)
+checksums.` for the state bucket. The plugin's own compatibility table lists `checksumAlgorithm=""` as
+the remedy for the same failure on Ceph S3, Backblaze B2 and IBM COS. If backups start failing with
+**HTTP 400 `InvalidArgument`** after a plugin bump, look here first. `s3check` probes for this on
+purpose and names it.
+
+**Path-style.** `s3ForcePathStyle: "true"` is authored and correct for R2 as well as MinIO. Never put
+the bucket in the hostname.
+
+**No bucket auto-creation.** Velero does not create buckets and the token deliberately cannot. A
+missing bucket is a `NoSuchBucket` at runtime; `s3check` catches it first.
+
+**Token scope is per-bucket.** A token scoped only to `velero` authenticates fine and then 403s on
+`cnpg` — the two consumers fail independently and quietly. `s3check` proves both.
+
+**Both clusters share one endpoint.** Isolation is by path only (`VELERO_BACKUP_PREFIX`,
+`CNPG_BACKUP_PATH`), because R2 is one environment-independent URL. Those two knobs are load-bearing:
+without them prod would adopt and TTL-delete dev's Velero backups, and both clusters' identically-named
+CNPG Clusters would interleave their WAL into one Barman stream, destroying PITR for both.
+
+---
+
+## 7. What is proven, and what is not
+
+**Proven (2026-09-05, WS-BACKUP, against the live dev cluster and its MinIO at
+`http://192.168.1.9:9000`):**
+
+- the rewritten `s3check` itself: 7 assertions per credential path, both credential shapes, from
+  in-cluster pods — credential load, bucket list, PUT/GET/byte-compare/DELETE/confirm-gone, no object
+  and no pod left behind;
+- its failure paths: the `TBD` placeholder is refused; an unreachable endpoint (`:9099`) produces a
+  named no-egress failure and exit 1, not a pass;
+- MinIO accepts a checksum-bearing PUT, i.e. the trap is a *non-*MinIO problem, and
+  `checksumAlgorithm: ""` is harmless there.
+
+**Established from the pinned versions (documentation and image inspection, not measured against R2):**
+
+- `velero-plugin-for-aws` **v1.14.2** pairs with Velero **v1.18.x** (upstream compatibility table) and
+  defaults `checksumAlgorithm` to `CRC32` — so the authored `""` is load-bearing, not decoration.
+- That plugin's "Non-AWS S3 compatible provider known issues with plugin v1.14.x" table **does not
+  mention Cloudflare R2 at all** — neither as working nor as broken. There is no upstream statement to
+  lean on in either direction.
+- **The PVC-data path is a different S3 client.** `checksumAlgorithm` is a BackupStorageLocation
+  *object-store plugin* setting. Velero's File System Backup uploads through the unified repository's
+  kopia S3 backend (`pkg/repository/udmrepo/kopialib/backend/s3.go` in Velero v1.18.0), which
+  references only endpoint/region-style options and has **no checksum knob at all**. So R2
+  compatibility for backup *metadata* and for *volume data* are two separate questions, and only step
+  5 of §5 answers the second.
+- **CNPG/barman is likely clear of the checksum trap, by version.** Inspected inside the live
+  `ghcr.io/cloudnative-pg/postgresql:16.4` image on dev: `barman-cloud-wal-archive 3.11.1`,
+  `botocore 1.35.60`, `boto3 1.35.60`. That botocore predates the 1.36 line in which botocore began
+  attaching CRC32 request checksums by default (the change that introduced
+  `AWS_REQUEST_CHECKSUM_CALCULATION`), so this image should not send them.
+  **Forward risk:** barman-cloud exposes **no** `checksumAlgorithm` equivalent. If the CNPG image is
+  ever bumped to one carrying botocore ≥ 1.36, WAL archiving to R2 could start failing with 400
+  `InvalidArgument` and there is no config knob to turn it off — only `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`
+  in the Pod environment, which CNPG does not currently set. Treat a CNPG image bump as a change that
+  must be re-verified against R2.
+
+**Not proven, at all:**
+
+- that Velero can write a usable backup to R2 (metadata or kopia volume data);
+- that CNPG/barman can archive WAL to R2 and that a `bootstrap.recovery` from R2 succeeds;
+- that a restore from R2 yields correct data. R-STATE-03's PROVEN state is **MinIO-only**.
+
+---
+
+## 8. Going back to MinIO
+
+The MinIO copy is **not deleted** and stays restorable. To read pre-migration backups:
+
+1. `clusters/base/cluster-vars.yaml` → `S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"`
+2. `VELERO_BACKUP_PREFIX` → `""` (the MinIO backups are at the **bucket root**, not under `dev/`) and
+   `CNPG_BACKUP_PATH` → `s3://cnpg` (Barman objects are at `s3://cnpg/<clusterName>/`).
+3. Restore the MinIO credential into the four Secrets. `platform/backup/minio-credentials.sops.yaml`
+   still holds it: `sops -d --extract '["stringData"]["ACCESS_KEY_ID"]'` it and feed the two values to
+   `platform/backup/set-r2-credentials.sh` (the script cares about the *shape*, not the provider).
+4. `S3_BACKUP_REGION` may stay `auto` — MinIO ignores the region.
+5. **Dev only.** Prod cannot reach that MinIO at all (§Why R2); pointing prod back is not an option.
+
+Reverting for a restore and then returning to R2 costs another PITR reset. Prefer restoring from MinIO
+into a *scratch* namespace on dev with a second, temporary BackupStorageLocation over flipping the
+whole cluster's endpoint back and forth.
+
+---
+
+## 9. Merging this resets dev's PITR window
+
+Say it out loud before merging, because it is not recoverable by retrying:
+
+- Dev's R2 prefix starts **empty**. Velero writes a **fresh kopia repository** — the first backup is a
+  full upload of every included PVC, and no earlier restore point exists in R2.
+- Every CNPG Cluster takes **new base backups** (`ScheduledBackup … immediate: true`), and the 30-day
+  Barman retention window (`retentionPolicy: "30d"`) rebuilds **from zero**. For roughly a month, dev's
+  point-in-time recovery reaches back only as far as the merge.
+- The MinIO copy still holds the old window (§8), but reaching it means reverting the endpoint — so it
+  is a deliberate excursion, not a fallback that happens automatically.
+
+**Do not merge this while dev's proven MinIO restore is the only recovery path you are relying on.**
+The same applies to prod the day it first backs up anywhere.

infra/PHASE1-STATUS.md

@@ -84,6 +84,9 @@ toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image
 infra/devcontainer/dev.sh build                    # once
 infra/devcontainer/dev.sh run make -C ci gate      # exactly what CI runs
 infra/devcontainer/dev.sh                          # interactive shell at /work
+
+infra/devcontainer/dev.sh run kubectl get nodes                # DEV (the default)
+CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # PROD
 ```
 
 Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
@@ -95,10 +98,21 @@ which is what broke the old `dmacvicar/libvirt` provider in a container.
 file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
 check runs instead of silently skipping.
 
-> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
+> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
-> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
+> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
-> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
+> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
-> context cannot work there anyway (no browser). Not yet fixed.
+> **dev remains the default**, so every existing invocation means what it always meant. Inside the
+> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
+> **`k3s-prod`** (never `default`), regenerated each run by
+> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
+> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
+> with context, cluster and user all called `default`, and kubectl's merge keeps the first
+> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
+> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
+> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
+> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
+> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
+> above. Details: [`devcontainer/README.md`](devcontainer/README.md).
 
 ## Off-repo state you must not lose
 

infra/PROD-STANDUP.md

@@ -0,0 +1,519 @@
+# Prod cluster standup — the complete command sequence
+
+**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
+container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
+this file it is not part of standup.
+
+State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s
+v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready**
+(`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and
+**all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is
+confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not
+biting on the backports kernel).
+
+Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory
+upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** —
+though only after a leftover cordon was cleared; read
+[§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing)
+before doing node work, that cordon silently prevents Longhorn from rebuilding.
+
+What remains is in [§Blockers](#blockers), and every item there needs an operator decision:
+no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found
+2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod.
+
+> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
+> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
+> therefore **requires** `--expect-server` and compares it against the live API server before running
+> anything. There is no override flag. Do not add one.
+
+## Constants
+
+| | |
+|---|---|
+| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
+| Gateway / ingress VIP | `192.168.2.20` |
+| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
+| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
+| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
+| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |
+
+Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
+**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).
+
+---
+
+## Phase A — bare metal to a joined cluster (Ansible)
+
+Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
+network (§4), and SSH + the vaulted `ansible_become_password` in
+`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
+file. See `infra/PROD-PROVISIONING.md`.
+
+**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
+publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
+validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
+from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
+```
+
+Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
+required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/site.yml \
+  -i /work/infra/ansible/inventory/hosts.ini \
+  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
+```
+
+kube-vip is a separate play (not imported by `site.yml`):
+
+```bash
+timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
+```
+
+Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).
+
+> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
+> target var so they stay independently pointable while a bring-up run is scoped elsewhere
+> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
+> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
+> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
+> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
+> caught only by `--list-hosts`. **Always dry-run the target first:**
+>
+> ```bash
+> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+>   ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
+>   -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
+> ```
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
+```
+
+## Phase B — Flux bootstrap
+
+`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
+documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
+no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
+and refuses to continue rather than silently skipping decryption
+(`bootstrap/sops/age-key-secret.md`).
+
+## Phase C — bring the reconciliation graph up
+
+One script, six steps, each independently runnable and idempotent. **Run them in this order.**
+
+```bash
+K=infra/ansible/.cache/prod.kubeconfig
+S=https://192.168.2.9:6443
+```
+
+### C1 — preflight (read-only, changes nothing)
+
+Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
+closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).
+
+```bash
+timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
+```
+
+### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)
+
+```bash
+timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
+```
+
+Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization
+before applying any of them, so one unknown `kind` aborts the entire tier.**
+
+1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets
+   `crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them
+   server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).
+
+2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and*
+   `platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does
+   not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind →
+   the CR is never applied → the CRD is never created → `platform-config` can never converge, and
+   every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been
+   there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator`
+   Available for 12h with **no CDI CR** and no `storageprofiles` CRD.
+
+   The step breaks it by applying **only the CR** — the minimum — and leaving everything else to
+   Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
+   The CRD appeared in 10s.
+
+   **REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier,
+   `./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is
+   **revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy*
+   — which is precisely the event that creates the `StorageProfile` CRD — before the profiles are
+   applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster
+   already stuck in the broken state, and because it is idempotent.
+
+   **The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a
+   *controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers
+   exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and
+   `dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of
+   `platform-config` found `StorageProfile` is the only second-order kind there.
+
+   `apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic:
+   `apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment
+   previously claimed `platform-config` guaranteed the profiles, which the move would have silently
+   falsified.
+
+#### Merge precondition and the one-time transient
+
+`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both
+StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi`
+re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).
+
+**Measured on dev, so the exposure is known rather than assumed** — comparing our override against
+what CDI infers unaided for the same Longhorn provisioner:
+
+| profile | `spec` (ours) | CDI-derived `status` |
+|---|---|---|
+| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` |
+| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` |
+
+So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes`
+is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window*
+would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that
+(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand
+during the window.**
+
+- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If
+      it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in
+      exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with
+      `platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in
+      a second commit.
+- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries
+      `claimPropertySets` RWX/Block, and both are labelled
+      `kustomize.toolkit.fluxcd.io/name=platform-config-cdi`.
+
+### C3 — unwedge stuck Helm releases
+
+A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
+`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
+**Dry run by default** — it prints what it would do and changes nothing:
+
+```bash
+timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
+```
+
+To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
+CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
+  --confirm-destructive unwedge
+```
+
+### C4 — external-dns safe upgrade
+
+**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
+resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
+window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
+`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
+them**.
+
+So this step **proves the outcome before causing it**: it renders the chart with the *live*
+HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
+deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
+Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
+and diffs the zone.
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
+```
+
+Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
+`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
+regression — stop and investigate.
+
+### C5 — resume the tiers, in dependency order
+
+`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
+apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
+partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
+a placeholder (see §Blockers). Long-running; expect it to background.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
+```
+
+### C6 — verify
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
+```
+
+### Everything at once
+
+Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
+destructive step by design:
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
+```
+
+---
+
+## VERIFIED WORKING — 2026-09-05, standup complete
+
+| | |
+|---|---|
+| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
+| Pods | **113 total, 0 not Running/Completed** |
+| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
+| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
+| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
+| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |
+
+End-to-end HTTPS **through the workstation's own resolver — no `--resolve` override, no `-k`** (after
+installing the dnsmasq profile below). Every request landed on `192.168.2.20`:
+
+```
+prod.home.keithjackson.dev  200   headlamp.  200   keycloak.  302 (login)   nextcloud. 302
+gitea.       200            semaphore. 200        grafana.   302 (SSO)      pihole.    302
+```
+
+The served certificate is prod's own, confirmed by SNI — **not** dev's wildcard, which is what the
+resolver bug previously made it look like:
+
+```
+subject = CN=*.prod.home.keithjackson.dev
+issuer  = C=US, O=Let's Encrypt, CN=YR1          (LE production)
+SAN     = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
+valid   = 2026-09-05 .. 2026-12-04
+```
+
+Dev, checked in the same pass, still serves `CN=*.home.keithjackson.dev` — **unchanged**.
+
+> Ignore `curl: (91) No OCSP response received` if you pass `--cert-status`: Let's Encrypt retired
+> OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain
+> requests above all validated normally.
+
+### Workstation DNS — install the prod dnsmasq profile
+
+**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
+notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.
+
+`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
+
+```
+address=/.home.keithjackson.dev/192.168.2.50
+```
+
+dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
+the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
+correctly returns prod's `192.168.2.20`.
+
+**Fix — one command, already authored in this repo:**
+
+```bash
+sudo infra/workstation/dnsmasq/install.sh
+```
+
+It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
+`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
+Verified against a throwaway dnsmasq loaded with both profiles:
+
+| query | answer |
+|---|---|
+| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
+| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
+| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
+| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
+
+> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
+> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
+> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
+> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
+
+**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
+convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
+`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
+a broken prod certificate. It is the wrong server answering.
+
+```bash
+curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
+  https://gitea.prod.home.keithjackson.dev/
+```
+
+## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing
+
+Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade
+on prod-2) produced one finding worth more than either incident.
+
+### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing
+
+`kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods,
+DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.**
+Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica
+scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with
+prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod
+was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged
+17/17 healthy in ~6 minutes.
+
+**So: after any drain — successful or not — `kubectl uncordon <node>` is not tidying up, it is what
+re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\
+U:.spec.unschedulable` before believing a green pod list.
+
+### `drain` cannot complete on this cluster, and the two usual flags do not fix it
+
+`--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs
+indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and
+`node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an
+instance-manager be evicted while it runs volume engines. That is data protection, not a bug.
+
+**Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no
+fourth node to move a replica to, so eviction can never complete. This is structural, not a setting.
+
+Options, in order of preference:
+
+1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on
+   return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10
+   minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel
+   work.
+2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it —
+   that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with:
+   `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`.
+3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades
+   the protection for exactly the data risk the default exists to prevent.
+
+### Check the kube-vip leader before rebooting a server
+
+`kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so
+`--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the
+lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault.
+Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`.
+
+### Post-maintenance checklist
+
+- [ ] `kubectl uncordon <node>` — **and confirm** `.spec.unschedulable` is clear on all three.
+- [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**.
+      Rebuilds are serial, so allow minutes, and watch rather than assume.
+- [ ] Replica symmetry: each node runs the same count
+      (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`).
+- [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and
+      a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`.
+- [ ] Pods: none `Running`-but-not-`Ready`.
+
+> **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread
+> was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA
+> apps here a spread constraint would change nothing (one replica lives on one node either way). It
+> self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing
+> **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and
+> Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the
+> operator owns — see §Blockers.
+
+## Blockers
+
+### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
+
+**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
+`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.
+
+**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
+from the workstation, which proves nothing about pod egress:
+
+```
+curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
+```
+
+**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
+**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
+**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
+`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
+pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
+a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
+
+Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
+
+> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
+> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
+> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
+> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
+
+Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
+token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
+**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
+coverage.**
+
+#### Sharing one MinIO is only safe because of two new knobs
+
+Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
+MinIO without these would have been the external-dns incident again, in a worse place:
+
+| | before | after |
+|---|---|---|
+| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
+| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |
+
+- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
+  when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
+- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
+  so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
+  WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
+  **proven** (R-STATE-03).
+
+**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
+backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
+backups its proven restore depends on** — the same reasoning that keeps dev's
+`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.
+
+> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
+> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
+> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
+> commits. No urgency: velero cannot be enabled until the routing above is fixed.
+
+### 2. Prod is on `prod.home.keithjackson.dev`, not the apex
+
+The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
+Prod cannot take the apex until dev vacates it, because both clusters would request a
+`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
+records, so neither reliably solves DNS-01.
+
+**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
+nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
+the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
+once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
+
+> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
+> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
+> production environment**, which the operator requires to stay live until the switch is deliberately
+> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
+> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
+> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
+> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
+>
+> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
+> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
+
+---
+
+## Not covered here
+
+- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
+  redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
+- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
+  `clusters/prod/kustomization.yaml` owns that procedure.
+- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
+  §9's "suite passes green against the prod cluster" can be claimed.

infra/ansible/README.md

@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 
+> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
+> ```
+> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
+> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
+> ```
+> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
+> until `-e memattr_survey_reboots_authorised=true` is passed. See
+> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
+
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -70,11 +79,30 @@ Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
 afterward. The parity-critical values **must match** between this role's
 `templates/cilium-values.yaml.j2` and that HelmRelease:
 - `kubeProxyReplacement: true`
-- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
+- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
-- `k8sServicePort: 6443`
+  (`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
+- `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)
 
-The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
+### Two API addresses, split 2026-09-06 — do not re-merge them
-to the same value this role uses (surfaced by `tofu output primary_node_ip`).
+`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
+prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
+HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
+service or policy update ever lands, which is a silent failure, not an outage.
+
+| variable | value | consumed by | why it must be that address |
+|---|---|---|---|
+| `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
+| `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |
+
+Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
+and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
+last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
+so no cert reissue and no k3s restart are involved.
+
+The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
+set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
+`k3s_apiserver_endpoint` for the SAME environment. Asserted by
+`conformance/tests/l0-static/ha-control-plane-as-code`.
 
 ## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
 `roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
@@ -109,6 +137,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
+`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
+for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
+role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
+can be mistyped; inventory membership cannot.
 
 ```bash
 # dev — completely unchanged, no extra vars
@@ -135,7 +167,9 @@ ansible-playbook playbooks/site.yml \
 What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
 `k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
 target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
-node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
+node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
+`kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
+Cilium dials — and
 `cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
 `.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
 

infra/ansible/collections/requirements.yml

@@ -2,7 +2,8 @@
 # Ansible Galaxy collections required by infra/ansible. Install with:
 #   ansible-galaxy collection install -r collections/requirements.yml
 collections:
-  # node_baseline role: community.general.modprobe, community.general.filesystem
+  # node_baseline role: community.general.modprobe, community.general.filesystem,
+  #                      community.general.timezone
   - name: community.general
   # node_baseline role: ansible.posix.mount
   - name: ansible.posix

infra/ansible/inventory/group_vars/all.yml

@@ -27,19 +27,85 @@ k3s_servers_group: k3s_servers
 # a kubeconfig is a credential.)
 cluster_kubeconfig_name: dev.kubeconfig
 
-# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
+# --- TWO API ADDRESSES, AND THEY ARE NOT INTERCHANGEABLE (split 2026-09-06) ----------
+# This used to be ONE variable, k3s_api_host, consumed by BOTH the cold-bootstrap join and
+# Cilium's k8sServiceHost. That conflation is what pointed every cilium-agent on the LIVE
+# prod cluster at 192.168.2.10 — k3s-prod-1's own address — on a cluster whose entire
+# purpose is HA: verified 2026-09-06, all three agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held 192.168.2.9. Lose that one node
+# and no service/policy update reaches the CNI until it returns (the existing datapath
+# survives on Cilium's cache, so it fails SILENTLY). The two consumers have genuinely
+# different requirements, so they now have two variables.
+#
+#   k3s_api_host            THE FOUNDER'S OWN ADDRESS. Bootstrap-time only.
+#   k3s_apiserver_endpoint  THE HA ENDPOINT (the kube-vip VIP). Steady-state.
+#
+# DO NOT "simplify" these back into one. See each block below for why each half must be
+# the address it is.
+
+# THE FOUNDER'S OWN NODE IP — a BOOTSTRAP-TIME address, deliberately NOT the VIP.
 # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
-# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
+# IP — so we read the real 192.168.1.x from k3s_node_ip instead.
-# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
+#
-# HelmRelease stay in PARITY (docs/02 contract).
+# EXACTLY TWO CONSUMERS, and both of them need the founder specifically:
+#   1. roles/k3s_server — the secondary `--server https://<host>:<port>` JOIN URL. On a COLD
+#      build the VIP DOES NOT EXIST YET at join time: kube-vip is a k3s auto-deploy manifest
+#      that only lands once a cluster exists, so a join aimed at the VIP is a join aimed at
+#      an unanswered address and the cold build is unjoinable. The founder is up (it just ran
+#      --cluster-init), which is the whole reason this variable is the founder.
+#   2. roles/k3s_server — one entry of k3s_tls_sans, so the founder's own address is a SAN
+#      alongside the VIP and every other server's IP.
+#
+# ANYTHING THAT RUNS AGAINST A CLUSTER THAT ALREADY EXISTS MUST NOT USE THIS. It is a
+# single point of failure by construction. Use k3s_apiserver_endpoint below.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
+
+# Shared by both halves: k3s serves the API on 6443 in every environment, so the port is
+# not a per-environment knob and is not duplicated per address.
 k3s_api_port: 6443
 
+# THE HA kube-apiserver ENDPOINT — a VIRTUAL address that reaches the cluster wherever the
+# control plane currently is, not a machine. OPERATOR DECISION 2026-09-06: in-cluster
+# components dial the VIP, as kube-vip was always intended to provide.
+#
+# DERIVED from kube_vip_address (below), never restated: an environment overrides the VIP in
+# ONE place (group_vars/k3s_prod/vars.yml sets kube_vip_address: 192.168.2.9) and this
+# follows automatically, so prod needs no override of its own and a fourth environment
+# cannot set one of the two and forget the other.
+#
+# NAMED FOR THE ROLE, NOT THE MECHANISM. Consumers want "the address that reaches the
+# apiserver, HA"; that it is currently floated by kube-vip is an implementation detail. If
+# the mechanism is ever something else (an external LB, a DNS name), this one line moves and
+# no consumer changes.
+#
+# CONSUMERS:
+#   - roles/cilium -> cilium-values.yaml.j2 -> k8sServiceHost.  [parity] with the Flux side:
+#     clusters/base/cluster-vars.yaml K3S_API_HOST (and each overlay's patch) MUST equal
+#     THIS value for that environment, or Flux and the bootstrap fight over the CNI's
+#     control-plane path. See platform/cilium/helmrelease.yaml's PARITY CONTRACT header.
+#
+# TLS IS ALREADY COVERED — verified, not assumed: k3s_tls_sans (roles/k3s_server/defaults)
+# has always included kube_vip_address, and on 2026-09-06 the prod apiserver was reached
+# through https://192.168.2.9:6443 with FULL CA verification and tls-server-name=192.168.2.9
+# (infra/ansible/.cache/prod.kubeconfig, no insecure-skip-tls-verify). So the serving cert
+# needs no change and no k3s restart for this switch.
+#
+# ORDERING CONSEQUENCE, and it is why playbooks/site.yml changed too: Cilium now needs the
+# VIP to be UP before it installs. site.yml therefore imports kube-vip.yml BETWEEN k3s.yml
+# and cilium.yml, and roles/kube_vip waits for the VIP to answer before returning. kube-vip
+# can precede the CNI because its DaemonSet is hostNetwork with
+# KUBERNETES_SERVICE_HOST=127.0.0.1 — it needs neither a CNI nor kube-proxy.
+k3s_apiserver_endpoint: "{{ kube_vip_address }}"
+
 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
 # endpoint for the kube-apiserver, fronted by an operator pfSense port-forward at
-# 192.168.1.49:6443. Shared here so BOTH kube_vip AND the k3s_server tls-san list
+# 192.168.1.49:6443. Shared here so THREE things agree on the one address:
-# agree on the address (the serving cert must carry this IP or TLS via the VIP
+#   - roles/kube_vip          — what to ARP and lease-elect for;
-# fails to validate).
+#   - roles/k3s_server        — the tls-san entry (the serving cert must carry this IP or
+#                               TLS via the VIP fails to validate);
+#   - k3s_apiserver_endpoint  — above; what in-cluster components actually dial.
+# THIS is the one line an environment overrides (prod: 192.168.2.9, in
+# group_vars/k3s_prod/vars.yml); all three follow.
 kube_vip_address: "192.168.1.49"
 
 # Controller (dev container) reaches the kube-API DIRECTLY via the pfSense NAT

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -51,12 +51,18 @@ k3s_servers_group: k3s_prod
 # than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
 cluster_kubeconfig_name: prod.kubeconfig
 
-# --- k3s API host — safe to set now, the primary's real IP is already known -------
+# --- k3s API host — THE FOUNDER'S ADDRESS, bootstrap-time only --------------------
 # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
 # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
 # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
 # address (see the KEY DIFFERENCE note there) — so point straight at it instead of
 # needing a k3s_node_ip split.
+#
+# WHAT THIS IS *NOT*, since 2026-09-06: it is not the endpoint anything steady-state
+# dials. That is k3s_apiserver_endpoint (group_vars/all.yml), which derives from
+# kube_vip_address below and therefore needs NO override here. k3s_api_host is consumed
+# only by the secondary `--server` cold-join URL and by one tls-san entry. Read the
+# TWO API ADDRESSES block at the top of group_vars/all.yml before touching either.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
 # by template reference, so it needs no separate override here — it picks up the real
@@ -75,8 +81,15 @@ k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # round (control-plane .49, ingress pool .53-.60).
 #
 # This is the address kube-vip floats for the API and that k3s tls-san's onto its
-# serving cert; secondaries use it as their --server join target. Dev's equivalent is
+# serving cert. Dev's equivalent is kube_vip_address: 192.168.1.49 in group_vars/all.yml.
-# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+#
+# CORRECTION 2026-09-06: an earlier version of this comment claimed "secondaries use it as
+# their --server join target". They do not, and they must not — on a COLD build the VIP is
+# not up when the secondaries join (kube-vip is a k3s auto-deploy manifest and needs a
+# cluster first), so the join uses k3s_api_host, the founder. What DOES use this address is
+# k3s_apiserver_endpoint (group_vars/all.yml, derived from this line), i.e. Cilium's
+# k8sServiceHost and any other steady-state client. Overriding kube_vip_address here moves
+# BOTH the VIP and prod's Cilium endpoint together, which is the point of deriving it.
 #
 # HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
 # VIP, not the control-plane VIP. Corrected. Keep the two distinct.
@@ -87,24 +100,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
+# backports kernel as a side effect of sharing the role), and this line switches it on for
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
-# updates these two lines. All three nodes must end up with an identical layout
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# in §3.
+#
+# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
+# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
+# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
+# back to false, roll back per the note the role writes to
+# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
+# roles/node_baseline.
+#
+# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
+# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
+node_baseline_backports_kernel: true
+
+# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
+# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
+#
+#   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
+#                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
+#                     (LABEL=k3s-data), then split between TWO tenants:
+#                       /data/longhorn   Longhorn's dataPath
+#                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
+#   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
+#
+# ================================ THE 2026-09-04 REVISION ================================
+# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
+# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
+# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
+# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
+# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
+# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
+# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
+# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
+# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
+# decided then — it is cheap now and very expensive after bring-up.
+#
+# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
+# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
+# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
+# MEASURED, not assumed — exit condition in §2.
+# =========================================================================================
+#
+# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
+# not as a device path and a blank. That distinction is the point of this block:
+#
+# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
+#    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
+#    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
+#    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
+#    freezes a fact about today's hardware into inventory, where it goes stale silently the
+#    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
+#    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
+#    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
+#    role discovers the device on the node, then proves it is safe to claim (a partition or
+#    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
+#    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
+#
+# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
+#    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
+#    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
+#    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
+#    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
+#    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
+#    on every run, that the filesystem backing the etcd data dir really is /data. If the
+#    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
+#    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
+#    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
+#    why the two are distinct enum values rather than one with a comment.
+#
+# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
+#    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
+#    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
+#    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
+#    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
+#    Longhorn volumes are untouched.
+node_baseline_longhorn_source: claim-mountpoint
+node_baseline_longhorn_claim_mountpoint: /srv
+node_baseline_etcd_source: shared-data-partition
+
+# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
+# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
+# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
+# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
+# as the destination would make "claimed" and "not claimed" indistinguishable.
+# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
+# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
+# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
+# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
+# `k3s-etcd` label convention already in this repo.
+node_baseline_data_mount: /data
+node_baseline_data_fs_label: k3s-data
+# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
+# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
+# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
+# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
+node_baseline_longhorn_data_path: /data/longhorn
+
+# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
+# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
+# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
+# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
+# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
+# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
+# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
+# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
+# the partition is empty (nothing orphaned) and is the device the retired label names.
+# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
+# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
+# then skip on the retired-label lookup and nothing else depends on them.
+node_baseline_data_retired_fs_label: longhorn
+node_baseline_data_retired_mount: /var/lib/longhorn
+
+# --- k3s --data-dir (§2) -----------------------------------------------------------
+# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
+# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
+# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
+# which defers to this variable) creates the directory 0700 on the data partition and derives
+# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
+# Ansible side cannot disagree with itself.
+#
+# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
+# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
+# dir would orphan that mount under a running cluster.
+k3s_data_dir: /data/k3s
+
+# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
+# source uses a device path any more, but leaving these UNSET would let prod silently
+# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
+# metal) the moment anyone flipped a source back to `device`. These values can never parse
+# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
+# before mkfs is ever reached.
+node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
+node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -158,15 +302,58 @@ node_baseline_dns_servers:
   #   LAN clients       -> PiHole. Not a cluster concern.
   #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
   #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
   #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  #
+  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # external-dns has actually published the record. Until then the issuer does not resolve
+  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
+  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
+
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
+# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
+# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
+# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
+# postBuild.substituteFrom never sees it. Full rationale in
+# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
+#
+# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
+# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
+# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
+# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
+# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
+k3s_base_domain: "prod.home.keithjackson.dev"
+
+# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
+# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
+# and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
+# Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
+# clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
+# exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
+# infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
+#
+# Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
+# FOURTH prod node inherits it from group membership with no per-host or per-run flag.
+#
+# WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
+# flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
+#   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
+#     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
+#   - the GitRepository url there is the TEMPORARY workstation Gitea
+#     (http://192.168.2.132:3000/...) and that container is actually running and reachable
+#     from the prod nodes — it is outside every cluster and can simply be powered off;
+#   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
+flux_bootstrap_env_path: "clusters/prod"

infra/ansible/inventory/hosts.ini

@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
-# later member-add dance.
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
+# single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 # node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
 # partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
 # nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
-# node 3 remains commented until it is racked.
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
+# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
+# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
+# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
+# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
+# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
+# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
+# three-way, not a node-1 outlier.
+# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
+# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
 k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).

infra/ansible/playbooks/kube-vip.yml

@@ -3,6 +3,13 @@
 # Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
 # Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
 #
+# PART OF THE ONE-COMMAND BRING-UP SINCE 2026-09-06, not a separate operator step:
+# playbooks/site.yml imports this play BETWEEN k3s.yml and cilium.yml, because Cilium's
+# k8sServiceHost is now this VIP (k3s_apiserver_endpoint, group_vars/all.yml) and the CNI
+# cannot be installed against an endpoint nothing announces yet. The role's final task waits
+# for the VIP to answer on 6443, so a failure to float it stops the run here with a named
+# cause instead of surfacing later as unreachable cilium-agents.
+#
 # TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
 #   ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
 # (the VIP itself comes from kube_vip_address, already scoped per environment in

infra/ansible/playbooks/memattr-survey.yml

@@ -0,0 +1,137 @@
+---
+# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
+# R-ENV-03, R-TEST-02).
+#
+# WHAT QUESTION THIS ANSWERS
+#   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
+#   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
+#   Linux 6.12 reserves everything from the table address to the top of RAM without
+#   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
+#   playbook measures them:
+#
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
+#        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
+#        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
+#        range, the causal chain closes with nothing touched.
+#     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+#   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
+#   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
+#   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
+#   boots per arm; detecting a mere odds shift needs ~50.
+#
+# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
+# package installs, no hardware moves, no node_baseline run. Its job is to establish real
+# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
+# a Fast Boot flip, a 6.13 kernel) compares against it.
+#
+# HOW TO RUN IT
+#   Dry run first — every stage except the reboots, so the pipeline is proven before 45
+#   minutes of wall clock are committed:
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml'
+#
+#   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml \
+#           -e memattr_survey_reboots_authorised=true'
+#
+#   Fewer rounds, or one node:
+#     -e memattr_survey_rounds=5        --limit k3s-prod-2
+#
+#   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
+#   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
+#     ansible-playbook playbooks/memattr-survey.yml --tags analyse
+#
+# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
+# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
+# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
+# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
+# engineering against.
+#
+# HOW TO READ THE OUTPUT
+#   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
+#     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
+#     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
+#                          at the end — the AUTHORITATIVE record
+#     summary.md           the analysis, also printed at the end of the run
+#   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
+#   with NO memattr log line is a healthy third state and not a missing measurement.
+
+# =====================================================================================
+# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
+# before a single reboot. That is the right asymmetry: the cost of a false stop is a
+# re-run, the cost of a false start is rebooting a machine that was doing something.
+# =====================================================================================
+- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  any_errors_fatal: true
+  tags: [survey]
+  tasks:
+    - name: Preflight and baseline probe
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: preflight
+
+# =====================================================================================
+# PLAY 2 — THE SURVEY.
+#
+# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
+# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
+# keep collecting.
+#
+# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
+# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
+# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
+# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
+# blocks belongs to exactly one host, so the other two sit idle through it and the survey
+# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
+# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
+# appeared in the output. `free` lets each host walk its own task list at its own pace,
+# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
+#
+# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
+# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
+# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
+# directory create on every host instead. Concurrency is also capped by `forks`
+# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
+# than letting a bigger fleet silently serialise again.
+# =====================================================================================
+- name: memattr survey — reboot rounds and journal harvest
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  strategy: free
+  tags: [survey]
+  tasks:
+    - name: Run the reboot rounds
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: rounds
+
+    - name: Harvest every retained boot from the persistent journal
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: harvest
+
+# =====================================================================================
+# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
+# even if every surveyed node died, because the artifacts are already on disk and the
+# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
+# vanished shows up as a short arm with a wider interval, not as a missing report.
+# =====================================================================================
+- name: memattr survey — analyse
+  hosts: localhost
+  gather_facts: false
+  become: false
+  tags: [analyse]
+  tasks:
+    - name: Produce the analysis
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: analyse

infra/ansible/playbooks/site.yml

@@ -1,7 +1,10 @@
 ---
 # Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
 # `tofu apply` has created and cloud-init has baselined the VMs):
-#   k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
+#   k3s HA install -> kube-vip control-plane VIP -> Cilium CNI -> verify readiness
+#   -> Flux + SOPS hand-off.
+# (kube-vip joined the chain 2026-09-06 — see the note at its import below; it is a
+# prerequisite of the CNI now, not a separate operator step.)
 #
 # TARGET (R-ENV-02): every imported play takes its host pattern from a var that
 # defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
@@ -20,6 +23,29 @@
 - name: Import - install k3s HA control plane
   ansible.builtin.import_playbook: k3s.yml
 
+# ADDED 2026-09-06, AND ITS POSITION IS LOAD-BEARING: kube-vip MUST come up between k3s and
+# Cilium. Cilium's k8sServiceHost is now the control-plane VIP (k3s_apiserver_endpoint =
+# kube_vip_address) rather than the founder's own node IP, so installing Cilium before the
+# VIP is announced would point every agent at an address nothing answers on — the exact
+# shape of the 2026-09-01 incident, where agents that could not reach the API server
+# deadlocked on a cold cluster.
+#
+# IT CANNOT COME EARLIER. kube-vip ships as a k3s auto-deploy manifest
+# (<data-dir>/server/manifests), so a cluster must exist before it can be applied — hence
+# after k3s.yml, not before.
+#
+# IT DOES NOT NEED THE CNI, which is what makes this ordering possible at all: the kube-vip
+# DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 (roles/kube_vip's template
+# explains why), so it reaches the node-local apiserver with no CNI and no kube-proxy, and
+# DaemonSet pods carry the default not-ready/network-unavailable tolerations that let them
+# schedule onto a still-NotReady node. roles/kube_vip blocks until the VIP actually answers
+# on 6443, so cilium.yml below cannot start against a VIP that is not up.
+#
+# Re-running this against the LIVE dev cluster is a no-op for this play: dev already runs
+# kube-vip with the same rendered manifest, and the wait finds the VIP already answering.
+- name: Import - float the control-plane HA VIP (kube-vip) BEFORE the CNI dials it
+  ansible.builtin.import_playbook: kube-vip.yml
+
 - name: Import - install Cilium CNI
   ansible.builtin.import_playbook: cilium.yml
 

infra/ansible/roles/cilium/defaults/main.yml

@@ -2,7 +2,17 @@
 # cilium role defaults (ADR-0002, docs/02 Cilium-ownership contract).
 # This role performs the INITIAL Cilium install; the Flux platform/cilium
 # HelmRelease adopts config afterward. The parity-critical values
-# (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host.
+# (kubeProxyReplacement, k8sServiceHost/Port) are templated from
+# k3s_apiserver_endpoint / k3s_api_port (group_vars/all.yml).
+#
+# k8sServiceHost IS THE HA VIP, NOT THE FOUNDER (changed 2026-09-06). It reads
+# k3s_apiserver_endpoint — which derives from kube_vip_address, so it is dev 192.168.1.49 /
+# prod 192.168.2.9 — and deliberately NOT k3s_api_host, the founder's own node IP. Those two
+# were one variable until 2026-09-06, and the conflation is what left all three prod
+# cilium-agents dialing k3s-prod-1 on an HA cluster. Do not re-merge them.
+#
+# THIS ROLE THEREFORE DEPENDS ON THE VIP EXISTING. playbooks/site.yml runs kube-vip.yml
+# before cilium.yml and roles/kube_vip blocks until the VIP answers on 6443.
 
 # Exact pin — MUST equal the platform/cilium HelmRelease chart.spec.version (1.20.1, an
 # exact pin, not a range). Renovate bumps both together.
@@ -15,9 +25,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
   - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev

infra/ansible/roles/cilium/tasks/main.yml

@@ -89,7 +89,7 @@
   run_once: true
   changed_when: false
 
-- name: Render the Cilium values file (parity-critical settings from k3s_api_host)
+- name: Render the Cilium values file (parity-critical settings from k3s_apiserver_endpoint)
   ansible.builtin.template:
     src: cilium-values.yaml.j2
     dest: "{{ cilium_values_path }}"

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -4,9 +4,24 @@
 ---
 # [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary node IP). The Flux HelmRelease's
+# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
-# k8sServiceHost "REPLACE-ME" must be set to this same value.
+# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
-k8sServiceHost: "{{ k3s_api_host }}"
+# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
+# losing that machine stops every service/policy update reaching the datapath, silently
+# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
+# was the live state of prod until 2026-09-06 — all three cilium-agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
+#
+# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
+# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
+# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
+# Flux and this bootstrap render different control-plane paths and fight over the CNI.
+#
+# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
+# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
+# answers. Rendering this against a VIP that nothing announces yet reproduces the
+# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
+k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
 k8sServicePort: {{ k3s_api_port }}
 
 # Ingress via Gateway API — no Traefik (docs/09, ADR-0002).

infra/ansible/roles/flux_bootstrap/defaults/main.yml

@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
 
 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
+
+# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
+# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
+# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
+# tasks/main.yml exports this value into that variable, so the role NEVER names an
+# environment and the task graph is identical for dev and prod.
+#
+# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
+# override, so a dev run resolves exactly the value it always had (clusters/dev) and
+# behaves byte-for-byte as before.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
+# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
+# playbook: group membership decides. Before this existed the role could bootstrap
+# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
+flux_bootstrap_env_path: "clusters/dev"

infra/ansible/roles/flux_bootstrap/tasks/main.yml

@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
+# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
+# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
+# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
+# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
 
 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
   ansible.builtin.stat:
@@ -29,11 +35,14 @@
   run_once: true
   changed_when: true
 
-- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
   ansible.builtin.command:
     cmd: "{{ flux_bootstrap_flux_cmd }}"
   environment:
     KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
+    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
+    # clusters/prod for [k3s_prod] hosts via group_vars.
+    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
   delegate_to: localhost
   become: false
   run_once: true

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 
+# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
+# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
+# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
+# image store (agent/containerd) — which is by far the largest tenant.
+#
+# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
+# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
+# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
+# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
+# running cluster's entire state directory out from under it and orphan that etcd mount.
+#
+# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
+# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
+# node inherits it from group membership rather than from anyone remembering. It is set
+# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
+# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
+# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
+# this same variable). One line moves both.
+#
+# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
+# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
+# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
+# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
+# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
+#
+# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
+# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
+# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
+# Restart handler for a value that is not changing. The install flags are the correct and
+# sufficient place, and prod has no k3s installed yet.
+k3s_data_dir: "/var/lib/rancher/k3s"
+
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
 
@@ -27,6 +59,17 @@ k3s_server_mode: "secondary"
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
 #
+# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
+# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
+# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
+# and every steady-state client dial). Both are already in this set, so repointing a
+# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
+# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
+# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
+# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
+# today only because it is also in the k3s_servers_group node-IP list below, and that
+# coincidence is not a contract.
+#
 # k3s_servers_group — NOT a literal group name — is what makes this role environment-
 # agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
 # group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
@@ -34,3 +77,33 @@ k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
       + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
+# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
+# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
+# what it already is and the Restart handler does NOT fire on a re-run.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
+# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
+# from group membership rather than from anyone remembering.
+k3s_base_domain: "home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -22,6 +22,7 @@
       --write-kubeconfig-mode=644
       {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
       --node-ip={{ k3s_node_ip }}
+      --data-dir={{ k3s_data_dir }}
 
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -74,13 +75,19 @@
       # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
       # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
       # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
       # prefixes namespace federated identities away from cert / service-account subjects.
       # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
       # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
       kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
         - "oidc-username-claim=email"
         - "oidc-username-prefix=oidc:"
@@ -97,15 +104,20 @@
     INSTALL_K3S_VERSION: "{{ k3s_version }}"
   when: k3s_server_mode == 'primary'
 
+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
   ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
     timeout: 180
   when: k3s_server_mode == 'primary'
 
 - name: Read the k3s node-token (primary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   register: k3s_token_raw
   when: k3s_server_mode == 'primary'
 
@@ -125,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:
@@ -141,6 +155,18 @@
          | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
   when: k3s_server_mode == 'secondary'
 
+# THE JOIN TARGET IS k3s_api_host — THE FOUNDER — AND MUST NOT BECOME THE VIP.
+# group_vars/all.yml splits the two API addresses (k3s_api_host = the founder's own node IP,
+# k3s_apiserver_endpoint = the kube-vip HA VIP) and this is the canonical founder-only
+# consumer. On a COLD build the VIP does not exist at this point in the sequence: kube-vip
+# is a k3s auto-deploy manifest, so it cannot be applied until a cluster exists, and
+# playbooks/site.yml only imports kube-vip.yml AFTER k3s.yml. A `--server` pointed at the
+# VIP would therefore dial an address nothing announces and the cold build would be
+# unjoinable — with no diagnostic beyond a connection timeout.
+# On a WARM single-node rejoin (`--limit k3s-dev-N`) the founder is up and answering too, so
+# the founder works in both cases and the VIP works in only one. Cilium's k8sServiceHost is
+# the opposite trade (steady state, must survive losing the founder) and uses the endpoint
+# variable instead.
 - name: Install k3s secondary (join as server)
   ansible.builtin.command:
     cmd: >-

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -24,6 +24,16 @@ kube_vip_port: 6443
 # only if detection is wrong.
 kube_vip_interface: ""
 
+# How long to wait for the VIP to actually ANSWER on kube_vip_port after the manifest is
+# written (seconds). This is a real gate, not cosmetics: since 2026-09-06 Cilium's
+# k8sServiceHost is this VIP (k3s_apiserver_endpoint), and playbooks/site.yml runs this role
+# immediately before cilium.yml — so if the VIP never comes up, the run must stop HERE with
+# a clear message rather than proceed to install a CNI pointed at an address nothing
+# answers. On a cold cluster the budget covers k3s's deploy controller applying the
+# manifest, the ghcr.io image pull and leader election; on an already-running cluster the
+# check returns immediately.
+kube_vip_wait_timeout: 300
+
 # Leader-election tuning (seconds) — a snappy failover without thrashing.
 kube_vip_lease_duration: 15
 kube_vip_renew_deadline: 10
@@ -31,4 +41,14 @@ kube_vip_retry_period: 2
 
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -1,17 +1,34 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
 
-- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
   ansible.builtin.shell: >-
     set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
   args:
     executable: /bin/bash
   register: kube_vip_iface_detect
   changed_when: false
+  # RUN THIS EVEN UNDER --check. It is `ip -o -4 addr show`: it reads and changes nothing.
+  # Without check_mode: false the shell module is SKIPPED in check mode, kube_vip_iface_detect
+  # has no stdout, and the assert two tasks down aborts the play with "Could not detect the
+  # interface ... set kube_vip_interface explicitly" — a misleading failure that says nothing
+  # about the node. That mattered less while this play was an out-of-band operator step; it
+  # matters now that playbooks/site.yml imports it, because `--check` on the bring-up before a
+  # cold prod build is exactly the rehearsal an operator should be able to run.
+  check_mode: false
 
 - name: Resolve the kube-vip VIP interface (override wins, else detected)
   ansible.builtin.set_fact:
@@ -25,7 +42,7 @@
       - kube_vip_vip_interface is defined
       - kube_vip_vip_interface | length > 0
     fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
       {{ inventory_hostname }} — set kube_vip_interface explicitly.
 
 - name: Ensure the k3s auto-deploy manifests directory exists
@@ -43,3 +60,34 @@
     owner: root
     group: root
     mode: "0600"
+
+# PROVE THE VIP IS ACTUALLY UP BEFORE THIS PLAY RETURNS (added 2026-09-06).
+#
+# Templating the manifest is not the same as having a floating endpoint: k3s's deploy
+# controller still has to apply it, the image still has to pull, and leader election still
+# has to pick an owner. Nothing used to check, which was tolerable while the VIP was only an
+# operator convenience. It is not tolerable now — Cilium's k8sServiceHost IS this VIP
+# (k3s_apiserver_endpoint) and playbooks/site.yml runs cilium.yml straight after this play,
+# so an unannounced VIP would mean installing a CNI that cannot reach the API server. That
+# failure is the 2026-09-01 shape and it is very expensive to diagnose from the far side;
+# failing here instead names the actual cause.
+#
+# run_once: the VIP is ONE address for the whole environment, so one probe answers for the
+# environment. Probing from a server host (not the controller) is deliberate: the servers
+# are on the VIP's own LAN in both environments, whereas the dev controller reaches dev's
+# API only through a pfSense NAT — a controller-side probe would have been a property of
+# where the play was launched from. READ-ONLY: it opens a TCP connection and changes nothing.
+- name: Wait for the control-plane VIP to answer (the CNI's endpoint depends on it)
+  ansible.builtin.wait_for:
+    host: "{{ kube_vip_address }}"
+    port: "{{ kube_vip_port }}"
+    timeout: "{{ kube_vip_wait_timeout }}"
+  run_once: true
+  changed_when: false
+  vars:
+    ansible_become: false
+  # DELIBERATELY NOT check_mode: false, unlike the interface probe above — so a `--check`
+  # rehearsal SKIPS this and does not sit on a socket for kube_vip_wait_timeout. wait_for
+  # declares no check-mode support, so the skip is the module's own behaviour and this comment
+  # is here only so the skip does not read as an oversight. CONSEQUENCE, stated plainly: a
+  # `--check` run does NOT verify the VIP is up. Only a real run gates the CNI on it.

infra/ansible/roles/memattr_survey/README.md

@@ -0,0 +1,201 @@
+# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
+
+**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
+**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
+is settled there and this role does **not** re-derive it.
+
+## The question
+
+§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
+Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
+and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
+the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
+
+Two things §0b leaves unmeasured, and this role measures:
+
+1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
+   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
+   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
+   the causal chain closes without touching hardware or firmware.
+2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
+
+## Why 20 rounds
+
+Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
+interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
+worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
+mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
+is a variable; lowering it below 20 produces an arm that cannot answer the question it was
+built for, and the analysis will say so.
+
+## This is the BASELINE arm
+
+**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
+moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
+write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
+own allocation footprint — initrd size, module set, boot device — is *part of the causal
+chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
+the kernel release and cmdline were constant on every node, for exactly this reason.
+
+## Running it
+
+```bash
+# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml'
+
+# The real thing. Reboots are OPT-IN; the interlock is deliberate.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml \
+      -e memattr_survey_reboots_authorised=true'
+
+# Re-analyse existing artifacts, no hardware touched.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
+```
+
+Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
+`-e memattr_survey_hosts=<group>`.
+
+20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
+independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
+parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
+preflight asserts the latter. See the playbook comment for why the default linear
+strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
+
+## Safety — verified, not assumed
+
+The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
+premise is re-checked on **every** run, because a document and a machine drift apart:
+
+| Check | Failure mode it prevents |
+|---|---|
+| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
+| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
+| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
+| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
+| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
+
+Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
+whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
+false start is a rebooted production machine.
+
+## Fail-safe and resume
+
+* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
+  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
+  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
+  host only** and the others keep collecting. The abort is announced immediately and again
+  in the report — which counts rounds from the **artifacts**, not from Ansible's host
+  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
+  appears as a short arm with a wider interval, never as a silently missing report.
+* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
+  interruption continues where it stopped and can neither lose nor duplicate a round. The
+  boot found on a resume is *not* re-appended: it is already round N of the interrupted
+  run, and double-counting it would bias the sample toward whichever outcome survived the
+  interruption.
+* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
+  failing task and the reason, so the report can say *why* an arm is short instead of
+  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
+  run and re-written only if that run aborts. Its *absence* is not proof of success — a
+  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
+  runs — so completeness is still judged on round count, and the marker only supplies the
+  reason when there is one.
+* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
+  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
+  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
+
+## Belt and braces on the data
+
+Two independent records, joined on `boot_id`:
+
+| | written by | authoritative for |
+|---|---|---|
+| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
+| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
+| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
+
+The journal does not care whether the driver was alive: the boot happened, the kernel
+logged it, journald persisted it. So the harvest can only ever be a *superset* of the
+ledger, and a round the driver failed to append still shows up.
+
+## Reading the output
+
+Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
+`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
+committed. `summary.md` is also printed at the end of the run.
+
+### The verdict is measured, never inferred from a log line
+
+This is the subtlest part of the design. There are **three** kernel readings, not two:
+
+| `kernel_read` | What the kernel logged | Memory |
+|---|---|---|
+| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
+| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
+| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
+
+Silence is therefore *evidence of health*, not a missing measurement — which is why the
+record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
+And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
+*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
+cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
+`kernel_read` is recorded alongside it as an independent observation. §5a of the report
+cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
+
+Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
+clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
+landing between them is reported as `indeterminate` and excluded from `n` — a new third
+thing deserves human eyes, not a coin flip into a bucket.
+
+### What is and is not pooled
+
+Boots the survey did **not cause** are shown in the per-round table for context and
+excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
+each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
+pooling it would double-count prior evidence into a run whose point is fresh trials).
+
+### The statistics
+
+`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
+
+* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
+  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
+  instead of needing a special case. A bare fraction is what this exercise exists to stop
+  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
+* **The address↔outcome question is tested three ways**, because the tempting version is
+  circular:
+  * **4b determinism** — was any single address seen with *both* outcomes? One such
+    address falsifies "the address determines the outcome" outright.
+  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
+    up as a p-value: the bad set is defined after seeing the data.
+  * **4d the pre-registered prediction** — §0b, written before this run, names
+    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
+    non-circular test available.
+  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
+    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
+    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
+    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
+    address doing causal work, even when none of those parses was catastrophic.
+
+## Files
+
+| Path | What it is |
+|---|---|
+| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
+| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
+| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
+| `tasks/harvest.yml` | The authoritative journal pass |
+| `tasks/analyse.yml` | Controller-side analysis |
+| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
+| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
+
+## What this cannot tell you
+
+It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
+the difference between nodes, because node 2 is simultaneously the only single-DIMM node
+and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
+rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
+settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
+role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
+three nodes, and that is not in dispute.

infra/ansible/roles/memattr_survey/defaults/main.yml

@@ -0,0 +1,92 @@
+---
+# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
+# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
+# settled and is NOT re-derived here.
+#
+# THE QUESTION THIS ROLE ANSWERS
+#   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
+#   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
+#        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
+#        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
+#        address range, the causal chain is closed with no hardware or firmware touched.
+#     2. Is the fault node-2-specific, or does any node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+# WHY 20 ROUNDS AND NOT FIVE
+#   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
+#   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
+#   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
+#   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
+#   not a round number.
+#
+# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
+#   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
+#   installs nothing and configures nothing; its only write to a node is its own append-
+#   only ledger under /var/lib. Anything that alters the boot chain's allocation
+#   footprint — and per §0b that includes the initrd, the module set and the boot device —
+#   changes the variable under test and voids the arm.
+
+# --- Sample size --------------------------------------------------------------------
+# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
+# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
+memattr_survey_rounds: 20
+
+# --- Safety interlock ----------------------------------------------------------------
+# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
+# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
+# That is a genuinely useful dry run and it is also how you prove the pipeline works
+# before committing 45 minutes of wall clock to it.
+memattr_survey_reboots_authorised: false
+
+# The role refuses to run against a host that is not in this group. The prod boxes carry
+# no k3s and nothing is running on them, which is the entire reason rebooting them is
+# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
+memattr_survey_required_group: k3s_prod
+
+# Unit names whose presence or activity means something IS running on this node. Any hit
+# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
+memattr_survey_forbidden_units:
+  - k3s
+  - k3s-agent
+  - kubelet
+  - containerd
+  - docker
+# Paths whose existence means k3s has been installed here, even if not currently running.
+memattr_survey_forbidden_paths:
+  - /etc/rancher/k3s
+  - /var/lib/rancher/k3s
+  - /usr/local/bin/k3s
+
+# --- Reboot behaviour -----------------------------------------------------------------
+# A node that does not answer within this is declared lost: the role stops rebooting THAT
+# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
+# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
+# slow boot.
+memattr_survey_reboot_timeout: 600
+memattr_survey_connect_timeout: 20
+# Let journald settle before reading the new boot's kernel log. The kernel log itself is
+# complete long before SSH is up; this is purely margin.
+memattr_survey_post_reboot_delay: 15
+
+# --- On-node ledger (belt) -------------------------------------------------------------
+# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
+# the controller dies. This is what makes the survey resumable: re-running skips rounds
+# already recorded here rather than starting over.
+memattr_survey_state_dir: /var/lib/memattr-survey
+memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
+
+# --- Controller artifacts (braces) ------------------------------------------------------
+# Same convention as roles/node_health and conformance/_artifacts/: generated output under
+# a `_artifacts/` directory that .gitignore excludes, never committed.
+#   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
+#                        mid-survey still leaves every completed round on the controller
+#   <host>.boots.json    the journal harvest — every retained boot re-derived in one
+#                        pass. THIS is the authoritative record; the ledger is the backup.
+#   summary.md           the analysis (also printed)
+memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
+memattr_survey_write_artifacts: true
+
+# Print each round's one-line result as it happens. A 45-minute unattended run with no
+# output is indistinguishable from a hung one.
+memattr_survey_show_progress: true

infra/ansible/roles/memattr_survey/files/memattr_analyse.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
+
+Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
+  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
+  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
+and joins them on boot_id.
+
+WHAT IT ANSWERS, in order:
+
+  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
+     loudly and up front, because every number below it is weaker than it looks.
+
+  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
+     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
+     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
+     interval — no normal approximation, correct at 0/n and n/n where the approximation is
+     degenerate and where a clean arm is most likely to land.
+
+  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
+     ways, deliberately, because the tempting version of this test is circular:
+       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
+           address falsifies "the address determines the outcome" outright, and no amount
+           of correlation elsewhere repairs it.
+       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
+           as sorted min/max per outcome plus the count of good addresses falling inside
+           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
+           the bad set was chosen after seeing the data.
+       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
+           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
+           this run observed, which is the one non-circular test available.
+
+  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
+     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
+
+Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import sys
+from collections import Counter, defaultdict
+
+# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
+# both times, while all five successes fleet-wide were at other addresses.
+PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
+
+
+# --- Exact binomial interval ----------------------------------------------------------
+def binom_tail_ge(n: int, k: int, p: float) -> float:
+    """P(X >= k) for X ~ Bin(n, p)."""
+    if k <= 0:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
+
+
+def binom_tail_le(n: int, k: int, p: float) -> float:
+    """P(X <= k) for X ~ Bin(n, p)."""
+    if k >= n:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
+
+
+def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
+    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
+
+    Solved by bisection on the exact tail probabilities rather than via the Beta/F
+    quantile identity, so it needs nothing outside the standard library and the
+    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
+      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
+      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
+    """
+    if n == 0:
+        return (0.0, 1.0)
+
+    def bisect(fn, target, lo, hi):
+        for _ in range(200):
+            mid = (lo + hi) / 2
+            if fn(mid) < target:
+                lo = mid
+            else:
+                hi = mid
+        return (lo + hi) / 2
+
+    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
+    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
+    return (lower, upper)
+
+
+def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
+    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
+
+    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
+    never for a bad set defined after the fact — that would be a garden of forking paths
+    with a p-value stapled on.
+    """
+    n = a + b + c + d
+    if n == 0:
+        return 1.0
+    row1, col1 = a + b, a + c
+
+    def prob(x):
+        return (
+            math.comb(row1, x)
+            * math.comb(n - row1, col1 - x)
+            / math.comb(n, col1)
+        )
+
+    observed = prob(a)
+    lo = max(0, col1 - (n - row1))
+    hi = min(row1, col1)
+    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
+    # float noise in the equality case.
+    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
+
+
+# --- Loading ---------------------------------------------------------------------------
+def load_host(art_dir: str, host: str) -> dict:
+    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
+    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
+        harvest = json.load(handle)
+
+    rounds_by_boot: dict[str, int] = {}
+    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
+    ledger: list[dict] = []
+    if os.path.exists(ledger_path):
+        with open(ledger_path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                ledger.append(record)
+                if record.get("boot_id") and isinstance(record.get("round"), int):
+                    rounds_by_boot[record["boot_id"]] = record["round"]
+
+    boots = []
+    for boot in harvest.get("boots", []):
+        entry = dict(boot)
+        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
+        boots.append(entry)
+    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
+
+    # Written by the per-round rescue when a node stops coming back. Its absence is not
+    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
+    # before the rescue runs — so completeness is still judged on round COUNT, and this
+    # only supplies the reason when one exists.
+    abort = None
+    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
+    if os.path.exists(abort_path):
+        try:
+            with open(abort_path, encoding="utf-8") as handle:
+                abort = json.load(handle)
+        except (OSError, json.JSONDecodeError):
+            abort = {"reason": "abort marker present but unreadable"}
+
+    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
+
+
+def discover_hosts(art_dir: str) -> list[str]:
+    return sorted(
+        os.path.basename(p)[: -len(".boots.json")]
+        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
+    )
+
+
+# --- Rendering ---------------------------------------------------------------------------
+def gib(kb) -> str:
+    return "?" if kb is None else f"{kb / 1048576:.2f}"
+
+
+def pct(x) -> str:
+    return "?" if x is None else f"{100 * x:.0f}%"
+
+
+def render(hosts: list[dict], expected_rounds: int) -> str:
+    out: list[str] = []
+    w = out.append
+
+    all_boots = [b for h in hosts for b in h["boots"]]
+    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
+    #   * pre-existing journal boots (round None) — observed under an unknown selection
+    #     process, and already counted once in §0b's table;
+    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
+    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
+    #     count prior evidence into a run whose entire point is fresh, independent trials.
+    # Both are still SHOWN in the per-round table, because context is not contamination.
+    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
+
+    w("# EFI Memory Attributes Table — baseline reboot survey")
+    w("")
+    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
+    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
+    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
+    w("the firmware published the table at.")
+    w("")
+
+    # --- 1. Completeness -----------------------------------------------------------------
+    w("## 1. Completeness — read this before any number below it")
+    w("")
+    w("| node | rounds recorded | target | journal boots harvested | short? |")
+    w("|---|---:|---:|---:|---|")
+    short_arms = []
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        is_short = n < expected_rounds
+        if is_short:
+            short_arms.append((h["host"], n))
+        w(
+            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
+            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
+        )
+    w("")
+    if short_arms:
+        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
+        w("> Every interval below is correspondingly wider than planned.")
+        w("")
+        for h in hosts:
+            if h.get("abort"):
+                a = h["abort"]
+                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
+                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
+                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
+                w("")
+
+    # --- 2. Per-round table ---------------------------------------------------------------
+    w("## 2. Per-round results")
+    w("")
+    for h in hosts:
+        installed = h["harvest"].get("smbios_installed_kb")
+        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
+        w("")
+        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
+        w("|---:|---|---|---|---:|---:|---:|---|")
+        for b in h["boots"]:
+            rnd = b.get("round")
+            label = "pre" if rnd is None else str(rnd)
+            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
+            w(
+                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
+                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
+                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
+                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
+                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
+            )
+        w("")
+        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
+        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
+        w("from every statistic below. `kernel read`: `valid` = table")
+        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
+        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
+        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
+        w("the presence of a log line.")
+        w("")
+
+    # --- 3. p(bad) -------------------------------------------------------------------------
+    w("## 3. p(bad) per node — 95% Clopper-Pearson")
+    w("")
+    w("| node | bad | n | p(bad) | 95% CI | reading |")
+    w("|---|---:|---:|---:|---|---|")
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
+        lo, hi = clopper_pearson(bad, n)
+        if n == 0:
+            reading = "no data"
+        elif bad == 0:
+            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
+        elif bad == n:
+            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
+        else:
+            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
+        w(
+            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
+            f"{lo:.2f} – {hi:.2f} | {reading} |"
+        )
+    w("")
+    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
+    if indet:
+        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
+        w("> reported as `indeterminate` rather than forced into a bucket. They are")
+        w("> excluded from n above. Inspect them by hand.")
+        w("")
+
+    # --- 4. Address vs outcome -----------------------------------------------------------
+    w("## 4. Is the outcome predicted by where the table landed?")
+    w("")
+    by_addr: dict[str, Counter] = defaultdict(Counter)
+    addr_hosts: dict[str, set] = defaultdict(set)
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        by_addr[addr][b.get("verdict")] += 1
+        addr_hosts[addr].add(b.get("host") or "?")
+
+    w("### 4a. Every address observed in the survey")
+    w("")
+    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
+    w("|---|---:|---:|---:|---|---|")
+    ambiguous = []
+    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
+        counts = by_addr[addr]
+        good, bad = counts.get("good", 0), counts.get("bad", 0)
+        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
+        if good and bad:
+            note = "**BOTH — address does not determine outcome**"
+            ambiguous.append(addr)
+        elif bad:
+            note = "always bad"
+        elif good:
+            note = "always good"
+        else:
+            note = "—"
+        w(
+            f"| `{addr}` | {good} | {bad} | {other} | "
+            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
+        )
+    w("")
+
+    w("### 4b. Determinism")
+    w("")
+    if not by_addr:
+        w("No surveyed boots — nothing to test.")
+    elif ambiguous:
+        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+          + ", ".join(f"`{a}`" for a in ambiguous)
+          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
+    else:
+        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
+        w("consistent with the address determining the outcome, though with this many")
+        w("distinct addresses most were seen only once, which makes the test weak.")
+        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
+        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
+    w("")
+
+    w("### 4c. Separation of the address ranges")
+    w("")
+    bad_addrs = sorted({b["memattr_address"] for b in surveyed
+                        if b.get("verdict") == "bad" and b.get("memattr_address")})
+    good_addrs = sorted({b["memattr_address"] for b in surveyed
+                         if b.get("verdict") == "good" and b.get("memattr_address")})
+    if bad_addrs and good_addrs:
+        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
+        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
+        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
+        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
+        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
+             if inside else " — the ranges are disjoint"))
+        w("")
+        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
+        w("picture, not a test.")
+    elif not bad_addrs:
+        w("No bad boot was observed in this survey, so there is no bad range to separate.")
+    else:
+        w("No good boot was observed in this survey.")
+    w("")
+
+    w("### 4d. The pre-registered prediction")
+    w("")
+    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
+    w("bad address (both prior failures were there; all five prior successes elsewhere).")
+    w("Tested against this survey's boots only:")
+    w("")
+    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    w(f"| | bad | good |")
+    w(f"|---|---:|---:|")
+    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
+    w(f"| elsewhere | {c} | {d} |")
+    w("")
+    if a + b_ == 0:
+        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
+        w("prediction had no opportunity to be right or wrong. That is itself informative:")
+        w("the address moves every boot, so a specific address is not a reproducible handle.")
+    else:
+        p = fisher_exact_2x2(a, b_, c, d)
+        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+          + ("  — the prediction holds." if p < 0.05 and a and not b_
+             else "  — not significant at 0.05." if p >= 0.05
+             else "  — significant, but read the cell counts before believing it."))
+    w("")
+
+    w("### 4e. Does the address predict how the KERNEL READ the table?")
+    w("")
+    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
+    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
+    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
+    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
+    w("If the address determines the reading, the address is doing causal work, one step")
+    w("short of the catastrophic path.")
+    w("")
+    reads_by_addr: dict[str, set] = defaultdict(set)
+    n_by_addr: Counter = Counter()
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        reads_by_addr[addr].add(b.get("kernel_read"))
+        n_by_addr[addr] += 1
+    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
+    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
+    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
+    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
+      f"**{len(repeated)}** were observed more than once")
+    w(f"- distinct kernel readings across the survey: "
+      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
+    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
+    w("")
+    if len(distinct_reads) < 2:
+        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
+    elif inconsistent:
+        w("**The address does NOT determine the reading:** "
+          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
+    elif not repeated:
+        w("No address recurred, so the test could not be run.")
+    else:
+        w(f"**The address determines the reading, on every address observed more than once**")
+        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
+        w("readings present in the sample). This is direct evidence for the causal chain")
+        w("§0b describes: the same address reproduces the same parse, boot after boot.")
+        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
+        w("only that WHERE the table lands is what decides what the kernel finds there.")
+    w("")
+    w("| MEMATTR | boots | kernel reading(s) |")
+    w("|---|---:|---|")
+    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
+        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
+        w(f"| `{addr}` | {n_by_addr[addr]} | "
+          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
+    w("")
+
+    # --- 5. Corroboration -------------------------------------------------------------------
+    w("## 5. Corroborating evidence")
+    w("")
+    w("### 5a. Kernel's own reading vs the measured outcome")
+    w("")
+    cross: dict[tuple, int] = Counter()
+    for b in surveyed:
+        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
+    w("| kernel read | verdict | boots |")
+    w("|---|---|---:|")
+    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
+        w(f"| {read} | {verdict} | {count} |")
+    w("")
+    disagreements = [
+        (read, verdict, count)
+        for (read, verdict), count in cross.items()
+        if (read == "failed-map") != (verdict == "bad")
+    ]
+    if disagreements:
+        w("> **The log line and the measurement DISAGREE** in "
+          f"{sum(c for _, _, c in disagreements)} boot(s): "
+          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
+        w("> `failed-map` costs memory.")
+    else:
+        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
+        w("exactly as §0b's mechanism predicts, and independent of the address question.")
+    w("")
+
+    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
+    w("")
+    w("| verdict | NODE_DATA end address | boots |")
+    w("|---|---|---:|")
+    nd: Counter = Counter()
+    for b in surveyed:
+        line = b.get("node_data_line") or ""
+        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
+        nd[(b.get("verdict"), end)] += 1
+    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
+        w(f"| {verdict} | `{end}` | {count} |")
+    w("")
+
+    # --- 6. Invariants ------------------------------------------------------------------
+    w("## 6. Nothing under test changed")
+    w("")
+    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
+    w("|---|---|---:|---|")
+    for h in hosts:
+        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
+        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
+        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
+          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
+    w("")
+    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
+    w("")
+    return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--artifact-dir", required=True)
+    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
+    parser.add_argument("--out", default=None, help="also write the report here")
+    args = parser.parse_args()
+
+    names = discover_hosts(args.artifact_dir)
+    if not names:
+        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
+        return 1
+    hosts = [load_host(args.artifact_dir, name) for name in names]
+    report = render(hosts, args.rounds)
+    if args.out:
+        with open(args.out, "w", encoding="utf-8") as handle:
+            handle.write(report)
+    sys.stdout.write(report)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())