feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

.gitignore

@@ -44,3 +44,6 @@ infra/tofu/.r2.env
 
 # Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
 infra/ansible/collections/
+
+# standup.sh DNS snapshots — run artifacts, not authored state
+.standup-dns/

DECISIONS-LOG.md

@@ -1758,3 +1758,279 @@ into a transient hard Kustomization failure.
 - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
   with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
   Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
+
+## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched
+
+Operator asleep; standing instruction was "finish out what is necessary and I want it working by
+the morning." **Prod is serving.** Dev was verified unharmed at every step.
+
+### Final state
+
+| | prod | dev |
+|---|---|---|
+| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** |
+| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` |
+| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) |
+| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready |
+| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged |
+| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** |
+
+End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`,
+keycloak `302` (login), gitea `200`, grafana `302` (SSO).
+
+### What was actually blocking, and what it taught
+
+**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a
+Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.**
+
+1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation
+   limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers
+   with no API types to watch. The failed install then tripped `install.remediation.retries: 3`,
+   whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller
+   cannot advance from. Cleared with `helm uninstall` + a cold reinstall.
+2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR
+   *and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until
+   `cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier
+   can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev**
+   (its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying
+   only the CR; the CRD appeared in 10s.
+
+> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own
+> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation
+> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a
+> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster.
+
+### The armed footgun, disarmed
+
+Prod's external-dns Deployment was running `--policy=sync --registry=txt
+--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`,
+the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the
+records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was
+`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a
+guard, it is luck.**
+
+The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The
+fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported
+`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the
+current config, so it would never have upgraded on its own.
+
+**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets
+replicas to 1 and applies new args in one operation — there is no window to check first. So
+`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=<owner>`
+appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into
+a precondition.
+
+### Three bugs in my own verification — recorded because a check you cannot trust is worse than none
+
+1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline
+   was discarded between steps and the before/after comparison — the check that detects damage to the
+   other cluster — could never run across invocations. Now repo-relative and gitignored.
+2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing
+   pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev`
+   differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots
+   now strip CRs and sort under a fixed collation.
+3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script
+   claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight
+   is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is
+   what actually causes harm.
+
+### I also fell into the pfSense trap — ACTION REQUIRED
+
+The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new
+`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare
+correctly returns `192.168.2.20`.
+
+**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the
+override, or add one pointing `prod.home.*` at `192.168.2.20`.
+
+It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches
+`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that
+as a broken prod cert before checking the resolver. **This is the same trap that produced the false
+healthy reading during the incident. Never verify DNS or TLS here through a local resolver.**
+
+### Still open for the operator
+
+- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes
+  it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup
+  coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`).
+- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a
+  rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
+- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease
+  change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
+- **Six L2 tests hardcode the apex** and must be parameterized per environment.
+- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.
+
+## 2026-09-05 (afternoon) — **PM autonomous session.** Backups to R2, CDI deadlock repaired, coverage 82%→88% — and a guard I broke
+
+Operator away ~2h, PM directed to work autonomously through agents. Four agents ran;
+every result was re-verified by the PM before commit. **Nothing merged, nothing pushed.**
+Both clusters healthy throughout — dev 13/13, prod 12/13 (velero deliberately suspended).
+
+Commits: `32d6766` R2 · `765ad03` CDI · `59ffbaf` L2 params · `8f9b2e4` s3check · `fe45d0f` coverage.
+
+### THE MISTAKE, AND WHY IT MATTERS MORE THAN THE FEATURES
+
+In `32d6766` I changed the backup placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
+`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`. The guards in `standup.sh` matched
+`TBD-*` — a **prefix** glob. `fnmatch('https://TBD-R2-…','TBD-*')` is **False**, so all three
+guards silently died, and **`resume` would have enabled velero on prod against a nonexistent
+hostname**: installs clean, reports healthy, no backup exists. Precisely the failure the guard
+existed to prevent, disarmed by a value change in a *different file*.
+
+**The class, now seen three times today** (this; the CRD count that read a blank line as `1`;
+two coverage checks that passed when perturbed): *a check that degrades to success*. The
+structural lesson is narrower and worth keeping — **a guard keyed on a value's SHAPE, where the
+value lives in another file, has no test tying the two together.** Fixed by substring match;
+a test for the coupling is **not** yet written and should be.
+
+### Operator decisions waiting
+
+1. **R-TLS-02 "revoked" is NOT satisfied — a real platform gap.** The spec requires certs be
+   "issued, renewed, **and revoked** without manual steps"; docs/08 reads revocation as a
+   declarative removal. That needs `enableCertificateOwnerRef: true`, which
+   `platform/cert-manager/helmrelease.yaml` does not set (chart default false). **Verified on
+   both live clusters: `gateway/wildcard-home-tls` has ZERO ownerReferences** — deleting the
+   Certificate leaves the private key serving indefinitely.
+   **Deliberately not flipped.** The wildcard Certificate is under `./platform`, which runs
+   `prune: true`. With owner refs ON, a transient prune of that Certificate — *exactly what
+   happened to the CDI StorageProfiles today* — deletes the SERVING TLS Secret for every host,
+   and reissuance is bounded by LE's duplicate-certificate limit (5/week per identical name set),
+   risking **no trusted cert for a week**. Security gap vs plausible week-long outage: operator's
+   call. The L2 test asserts the requirement as written and **will fail until decided** — it was
+   not softened.
+2. **R2 needs the two token values.** `infra/BACKUP-R2.md` is the procedure. The **account id is
+   already committed** (`infra/tofu/backend.r2.hcl` → `7ce452a1b3d6c86daf2c020edcb82d37`), so if
+   the buckets go in that account only the token is missing. Placeholder kept on purpose.
+3. **The apex flip is also the legacy-prod 443 cutover.** The SOHO router NATs inbound 443 to the
+   pfSense WAN for the non-k3s prod. §10a sequence P does not cover repointing it.
+
+### Merge preconditions — read before merging anything here
+
+- **CDI split:** `flux get kustomization platform-config` must be **Ready on the target first**.
+  Otherwise the two StorageProfiles are pruned and sit at CDI defaults until it recovers (prod was
+  in that state hours earlier). Measured exposure: CDI unaided infers **ReadWriteOnce**
+  (`longhorn-static`) where we author **ReadWriteMany** (`longhorn-vm`) — so a DataVolume created
+  in the ~30–60s window is **permanently non-live-migratable**. `cloneStrategy` is *not* at risk.
+- **R2 switch RESETS dev's PITR window** — fresh kopia repo, new CNPG base backups, the 30-day
+  Barman window rebuilds from zero. The MinIO copy is retained and restorable.
+- **R-STATE-03 is PROVEN MinIO-ONLY** and its ledger row predates the R2 move. `BACKUP-R2.md` §5
+  is the re-proof and it ends in a **restore**, not an upload.
+
+### Findings worth keeping
+
+- **Second-order CRDs need a third pass.** A CR whose CRD is created by a *controller at runtime*
+  cannot share a Kustomization with resources of that CRD. `cdi-operator` registers exactly one
+  CRD; `storageprofiles`/`datavolumes`/`datasources` appear only after it reconciles the CDI CR.
+  Audit found `StorageProfile` was the only second-order kind in `platform-config`.
+- **`dns-exposure`'s teardown is ASYNCHRONOUS.** It reports PASS when chainsaw deletes the
+  namespace; the Cloudflare record is reaped later by external-dns's 1-minute sync. A green run
+  does **not** prove cleanup — if external-dns were suspended the record would persist silently.
+  (Observed, then observed clearing.)
+- **The apex was not the only per-environment literal in the L2 suite** — two DNS tests hardcoded
+  dev's LB range `192.168.1.53-.60`. They now assert the *observed* `GATEWAY_VIP` / assigned LB IP,
+  which is stricter as well as portable.
+- **Velero's kopia backend has no checksum knob**, so BSL metadata and PVC-data R2 compatibility
+  are **separate questions**. CNPG's boto3 1.35.60 is below the 1.36 default-checksum line — clear
+  today, but **a CNPG image bump is an R2 re-verification event** (barman-cloud has no escape hatch).
+- **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into
+  the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI.
+  Confirmed no plaintext reached disk.
+
+## 2026-09-06 — Prod solidified. Node 2 at full 32 GB, storage recovered, four latent cold-start bugs closed
+
+Operator away; PM directed to oversee, delegate, and pay down debt while skipping anything needing
+their input. Five agents ran; every result re-verified by the PM before commit. **Nothing merged,
+nothing pushed. Both clusters healthy.**
+
+Commits: `04569ab` node-maintenance runbook · `da3e9cc` Kyverno/CDI · `a984442` timezone + guard test.
+
+### PROD IS FUNCTIONING
+
+| | |
+|---|---|
+| Nodes | 3/3 Ready, **all three at ~`32638256Ki`** — node 2's DIMM upgrade landed, matching 1 and 3 to within 4 KiB |
+| Flux | **12/13 tiers Ready** (`velero` suspended: no backup endpoint) |
+| Pods | **112 Running**, 0 Running-but-NotReady |
+| Storage | **17/17 volumes healthy**, replicas symmetric 17-running+1-stopped per node |
+| TLS | LE wildcard `*.prod.home.keithjackson.dev` Ready |
+| etcd | 3 members, survived an ungraceful power-down without losing a member |
+
+The memattr saga (§0b) is **closed on the hardware side**: with the 7.1.8 backports kernel, all three
+nodes address their full 32 GB.
+
+### THE FINDING THAT MATTERS MOST — a failed drain silently stops storage self-healing
+
+`kubectl drain` **cordons before it fails**, and Longhorn's `disable-scheduling-on-cordoned-node` is
+`true`. After prod-2's memory-upgrade reboot, **all 17 volumes sat `attached/degraded` with prod-2
+holding 2 of 18 replicas and ZERO rebuilds in flight** — nothing driving recovery, every pod
+`Running`, no alert. `kubectl uncordon` started rebuilds immediately; converged 17/17 in ~6 minutes.
+
+**Uncordon is not tidying up — it is what re-enables redundancy.** Full runbook + post-maintenance
+checklist in `infra/PROD-STANDUP.md` §Node maintenance, including why `drain` can never complete here
+(instance-manager PDBs at 0 allowed disruptions; at 3 nodes/3 replicas replica eviction has nowhere to
+go, which is structural, not a setting).
+
+### FOUR BUGS OF ONE SHAPE — *a thing that looks applied and isn't*
+
+Worth naming as a class, because it has now appeared four times in two days:
+
+1. **Kyverno's own CRDs** — `crds.install: false`, nothing else installed them; pods crash-looped 55×.
+2. **CDI StorageProfiles** — CR and its dependent resource in the same Flux pass; the dry-run aborts
+   the tier, so the CR never applies and its CRD never appears. *Repaired in the graph* (`765ad03`).
+3. **The placeholder guard** — a `TBD-*` prefix glob stopped matching when the value it guards changed
+   *in a different file*; `resume` would have enabled velero against a nonexistent host. Now tested
+   (`a984442`).
+4. **PolicyException** — the CRD is installed but every controller runs
+   `--enablePolicyException=false`. An exception would apply cleanly through Flux and be **silently
+   ignored**. Verified live on prod.
+
+Two more of the same family: a coverage check that passed when perturbed (asked a question always
+satisfiable), and `ci/touched-requirements.sh` having **no `bootstrap/*` entry**, which scoped the L0
+run away from the very check guarding the placeholder class. Both fixed.
+
+**The rule:** dev's age hides cold-start defects. Anything that only runs on a fresh cluster has never
+been exercised on dev, and prod is the first honest test. And: *a guard keyed on the shape of a value
+that lives in another file needs a test tying the two together.*
+
+### Kyverno blocking CDI — fixed by SATISFYING the policy, not exempting it
+
+Prod's UniFi VM had no disk: Kyverno denied CDI's generated `importer-*` pod for missing the
+`auth-mode` / `egress-class` labels, which CDI cannot set. Fixed with a **mutate** policy that adds
+them when the requester is ServiceAccount `cdi/cdi-sa` — identity from AdmissionReview userInfo, **not
+a name prefix** (a prefix is attacker-choosable; the test proves it with a forged
+`importer-prime-forged-by-an-app` fixture that is still denied). Two-way proof: same pod, different
+requester, opposite outcomes. Kyverno L0 17 → 29 tests.
+
+Rejected `exclude` scoped to the same SA because any userInfo match forces `background: false`, which
+would drop background scanning of R-NET-05/R-IDP-02 for **all existing pods** to fix one pod class.
+
+### Conformance 82.4% → 88.2%
+
+45/51 scaffolded, 0 safety-critical gaps. Six L2 tests made environment-agnostic (they derive
+`BASE_DOMAIN`/`GATEWAY_VIP` from the cluster they run against, so they cannot drift); R-TLS-02,
+R-ENV-01 and R-ENV-03 covered — the last two matter because the Phase-7 entry-gate note recorded
+R-ENV-01/02/03 as uncovered while prod provisioning was proceeding.
+
+### STAGED FOR ONE PUSH — `prod/deliver-fixes`
+
+Prod reconciles bootstrap-Gitea `main@c8c1d29` and **cannot take the working branch** — that carries
+the DNS inversion, which would move prod to the apex and collide with dev. `prod/deliver-fixes` is
+`c8c1d29` + the CDI graph fix + the Kyverno mutate policy: a clean fast-forward, verified free of any
+cluster-vars/externaldns/overlay change, prod's `BASE_DOMAIN` untouched, render checked. **Not pushed
+— that changes what a live cluster reconciles.**
+
+### Waiting on the operator (deliberately skipped)
+
+- **R2 credentials.** Two token values; the account id is already committed in
+  `infra/tofu/backend.r2.hcl`. `infra/BACKUP-R2.md` is the procedure. **Prod has no backup coverage.**
+- **`enableCertificateOwnerRef`** — R-TLS-02's "revoked" clause is unmet; both clusters' TLS Secrets
+  have zero ownerReferences. Not flipped because the wildcard Certificate lives under a `prune: true`
+  tier, so a transient prune would delete the serving Secret and reissuance is capped at 5/week.
+- **Node timezone** — authored (`Etc/UTC`, matching dev), `--check` confirms the drift, **not applied**:
+  a live clock change during storage resync is not an unattended action.
+- **Database HA** — all three CNPG clusters are `instances: 1` and all three primaries are on prod-1.
+  Losing that node takes Keycloak, Nextcloud and Semaphore together. `instances: 2` spends resources.
+- **The apex flip** is also the legacy-prod 443 cutover; §10a sequence P does not cover the SOHO NAT.
+- **R-STATE-04** stays a gap: `bulk` is a non-provisioning stub pending **OQ-2**.
+- Orphaned `prime-be3f9d1c-…` 40 GiB scratch PVC from the blocked imports — operator cleanup.

README.md

@@ -69,6 +69,17 @@ infra/devcontainer/dev.sh run make -C ci gate   # the same gate CI runs
 infra/devcontainer/dev.sh                  # interactive shell (sops, flux, kubectl, tofu, ansible)
 ```
 
+`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
+target prod instead:
+
+```bash
+infra/devcontainer/dev.sh run kubectl get nodes                # dev (default)
+CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod
+```
+
+Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
+`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
+
 The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
 so the spec-immutability check can actually run. See
 [`infra/devcontainer/README.md`](infra/devcontainer/README.md).

apps/wave-0/kustomization.yaml

@@ -11,7 +11,6 @@ resources:
 - jackett-oauth2-proxy-credentials.sops.yaml
 - jackett.yaml
 - nextcloud-db-app.sops.yaml
-- nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
@@ -20,6 +19,5 @@ resources:
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml
-- semaphore-db-backup-credentials-prod.sops.yaml
 - semaphore-db-backup-credentials.sops.yaml
 - semaphore-db.yaml

apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml

@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: nextcloud-db-backup-credentials-prod
-    namespace: nextcloud-db
-type: Opaque
-stringData:
-    ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
-    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
-            M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
-            aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
-            ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
-            8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

apps/wave-0/nextcloud-db.yaml

@@ -68,38 +68,29 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
-  # s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
-  # s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
-  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
-  # scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
-  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
-  # generated golden — render.py places the -prod sidecar in this DB's wave); the
-  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
-  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
-  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
-  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
-  # disk-full clock below rather than a merely missing backup.
-  #
-  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
-  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
-  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
-  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
+      # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
+      # NAME — and both environments have a `nextcloud-db`. Sharing one MinIO with a shared path
+      # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
+      # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
       destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: nextcloud-db-backup-credentials
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: nextcloud-db-backup-credentials
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml

@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: semaphore-db-backup-credentials-prod
-    namespace: semaphore-db
-type: Opaque
-stringData:
-    ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
-    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
-            K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
-            T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
-            aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
-            7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

apps/wave-0/semaphore-db.yaml

@@ -69,38 +69,29 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
-  # s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
-  # s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
-  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
-  # scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
-  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
-  # generated golden — render.py places the -prod sidecar in this DB's wave); the
-  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
-  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
-  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
-  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
-  # disk-full clock below rather than a merely missing backup.
-  #
-  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
-  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
-  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
-  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
+      # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
+      # NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path
+      # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
+      # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
       destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: semaphore-db-backup-credentials
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: semaphore-db-backup-credentials
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

bootstrap/flux/README.md

@@ -11,7 +11,8 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
+| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
 | `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
@@ -74,6 +75,49 @@ flux get kustomizations -A                                  # all 11 must return
 **Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
 simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
 
+## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
+
+`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
+stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
+`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
+`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
+HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
+
+The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
+Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
+because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
+
+```
+helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
+  | yq 'select(.kind == "CustomResourceDefinition")'
+  | kubectl apply --server-side --force-conflicts
+```
+
+- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
+  read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
+  **constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
+  helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
+  of that one line carries this step with it. There is nothing here that can drift.
+- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
+  HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
+  HelmRelease over field managers. The `yq` filter is load-bearing.
+- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
+  hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
+  `serverside-applied` and exits 0.
+- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
+  `null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
+  exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
+  waited on for `condition=Established`. There is **no skip flag** on purpose.
+- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
+  repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
+
+Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
+
+```bash
+# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
+bash bootstrap/flux/preapply-crds.sh
+```
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
@@ -98,7 +142,7 @@ tofu apply → cloud-init → Ansible (k3s + Cilium)          [Infra worker, T-1
         ▼  (final Ansible play calls the entrypoints below)
 1. bootstrap/sops/age-keygen.sh          # operator generates age key OUTSIDE repo
 2. bootstrap/sops/age-key-secret.md      # operator installs private key as sops-age Secret
-3. bootstrap/flux/bootstrap.sh           # flux install → verify sops-age → apply -k clusters/dev
+3. bootstrap/flux/bootstrap.sh           # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
         │
         ▼
 Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
@@ -118,8 +162,10 @@ live in `infra/devcontainer/versions.env`).
    template). `bootstrap.sh` **fails fast** if this Secret is absent.
 3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
 
-`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
-never handles key material — it only *verifies* the Secret exists.
+`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
+`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
+exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
+both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
 
 ## Handoff — clusters owner / PM (files this worker may not edit)
 

bootstrap/flux/bootstrap.sh

@@ -33,10 +33,19 @@
 #                         the single apps.yaml was retired). Letting
 #                         bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
 #                         applying the authored clusters/dev overlay is the reconciling path.
-#   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
+#   2. CRD pre-apply   — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
+#                         platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
+#                         (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
+#                         which STALLS the HelmRelease and blocks the `platform` Kustomization and
+#                         every tier below it). With CRD install off, NOTHING else installs them, so
+#                         a cold cluster can never reconcile `platform` without this step. The chart
+#                         version is DERIVED from that same HelmRelease — no second pin. Dev only
+#                         ever had these CRDs because they were applied BY HAND; this is that step,
+#                         in git, idempotent and version-derived. See ./preapply-crds.sh.
+#   3. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
+#   4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -71,7 +80,10 @@ repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd
 echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
-for bin in flux kubectl; do
+# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
+# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
+# Flux controllers are installed, instead of half-way through.
+for bin in flux kubectl helm yq; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
@@ -99,7 +111,7 @@ done
 #
 # ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
 # strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
-echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
 flux_install_args=(--namespace="${FLUX_NAMESPACE}")
 [ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
 
@@ -130,18 +142,29 @@ else
   kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
-# --- 2. verify the out-of-band age key Secret exists -------------------------
+# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
+# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
+# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
+# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
+# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
+# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
+# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
+# cluster that looks bootstrapped and reconciles nothing.
+echo "==> [2/4] pre-applying chart CRDs (server-side)"
+bash "${script_dir}/preapply-crds.sh"
+
+# --- 3. verify the out-of-band age key Secret exists -------------------------
 # We DO NOT create it here (it holds the private key). The operator installs it per
 # ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
-echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
+echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
 if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
   echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
   echo "       Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
   exit 1
 fi
 
-# --- 3. apply the authored environment entrypoint ----------------------------
-echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
+# --- 4. apply the authored environment entrypoint ----------------------------
+echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 
 echo

bootstrap/flux/preapply-crds.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+#
+# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
+# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
+# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
+# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
+# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
+#
+# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
+# =============================================================================
+#   *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
+# =============================================================================
+#
+# WHY THIS EXISTS
+#
+#   platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
+#   the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
+#   annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
+#   STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
+#   it (the kps-cascade failure mode; see the HelmRelease's own comment).
+#
+#   The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
+#   Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
+#   cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
+#   undocumented manual step this repo exists to eliminate. This script is that step, in git,
+#   version-derived and idempotent.
+#
+# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
+#
+#   Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
+#
+#     chart name      spec.chart.spec.chart            (HelmRelease)
+#     chart version   spec.chart.spec.version           (HelmRelease)  e.g. "3.9.x"
+#     chart repo url  spec.url                          (HelmRepository)
+#     namespace       metadata.namespace                (HelmRelease)
+#
+#   The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
+#   constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
+#   through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
+#   one file and this step follows automatically — it cannot drift, because there is nothing here to
+#   drift from.
+#
+# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
+#
+#   The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
+#   into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
+#   ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
+#   filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
+#   (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
+#
+# IDEMPOTENCY
+#
+#   `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
+#   every later run reports "serverside-applied" and exits 0, and field ownership is taken from
+#   whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
+#   hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
+#   Server-side apply does not write the last-applied annotation, which is the whole reason the
+#   5.6 MiB bundle can be applied at all.
+#
+# LOUD FAILURE (this script must never degrade to a successful no-op)
+#
+#   `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
+#   class of bug this repo has been bitten by more than once:
+#     1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
+#     2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
+#        filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
+#     3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
+#        to mean "the API type exists", not merely "kubectl returned 0".
+#
+set -euo pipefail
+
+# --- config (env-overridable) ------------------------------------------------
+# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
+# Flux reconciles, so it is the file this step must agree with.
+CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
+# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
+CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
+
+echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
+
+# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
+# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
+# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
+for bin in helm yq kubectl; do
+  command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
+done
+
+helmrelease="${repo_root}/${CRD_HELMRELEASE}"
+[ -f "${helmrelease}" ] || {
+  echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
+  exit 1
+}
+
+# --- 1. derive the chart coordinates from the authored manifests --------------
+chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
+chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
+chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
+chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
+
+# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
+# "null" for a missing path and an empty string for a missing document, so both are rejected.
+for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
+  key="${pair%%:*}"; val="${pair#*:}"
+  case "${val}" in
+    ""|null|*$'\n'*)
+      echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
+      echo "       The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
+      exit 1
+      ;;
+  esac
+done
+
+echo "    chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
+echo "    (version is the constraint Flux resolves — NOT a second copy pinned here)"
+
+# --- 2. render the chart and keep ONLY the CRD documents ----------------------
+render_dir="$(mktemp -d)"
+trap 'rm -rf "${render_dir}"' EXIT
+crd_file="${render_dir}/crds.yaml"
+
+# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
+# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
+echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
+helm template "${chart_name}" "${chart_name}" \
+  --repo "${chart_repo}" \
+  --version "${chart_version}" \
+  --namespace "${chart_ns}" \
+  --include-crds \
+  --set crds.install=true \
+  | yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
+
+# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
+# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
+mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
+crd_count="${#crd_names[@]}"
+
+# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
+[ "${crd_count}" -gt 0 ] || {
+  echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
+  echo "       The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
+  echo "       applying nothing would look like success and leave 'platform' blocked forever." >&2
+  exit 1
+}
+echo "    ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
+
+# --- 3. server-side apply (idempotent upsert) ---------------------------------
+# --server-side: no last-applied annotation, so the size limit does not apply.
+# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
+#   Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
+echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
+kubectl apply --server-side --force-conflicts -f "${crd_file}"
+
+# --- 4. prove the API types actually exist -----------------------------------
+# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
+# controllers and the platform Kustomization actually need.
+echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
+crd_refs=()
+for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
+kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
+  "${crd_refs[@]}" >/dev/null || {
+  echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
+  echo "       Inspect: kubectl get crd | grep ${chart_name}" >&2
+  exit 1
+}
+
+echo "==> ${crd_count} ${chart_name} CRDs applied and Established."

bootstrap/standup.sh

@@ -0,0 +1,988 @@
+#!/usr/bin/env bash
+#
+# standup.sh — bring a bootstrapped cluster's reconciliation graph all the way up.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): it reads everything it needs from the cluster it is pointed at and
+# from the authored manifests. There is no dev/prod fork. What makes it safe to run against prod is
+# the MANDATORY --expect-server guard, not a name in the filename.
+#
+# WHY THIS EXISTS
+#   `bootstrap/flux/bootstrap.sh` installs Flux and applies the graph. It does NOT finish the job when
+#   tiers have been deliberately suspended (a DNS/TLS migration window, an incident recovery) or when
+#   a HelmRelease has wedged. Those last steps were living in shell history and in a human's head,
+#   which is the same failure the Kyverno CRD pre-apply had. This is that sequence, in git, ordered,
+#   idempotent, and gated.
+#
+# THE ORDERING IS LOAD-BEARING. Read infra/PROD-STANDUP.md before deviating from it.
+#
+# USAGE
+#   bootstrap/standup.sh --kubeconfig <path> --expect-server <url> [--confirm-destructive] <step...>
+#
+#   steps:  preflight  crds  unwedge  externaldns  s3check  resume  verify   (or: all)
+#
+#   --expect-server is REQUIRED and is compared against the live API server before ANY command runs.
+#   Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and
+#   shares a Cloudflare zone with prod. The guard is not optional and there is no override flag.
+#
+#   --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
+#
+#   --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
+#   cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
+#   `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
+#   nothing else and applies nothing. See infra/BACKUP-R2.md.
+#
+set -euo pipefail
+
+# ---------------------------------------------------------------------------- args
+KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE=""
+STEPS=()
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --kubeconfig)          KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;;
+    --expect-server)       EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;;
+    --confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;;
+    --s3-endpoint)         S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;;
+    -h|--help)             sed -n '2,30p' "$0"; exit 0 ;;
+    -*)                    echo "ERROR: unknown flag '$1'" >&2; exit 2 ;;
+    *)                     STEPS+=("$1"); shift ;;
+  esac
+done
+
+[ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; }
+[ -n "${EXPECT_SERVER}" ]  || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; }
+[ "${#STEPS[@]}" -gt 0 ]   || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; }
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))"
+cd "${repo_root}"
+
+export KUBECONFIG="${KUBECONFIG_ARG}"
+[ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; }
+
+# Where DNS snapshots go. This MUST be a path that survives between invocations: each `dev.sh run` is
+# a FRESH container, so a $TMPDIR default silently threw the baseline away and made `verify`'s
+# before/after comparison — the check that detects damage to the OTHER cluster's records — impossible
+# to run across steps. The repo is bind-mounted, so a repo-relative dir persists on the host.
+# Gitignored: these are run artifacts, not authored state.
+SNAP_DIR="${STANDUP_SNAP_DIR:-${repo_root}/.standup-dns}"
+mkdir -p "${SNAP_DIR}"
+
+say()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
+
+for bin in kubectl flux helm yq sops; do
+  command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
+done
+
+# ------------------------------------------------------------------ THE SAFETY GUARD
+# Checked once, before any step, and re-checked by every mutating helper. The failure mode this
+# prevents is running a prod bring-up against the LIVE dev cluster, whose public DNS records prod
+# has already destroyed once (2026-09-04).
+live_server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
+[ "${live_server}" = "${EXPECT_SERVER}" ] || die \
+  "TARGET MISMATCH — refusing to run.
+       --expect-server : ${EXPECT_SERVER}
+       live API server : ${live_server}
+       The kubeconfig does not point where you think it does. Nothing has been changed."
+say "target confirmed: ${live_server}"
+
+# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
+# each wait is meaningful instead of blocking on a dependency that is still suspended.
+#   flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
+#                                     -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
+#
+# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
+# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
+# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
+# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
+# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
+# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
+TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
+# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
+# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
+
+cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
+
+k_ready()  { kubectl -n flux-system get kustomization "$1" \
+               -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true; }
+k_msg()    { kubectl -n flux-system get kustomization "$1" \
+               -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' 2>/dev/null || true; }
+k_susp()   { kubectl -n flux-system get kustomization "$1" -o jsonpath='{.spec.suspend}' 2>/dev/null || true; }
+
+wait_ready() { # <kustomization> <timeout-seconds>
+  local n="$1" t="${2:-600}" i=0
+  while [ "$i" -lt "$t" ]; do
+    [ "$(k_ready "$n")" = "True" ] && { info "$n Ready (${i}s)"; return 0; }
+    sleep 10; i=$((i+10))
+  done
+  warn "$n did NOT reach Ready in ${t}s: $(k_msg "$n")"
+  return 1
+}
+
+# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable.
+# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard
+# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false
+# healthy reading during the 2026-09-04 incident and hid the damage for a while.
+dns_snapshot() { # <output-file>
+  local out="$1" tok zid page tp
+  tok="$(sops -d platform/externaldns/cloudflare-api-token.sops.yaml \
+         | yq '.stringData."api-token"' | tr -d '"')"
+  [ -n "${tok}" ] && [ "${tok}" != "null" ] || die "could not read the Cloudflare token (SOPS/age key available?)"
+  zid="$(curl -sS -H "Authorization: Bearer ${tok}" \
+        'https://api.cloudflare.com/client/v4/zones?name=keithjackson.dev' | yq '.result[0].id' | tr -d '"')"
+  [ -n "${zid}" ] && [ "${zid}" != "null" ] || die "Cloudflare zone lookup failed"
+  page=1; : > "${out}"
+  while :; do
+    local r; r="$(curl -sS -H "Authorization: Bearer ${tok}" \
+      "https://api.cloudflare.com/client/v4/zones/${zid}/dns_records?per_page=100&page=${page}")"
+    printf '%s' "${r}" | yq -o=json '.result' \
+      | yq -r '.[] | (.type + "|" + .name + "|" + .content)' | tr -d '"' >> "${out}"
+    tp="$(printf '%s' "${r}" | yq '.result_info.total_pages' | tr -d '"')"
+    [ "${page}" -ge "${tp:-1}" ] && break
+    page=$((page+1))
+  done
+  # Strip CRs, and sort under a FIXED collation. Both matter, and both were learned the hard way:
+  #   - a stray \r makes every line differ from a baseline captured without one;
+  #   - locale-dependent sort ordering puts `*.keithjackson.dev` in a different position on the host
+  #     than in the container, so a diff reports pairs of IDENTICAL lines as changed. That produced a
+  #     "RECORDS OUTSIDE ... CHANGED" alarm on a zone where nothing had changed at all.
+  # A regression check that cries wolf is worse than none — it trains you to skip the one alarm that
+  # matters. LC_ALL=C makes the ordering reproducible everywhere.
+  tr -d '\r' < "${out}" | grep -vE '^\s*$' | LC_ALL=C sort -o "${out}"
+  info "$(wc -l < "${out}") records -> ${out}"
+}
+
+dns_diff() { # <before> <after>  — returns non-zero if anything changed
+  if diff -u "$1" "$2" > "${SNAP_DIR}/dns.diff"; then
+    info "Cloudflare zone IDENTICAL — nothing created, modified or deleted."
+    return 0
+  fi
+  warn "Cloudflare zone CHANGED:"; cat "${SNAP_DIR}/dns.diff"
+  return 1
+}
+
+# ======================================================================= s3check plumbing
+# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
+# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
+# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
+# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
+# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
+# hold, from the network position they actually occupy.
+
+# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
+#   - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
+#     so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
+#   - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
+#     (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
+#     discovered by a silently-failing backup at 03:00;
+#   - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
+#     already broken one probe in this repo — and it cannot be made to send the checksum headers, so
+#     it cannot test for the trap at all.
+S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
+S3CHECK_RUN_ID="$(date +%s)"
+
+# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
+# must never leave a pod behind on a live cluster.
+S3CHECK_PODS=()
+s3_cleanup_pods() {
+  local p
+  for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
+    kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
+  done
+  S3CHECK_PODS=()
+}
+trap s3_cleanup_pods EXIT
+
+# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
+# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
+# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
+# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
+s3_is_placeholder() { # <endpoint>
+  case "$1" in
+    ""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
+  esac
+  return 1
+}
+
+# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
+# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
+# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
+# value with a trailing space or a stray quote is a defect, not something to paper over.
+s3_safe_token() { # <label> <value>
+  case "$2" in
+    *[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
+       That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
+  esac
+}
+
+# ---------------------------------------------------------------- the in-pod probe
+# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
+# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
+#   - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
+#   - every failure prints `S3CHECK-FAIL ...` and is counted;
+#   - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
+# The caller independently knows how many assertions this environment must produce and re-counts the
+# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
+# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
+# past; that class of bug is designed out here rather than commented about.)
+s3_probe_script() {
+  cat <<'PROBE'
+set -u
+umask 077
+A=0; F=0
+ok()   { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
+bad()  { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
+note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
+verdict() {
+  if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
+    printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
+  fi
+  printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
+}
+
+for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
+  eval "x=\${${v}-}"
+  [ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
+done
+
+export AWS_EC2_METADATA_DISABLED=true
+export AWS_PAGER=""
+export AWS_DEFAULT_REGION="${S3REGION}"
+export AWS_REGION="${S3REGION}"
+export AWS_MAX_ATTEMPTS=2
+export AWS_RETRY_MODE=standard
+export AWS_CONFIG_FILE=/tmp/s3check-aws-config
+# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
+# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
+# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
+# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
+# future CLI version.
+{
+  echo '[default]'
+  echo 'request_checksum_calculation = when_required'
+  echo 'response_checksum_validation = when_required'
+  echo 's3 ='
+  echo '    addressing_style = path'
+} > "${AWS_CONFIG_FILE}"
+
+# ---- (1) the credential loads at all -------------------------------------------------
+# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
+# byte counts and lengths leave this pod.
+case "${CRED_MODE}" in
+  file)
+    cf="${AWS_SHARED_CREDENTIALS_FILE-}"
+    if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
+      bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
+    elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
+      bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
+        the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
+    elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
+      || ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
+      bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
+    else
+      ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
+    fi
+    ;;
+  env)
+    if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
+      bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
+    else
+      ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
+    fi
+    ;;
+  *) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
+esac
+[ "${F}" -eq 0 ] || verdict     # no point talking to S3 with no credential
+
+OUT=""; RC=0
+awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
+
+detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
+
+# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
+diagnose() { # <operation label>
+  local op="$1"
+  case "${OUT}" in
+    *InvalidArgument*|*checksum*|*Checksum*)
+      bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
+        AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
+        Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
+        every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
+        backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
+        checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
+    *"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
+    *"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
+    *"Connection reset"*|*"Read timeout"*)
+      bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
+        install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
+        while no backup exists. This is the reading that matters: the workstation's own reachability
+        is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
+        pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
+        or give this environment its own endpoint." ;;
+    *NoSuchBucket*|*"(404)"*|*"Not Found"*)
+      bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
+        (R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
+    *InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
+      bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
+        different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
+    *AccessDenied*|*"(403)"*|*Forbidden*)
+      bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
+        needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
+    *) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
+  esac
+  detail
+}
+
+PAY=/tmp/s3check.payload
+GOT=/tmp/s3check.got
+head -c 4096 /dev/urandom | base64 > "${PAY}"
+[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
+
+TRAP_DONE=0
+for t in ${S3TARGETS}; do
+  B="${t%%:*}"; P="${t#*:}"
+  [ "${P}" = "${B}" ] && P=""
+  if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
+
+  # ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
+  #          WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
+  #          needs it and a correctly bucket-scoped R2 token may not grant it.
+  args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
+  [ -n "${P}" ] && args+=(--prefix "${P}/")
+  awsx "${args[@]}"
+  if [ "${RC}" -eq 0 ]; then
+    ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
+  else
+    diagnose "LIST ${B}"
+    continue      # a short assertion count is how the caller learns this target was abandoned
+  fi
+
+  # ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
+  #          nothing about whether an object can be STORED (wrong scope, checksum rejection and
+  #          read-only tokens all pass a HEAD and fail a backup).
+  awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
+  else diagnose "PUT ${B}/${K}"; continue; fi
+
+  rm -f "${GOT}"
+  awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
+  else diagnose "GET ${B}/${K}"
+       aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
+       continue; fi
+
+  sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
+  hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
+  hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
+  if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
+    ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
+  else
+    bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
+  fi
+
+  awsx s3api delete-object --bucket "${B}" --key "${K}"
+  if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
+  else diagnose "DELETE ${B}/${K}"; fi
+
+  awsx s3api head-object --bucket "${B}" --key "${K}"
+  if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
+  else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
+
+  # ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
+  #          checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
+  #          that line is LOAD-BEARING. Reported either way so the operator learns which world this
+  #          endpoint lives in before, not after, enabling velero.
+  if [ "${TRAP_DONE}" -eq 0 ]; then
+    TRAP_DONE=1
+    TK="${K}.checksum-trap-probe"
+    OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
+           aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
+    if [ "${RC}" -eq 0 ]; then
+      note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
+        present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
+      aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
+        || note "checksum probe: could NOT delete ${TK} — remove it by hand"
+    else
+      case "${OUT}" in
+        *InvalidArgument*)
+          note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
+        x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
+        checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
+        velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
+        *) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
+      esac
+    fi
+  fi
+done
+
+verdict
+PROBE
+}
+
+# ---------------------------------------------------------------- run one probe pod
+# <ns> <file|env> <secret> <space-separated bucket:prefix targets>  <endpoint> <region>
+s3_probe() {
+  local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
+  # $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
+  # would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
+  local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
+
+  # What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
+  # byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
+  # pod's own claim is never the only source of the expected count.
+  local t n=0
+  for t in ${targets}; do n=$((n+1)); done
+  local min=$((1 + 6 * n))
+
+  local volsjson="" mountsjson="" creds=""
+  case "${mode}" in
+    file)  # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
+      volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
+      mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
+      creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
+    env)   # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
+      creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
+    *) die "internal: unknown probe mode '${mode}'" ;;
+  esac
+
+  local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
+  [ -n "${b64}" ] || die "internal: probe script encoded to nothing"
+
+  # egress-class label: harmless here (the velero namespace is not an app namespace) but required by
+  # the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
+  # the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
+  # the check would report "unknown" instead of a result.
+  local overrides
+  overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
+
+  say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}'  → ${min} assertions required"
+  kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
+  S3CHECK_PODS+=("${ns}/${pod}")
+  kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
+      --overrides="${overrides}" >/dev/null 2>&1 \
+    || die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
+       That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
+
+  local i=0 phase=""
+  while [ "${i}" -lt 300 ]; do
+    phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+    case "${phase}" in Succeeded|Failed) break ;; esac
+    sleep 5; i=$((i+5))
+  done
+  case "${phase}" in
+    Succeeded|Failed) ;;
+    *) local reason
+       reason="$(kubectl -n "${ns}" get pod "${pod}" \
+                 -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
+       die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
+       NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
+  esac
+
+  local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
+  [ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
+       an empty log is the shape of a probe that never ran, not of a healthy backup target."
+  printf '%s\n' "${log}" | while IFS= read -r l; do printf '    %s\n' "${l}"; done
+
+  # ---- the four independent accounting checks that make a silent no-op impossible ----
+  local okcount failcount resline nres verd a f
+  okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
+  failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
+  nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
+  [ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
+       A missing verdict line means the probe died mid-way; it is never a pass."
+  resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
+  read -r _ verd a f <<<"${resline}"
+  a="${a#assertions=}"; f="${f#failures=}"
+  case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
+  case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
+  [ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
+       Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
+       or CNPG backups against this endpoint until they are all gone."
+  [ "${f}" = "0" ]          || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
+  [ "${failcount}" = "0" ]  || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
+  [ "${a}" -ge "${min}" ]   || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
+       (1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
+       degrade-to-a-successful-no-op this check exists to prevent."
+  [ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
+       lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
+  [ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
+  info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
+}
+
+# ============================================================================ steps
+step_preflight() {
+  say "PREFLIGHT (read-only)"
+  info "nodes Ready: $(kubectl get nodes --no-headers | awk '$2=="Ready"' | wc -l)/$(kubectl get nodes --no-headers | wc -l)"
+  info "flux controllers Running: $(kubectl -n flux-system get pods --no-headers | awk '$3=="Running"' | wc -l)"
+  # Count BOTH groups the Kyverno chart ships. Grepping only 'kyverno.io' undercounts by 2 — the
+  # policy-report types are in the wgpolicyk8s.io group — which reads as "CRDs are missing" when the
+  # pre-apply in fact succeeded. Expect 22 for kyverno 3.9.x.
+  info "kyverno CRDs present: $(kubectl get crd --no-headers 2>/dev/null | grep -cE 'kyverno\.io|wgpolicyk8s\.io' || true) (expect 22 for kyverno 3.9.x)"
+
+  local base owner filter s3 acme
+  base="$(cvar BASE_DOMAIN)"; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  filter="$(cvar EXTERNALDNS_DOMAIN_FILTER)"; s3="$(cvar S3_BACKUP_ENDPOINT)"; acme="$(cvar ACME_CLUSTER_ISSUER)"
+  say "cluster-vars guards"
+  info "BASE_DOMAIN                = ${base}"
+  info "EXTERNALDNS_TXT_OWNER_ID   = ${owner}"
+  info "EXTERNALDNS_DOMAIN_FILTER  = ${filter}"
+  info "ACME_CLUSTER_ISSUER        = ${acme}"
+  info "S3_BACKUP_ENDPOINT         = ${s3}"
+
+  # THE GUARD THAT MATTERS. Two clusters sharing one Cloudflare zone are isolated by txtOwnerId and
+  # by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the
+  # parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the
+  # single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records.
+  # preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the
+  # target guard, because that one means you are pointed at the wrong cluster. Everything else is
+  # reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting
+  # on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health
+  # report behind a condition that was, for dev, simply the long-standing status quo.
+  if [ -z "${owner}" ]; then
+    warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'.
+       Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse."
+  elif [ "${owner}" = "unset-see-cluster-vars" ]; then
+    warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply.
+       'externaldns' will refuse until a real per-environment value is set."
+  elif [ "${owner}" = "default" ]; then
+    # NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records
+    # are registered under it and cannot be renamed without orphaning them. Dying on it would make
+    # this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in
+    # a script whose whole claim is that it has none. preflight only READS, so it warns; the
+    # MUTATING path (step_externaldns) still refuses outright.
+    warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value.
+       That is correct for DEV (its records are registered under it) and WRONG for any second cluster
+       sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the
+       2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and
+       policy: sync deleted the records behind them."
+  else
+    info "owner id is distinct and non-default: OK"
+  fi
+
+  if s3_is_placeholder "${s3}"; then
+    warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
+       velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
+       so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
+       apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
+  else
+    info "S3 backup endpoint set: OK"
+  fi
+
+  # COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
+  # any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
+  # *and* CDI StorageProfiles in the same pass — but the StorageProfile CRD does not exist until the
+  # cdi-operator has reconciled that CR. On an established cluster the CRDs are long since there and
+  # this is invisible; on a COLD one it is a deadlock, and it is the same class of bug as Kyverno's
+  # 5.7 MiB CRD bundle. Report it here so `resume` does not burn 15 minutes discovering it.
+  say "CRDs that later tiers apply resources against"
+  local crd missing=0
+  for crd in storageprofiles.cdi.kubevirt.io cdis.cdi.kubevirt.io kubevirts.kubevirt.io \
+             volumesnapshotclasses.snapshot.storage.k8s.io; do
+    if kubectl get crd "${crd}" >/dev/null 2>&1; then
+      printf '    %-52s present\n' "${crd}"
+    else
+      printf '    \033[33m%-52s MISSING\033[0m\n' "${crd}"; missing=1
+    fi
+  done
+  if [ "${missing}" = "1" ]; then
+    warn "a MISSING crd above will fail platform-config's dry-run and block every tier below it.
+       CDI's own CRDs (StorageProfile, DataVolume, ...) are created by cdi-operator only AFTER it
+       reconciles the CDI CR. Check that the CDI CR is applied and cdi-operator is Available; if the
+       CRD appeared only recently this can also be a stale discovery cache in kustomize-controller,
+       which a forced reconcile clears."
+    kubectl -n cdi get cdi,deploy 2>/dev/null | head -12 || info "(no cdi namespace yet)"
+  fi
+
+  say "tiers"
+  local n
+  for n in flux-system cilium "${TIER_ORDER[@]}" velero; do
+    printf '    %-16s suspend=%-6s ready=%-6s %s\n' "$n" "$(k_susp "$n")" "$(k_ready "$n")" "$(k_msg "$n" | head -c 90)"
+  done
+
+  say "HelmReleases not Ready"
+  kubectl get helmrelease -A \
+    -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message \
+    --no-headers 2>/dev/null | grep -v ' True ' || info "(all Ready)"
+
+  # SUSPENDED releases are invisible above: a suspended HelmRelease keeps whatever Ready condition it
+  # last had, so a STALE `Ready=True` hides the fact that Flux is not managing it at all. external-dns
+  # is deliberately in this state until `standup.sh externaldns` runs.
+  say "HelmReleases SUSPENDED (Flux is not reconciling these — a stale Ready=True means nothing)"
+  kubectl get helmrelease -A \
+    -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend --no-headers 2>/dev/null \
+    | awk '$3=="true"' || true
+}
+
+step_crds() {
+  say "CRD PRE-APPLY (idempotent)"
+  # Charts whose CRD bundle exceeds kubectl's 256 KiB annotation limit set crds.install=false, so
+  # nothing installs them and `platform` can never go Ready. See bootstrap/flux/preapply-crds.sh.
+  bash "${script_dir}/flux/preapply-crds.sh"
+
+  # ---------------------------------------------------------------- the CDI cold-start deadlock
+  # SAME CLASS OF BUG, DIFFERENT OPERATOR. `platform-config` applies the CDI CR *and*
+  # platform/kubevirt/cdi-storageprofiles.yaml in ONE pass. But StorageProfile's CRD does not exist
+  # until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before
+  # applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD
+  # is never created, and platform-config can never converge. A true deadlock, invisible on an
+  # established cluster (the CRDs are long since there) and fatal on a cold one.
+  #
+  # Breaking it needs exactly one thing: get the CDI CR in. Then cdi-operator creates the remaining
+  # CDI CRDs and platform-config's next reconcile dry-runs clean. We apply ONLY the CR — the minimum
+  # — and leave every other object to Flux. The CR carries no ${} tokens, so no substitution is
+  # needed and applying it here cannot diverge from what Flux would render.
+  local cdi_cr="${repo_root}/platform/kubevirt/cdi-cr.yaml"
+  if kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1; then
+    info "storageprofiles CRD already present — CDI is past its cold start"
+  elif [ ! -f "${cdi_cr}" ]; then
+    warn "expected ${cdi_cr} and it is missing — skipping the CDI unblock"
+  else
+    say "CDI cold-start unblock: applying the CDI CR so cdi-operator can create its CRDs"
+    kubectl get ns cdi >/dev/null 2>&1 || die "namespace 'cdi' absent — the platform tier has not applied the CDI operator yet. Run 'resume' far enough to make platform Ready first."
+    kubectl apply --server-side --force-conflicts -f "${cdi_cr}"
+    local i=0
+    while [ "$i" -lt 300 ]; do
+      kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 && break
+      sleep 10; i=$((i+10))
+    done
+    kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 \
+      || die "cdi-operator did not create storageprofiles.cdi.kubevirt.io within ${i}s.
+       Inspect: kubectl -n cdi get cdi,pods,deploy — platform-config stays blocked until it exists."
+    kubectl wait --for=condition=Established --timeout=60s crd/storageprofiles.cdi.kubevirt.io >/dev/null
+    info "storageprofiles.cdi.kubevirt.io created and Established (${i}s)"
+  fi
+}
+
+step_unwedge() {
+  say "UNWEDGE STUCK HELM RELEASES"
+  # A Helm release left in uninstalling/pending-* cannot be advanced by helm-controller: it reports
+  # StateError ("unable to determine state for release with status 'uninstalling'") forever, which
+  # keeps `platform` (wait: true) not-Ready and every tier below it blocked.
+  #
+  # How prod got here: Kyverno's install timed out because its pods were crash-looping with no CRDs;
+  # install.remediation.retries:3 then fired an uninstall, and that uninstall timed out too.
+  #
+  # Pre-applied CRDs live OUTSIDE the Helm release and SURVIVE this — that is precisely why the
+  # pre-apply is a separate step and must run before this one.
+  local found=0 ns rel st
+  while read -r ns rel st; do
+    [ -n "${ns:-}" ] || continue
+    found=1
+    warn "${ns}/${rel} is '${st}'"
+    [ "${CONFIRM_DESTRUCTIVE}" = "1" ] || {
+      info "DRY RUN. This DELETES the release's Kubernetes objects and lets Flux install cold."
+      info "Re-run with --confirm-destructive to act."
+      continue
+    }
+    say "clearing ${ns}/${rel}"
+    # --no-hooks and --wait=false: the WAIT is what hung the original remediation, so do not repeat it.
+    helm uninstall "${rel}" -n "${ns}" --no-hooks --wait=false || warn "helm uninstall returned non-zero; continuing to check state"
+    if helm list -n "${ns}" --all -q 2>/dev/null | grep -qx "${rel}"; then
+      warn "${ns}/${rel} still present after uninstall — inspect manually, NOT deleting release secrets by hand."
+    else
+      info "${ns}/${rel} cleared"
+      flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile of ${ns}/${rel} did not complete in 10m"
+    fi
+  done < <(
+    for ns in $(kubectl get helmrelease -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do
+      helm list -n "${ns}" --all -o json 2>/dev/null \
+        | yq -r '.[] | select(.status | test("uninstalling|pending-install|pending-upgrade|pending-rollback")) | "'"${ns}"' " + .name + " " + .status' 2>/dev/null || true
+    done
+  )
+  [ "${found}" = "1" ] || info "no wedged releases found."
+}
+
+step_externaldns() {
+  say "EXTERNAL-DNS — SAFE UPGRADE (prove the owner id BEFORE the pod can run)"
+  # THE HAZARD: a Helm upgrade sets replicas back to the chart default (1) and applies the new args in
+  # ONE operation. There is no window in which to check the result first. If --txt-owner-id is missing
+  # the pod starts as owner `default`, claims the other cluster's TXT registry entries, and under
+  # `policy: sync` DELETES the records behind them. That is the 2026-09-04 incident.
+  #
+  # THE FIX IS TO VERIFY IN ADVANCE, NOT AFTER: render the chart with the LIVE HelmRelease values and
+  # assert the flag appears. Rendering is deterministic and touches nothing, so this converts a race
+  # into a precondition.
+  local ns=external-dns rel=external-dns
+  local owner chart ver repo vals
+  owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  [ -n "${owner}" ] && [ "${owner}" != "default" ] && [ "${owner}" != "unset-see-cluster-vars" ] \
+    || die "refusing: EXTERNALDNS_TXT_OWNER_ID is '${owner}'"
+
+  chart="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.chart}')"
+  ver="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.version}')"
+  repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' platform/externaldns/helmrepository.yaml | tr -d '"')"
+  [ -n "${repo}" ] || die "could not derive the external-dns chart repo url"
+
+  vals="${SNAP_DIR}/ed-values.yaml"
+  kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.values}' | yq -P > "${vals}"
+  info "chart=${chart} version=${ver} repo=${repo}"
+  info "live spec.values.txtOwnerId = $(yq '.txtOwnerId' "${vals}")"
+
+  say "rendering the chart with the LIVE values to prove the flag"
+  local rendered="${SNAP_DIR}/ed-rendered.yaml"
+  helm template "${rel}" "${chart}" --repo "${repo}" --version "${ver}" -n "${ns}" -f "${vals}" > "${rendered}"
+  local args; args="$(yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].args[]' "${rendered}")"
+  printf '%s\n' "${args}" | sed 's/^/      /'
+  printf '%s\n' "${args}" | grep -qx -- "--txt-owner-id=${owner}" \
+    || die "the render does NOT contain --txt-owner-id=${owner}.
+       An upgrade would start external-dns as owner 'default' and delete the other cluster's records.
+       REFUSING. Fix the chart values key before retrying."
+  info "PROVEN: the upgrade will carry --txt-owner-id=${owner}"
+
+  say "snapshotting the Cloudflare zone (baseline for the post-check)"
+  dns_snapshot "${SNAP_DIR}/dns-before.txt"
+
+  say "resuming + forcing the upgrade"
+  kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.suspend}' | grep -q true \
+    && flux resume helmrelease "${rel}" -n "${ns}" || info "(HelmRelease was not suspended)"
+  flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile did not complete in 10m"
+
+  say "verifying the LIVE deployment args"
+  local live; live="$(kubectl -n "${ns}" get deploy "${rel}" -o jsonpath='{.spec.template.spec.containers[0].args}' 2>/dev/null || true)"
+  printf '    %s\n' "${live}"
+  printf '%s' "${live}" | grep -q -- "--txt-owner-id=${owner}" \
+    || { kubectl -n "${ns}" scale deploy "${rel}" --replicas=0 || true
+         die "live args LACK --txt-owner-id=${owner}. Scaled to 0. Investigate before scaling up."; }
+  info "live deployment carries --txt-owner-id=${owner}"
+
+  say "post-check: did anything in the shared zone change?"
+  sleep 45   # external-dns --interval=1m; give it a full cycle to act before judging it safe
+  dns_snapshot "${SNAP_DIR}/dns-after.txt"
+  dns_diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-after.txt" \
+    || warn "records changed — expected ONLY additions under this cluster's own BASE_DOMAIN ($(cvar BASE_DOMAIN)).
+       Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION."
+}
+
+step_s3check() {
+  say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
+  # Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
+  # the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
+  # egress differ — and a healthy host-side reading has already misled this project once.
+  #
+  # WHAT THIS PROVES (and it fails closed on every one of them):
+  #   1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
+  #      (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
+  #   2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
+  #   3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
+  #      write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
+  #   4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
+  #
+  # WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
+  # against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
+  local ep live region
+  live="$(cvar S3_BACKUP_ENDPOINT)"
+  info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
+  if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
+    # Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
+    # backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
+    # the first evidence is silently-missing backups.
+    ep="${S3_ENDPOINT_OVERRIDE}"
+    info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
+  else
+    ep="${live}"
+  fi
+  if s3_is_placeholder "${ep}"; then
+    die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
+       cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
+       to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
+  fi
+  s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
+  case "${ep}" in
+    http://*|https://*) ;;
+    *) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
+  esac
+
+  # Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
+  # in cluster-vars silently builds a BSL with no region.
+  region="$(cvar S3_BACKUP_REGION)"
+  case "${ep}" in
+    *.r2.cloudflarestorage.com*)
+      if [ "${region}" != "auto" ]; then
+        warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
+       exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
+       velero, or the BackupStorageLocation is built with the wrong region."
+      fi
+      region=auto ;;
+    *)
+      if [ -z "${region}" ]; then
+        region="us-east-1"
+        warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
+       ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
+      fi ;;
+  esac
+  s3_safe_token S3_BACKUP_REGION "${region}"
+
+  # Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
+  # whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
+  local vbucket vprefix cpath cbucket cprefix rest
+  vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
+             platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
+  [ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
+    || die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
+  vprefix="$(cvar VELERO_BACKUP_PREFIX)"
+
+  cpath="$(cvar CNPG_BACKUP_PATH)"
+  if [ -z "${cpath}" ]; then
+    warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
+       substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
+       default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
+    cpath="s3://cnpg"
+  fi
+  case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
+  rest="${cpath#s3://}"; cbucket="${rest%%/*}"
+  if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
+  cprefix="${cprefix%/}"
+  [ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
+
+  s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
+  s3_safe_token cnpg-bucket   "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
+
+  say "what is about to be proven"
+  info "endpoint    ${ep}"
+  info "region      ${region}"
+  info "velero      bucket '${vbucket}'  prefix '${vprefix:-<bucket root>}'   (authored bucket + VELERO_BACKUP_PREFIX)"
+  info "cnpg        bucket '${cbucket}'  prefix '${cprefix:-<bucket root>}'   (from CNPG_BACKUP_PATH)"
+  info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
+
+  local ran=0 keys
+  # (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
+  if kubectl get ns velero >/dev/null 2>&1 \
+     && kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
+    # List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
+    # into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
+    # cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
+    # silently looked like "the key is missing".
+    keys="$(kubectl -n velero get secret velero-cloud-credentials \
+            -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
+    printf '%s\n' "${keys}" | grep -qx 'cloud' \
+      || die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
+       velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
+       at RUNTIME while the HelmRelease reports Ready."
+    s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
+    ran=$((ran+1))
+  else
+    warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
+       be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
+  fi
+
+  # (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
+  #     of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
+  #     ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
+  local pick cns csec
+  pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
+          | grep -- '-db-backup-credentials$' | head -n 1 || true)"
+  if [ -n "${pick}" ]; then
+    cns="${pick%%/*}"; csec="${pick#*/}"
+    s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
+    ran=$((ran+1))
+  else
+    warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
+       NOT be proven."
+  fi
+
+  [ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
+       proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
+       nothing to check'."
+
+  say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
+  info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
+  info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
+}
+
+step_resume() {
+  say "RESUME TIERS (dependency order)"
+  local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)"
+  for n in "${TIER_ORDER[@]}"; do
+    if [ "$(k_susp "$n")" = "true" ]; then
+      info "resuming $n"; flux resume kustomization "$n" --timeout=1s >/dev/null 2>&1 || true
+    else
+      info "$n already unsuspended"
+    fi
+    wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
+  done
+
+  # The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
+  # was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
+  # which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
+  # RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
+  # to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
+  if s3_is_placeholder "${s3}"; then
+    warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
+       Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
+       that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
+       it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
+  else
+    info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
+    wait_ready velero 900 || warn "velero did not converge"
+  fi
+}
+
+step_verify() {
+  say "VERIFY"
+  local base; base="$(cvar BASE_DOMAIN)"
+  info "BASE_DOMAIN = ${base}"
+  say "kustomizations"
+  kubectl get kustomization -A \
+    -o custom-columns=NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status --no-headers
+  say "certificates (must be Ready before any hostname is trusted)"
+  kubectl get certificate -A --no-headers 2>/dev/null || info "(none yet)"
+  say "pods not Running/Completed"
+  kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' || true
+  info "total pods: $(kubectl get pods -A --no-headers | wc -l)"
+  say "gateway / LoadBalancer VIPs"
+  kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true
+  say "published DNS for this cluster's own domain"
+  local owner; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
+  dns_snapshot "${SNAP_DIR}/dns-verify.txt"
+  info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)"
+
+  # Every TXT registry entry this cluster publishes must carry OUR owner id. One reading another
+  # cluster's id means we have claimed its records and `policy: sync` will delete them next cycle.
+  local foreign
+  foreign="$(grep "${base}" "${SNAP_DIR}/dns-verify.txt" | grep 'heritage=external-dns' \
+             | grep -v "external-dns/owner=${owner}" || true)"
+  if [ -n "${foreign}" ]; then
+    warn "TXT entries under ${base} NOT owned by '${owner}' — this cluster has claimed records it does not own:"
+    printf '%s\n' "${foreign}" | sed 's/^/      /'
+  else
+    info "every TXT registry entry under ${base} reads owner=${owner}: OK"
+  fi
+
+  # THE REGRESSION CHECK. Additions under our own BASE_DOMAIN are expected. ANY change to a name
+  # outside it means we have touched the other cluster's zone — the 2026-09-04 incident.
+  if [ -f "${SNAP_DIR}/dns-before.txt" ]; then
+    say "regression check against the pre-standup baseline"
+    local outside
+    outside="$(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" \
+               | grep -E '^[<>]' | grep -v "${base}" || true)"
+    if [ -n "${outside}" ]; then
+      warn "RECORDS OUTSIDE ${base} CHANGED — investigate before doing anything else:"
+      printf '%s\n' "${outside}" | sed 's/^/      /'
+    else
+      info "no record outside ${base} was added, changed or removed: OK"
+      info "added under ${base}: $(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" | grep -c '^>' || true)"
+    fi
+  else
+    warn "no ${SNAP_DIR}/dns-before.txt baseline — cannot prove the other cluster's records are untouched.
+       Take one BEFORE the next standup so this check can run."
+  fi
+}
+
+# ============================================================================ dispatch
+for s in "${STEPS[@]}"; do
+  case "$s" in
+    preflight)   step_preflight ;;
+    s3check)     step_s3check ;;
+    crds)        step_crds ;;
+    unwedge)     step_unwedge ;;
+    externaldns) step_externaldns ;;
+    resume)      step_resume ;;
+    verify)      step_verify ;;
+    all)         step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;;
+    *)           die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;;
+  esac
+done
+
+say "done: ${STEPS[*]}"

catalog/render.py

@@ -47,7 +47,8 @@ LBL = "homelab.keithjackson.dev"
 # The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
 # Every rendered manifest lands under a Flux path whose Kustomization declares
 # postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
-# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
+# reconcile time: dev renders dev.home.keithjackson.dev, prod renders home.keithjackson.dev (the
+# APEX — operator decision 2026-09-04; prod IS the apex, so there is no `prod.` label and no CNAMEs).
 # It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
 # the escape contract below reserves `$${...}` for expansions meant for a container's shell.
 # Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
@@ -606,7 +607,7 @@ def r_internet_egress(s: dict) -> dict:
 def r_httproutes(s: dict) -> list:
     """One HTTPRoute per ingressRoute, attaching host -> service port on the shared Gateway.
 
-    TLS is the Gateway's wildcard *.home.keithjackson.dev cert (no per-service cert step);
+    TLS is the Gateway's wildcard *.${BASE_DOMAIN} cert (no per-service cert step);
     ExternalDNS manages the record from the HTTPRoute (R-TLS/R-DNS, renderer-contract).
     """
     out = []
@@ -1321,11 +1322,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
       - `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
-      - `<svc>-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that
-        secret: the two environments carry DIFFERENT object-store credentials, selected in
-        the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters,
-        so both must be placed in the DB wave — the Secret has to exist before the Cluster
-        that names it.)
     Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
     Only files that actually exist in `present_files` are placed, so the plan is robust to a
     service having no bootstrap secret or no DB. Deterministic (sorted).
@@ -1353,9 +1349,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
             add(wdb, f"{db}.yaml")
             add(wdb, f"{db}-app.sops.yaml")
             add(wdb, f"{db}-backup-credentials.sops.yaml")   # CNPG Barman S3 creds (R-STATE-03)
-            # PROD half of the same credential (2026-09-07 per-env split). Placed only when
-            # the file exists, so a DB that has not been split is unaffected.
-            add(wdb, f"{db}-backup-credentials-prod.sops.yaml")
 
     for unit, cfg in EXTRA_UNITS.items():
         anchor = cfg["anchor"]
@@ -1838,7 +1831,6 @@ def _selftest() -> int:
     present = {
         "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
-        "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
         "gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
     }
@@ -1848,10 +1840,8 @@ def _selftest() -> int:
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
-          and flat.get("svc-db-backup-credentials.sops.yaml") == 0
-          and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
-          "CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret "
-          "must be in the DB's (lower) wave")
+          and flat.get("svc-db-backup-credentials.sops.yaml") == 0,
+          "CNPG DB manifest + its owner secret + its Barman backup secret must be in the DB's (lower) wave")
     check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
           "a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
     check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")

ci/l0.sh

@@ -137,44 +137,6 @@ else
   esac
 fi
 
-# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ----
-# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
-# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
-# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
-# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
-# Invalid client credentials") behind an entirely green cluster; dev had one. The check
-# asserts the declaration is self-consistent (every placeholder defined, every Secret key
-# present, names matching the reconciler's contract), that the Flux ordering is a REAL
-# dependsOn edge rather than a comment, and — by extracting the shipped script from its
-# ConfigMap and running its --selftest — that the reconciler is compare-then-write,
-# idempotent, non-logging and blast-radius-bounded.
-#
-# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance
-# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/,
-# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04
-# and the requirement-scoped selection would filter it out of exactly the change that
-# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no
-# side effects, so running it twice when 3b also selects it is not worth avoiding.
-KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}"
-log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})"
-if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then
-  skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green"
-  summary "- keycloak-client-secrets: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  kc_rc=0
-  bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$?
-  case "$kc_rc" in
-    0) ok "Keycloak client-secret declaration + reconciler passed"
-       summary "- keycloak-client-secrets: **passed**" ;;
-    3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass"
-       summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)"
-       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Keycloak client-secret declaration + reconciler FAILED"
-       summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;;
-  esac
-fi
-
 case "$rc" in
   0) ok "L0 stage complete" ;;
   3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;

ci/touched-requirements.sh

@@ -180,29 +180,6 @@ path_requirements() {
     platform/externaldns/*)    echo "R-DNS-02" ;;
     platform/velero/*)         echo "R-STATE-03" ;;
     platform/monitoring/*)     echo "R-OBS-01 R-OBS-02" ;;
-    # platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
-    # nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
-    # there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
-    # for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
-    # requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
-    platform/cnpg/*)           echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
-    # vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
-    # R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
-    # steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
-    # annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
-    # happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
-    # ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
-    # when a diff does carry the tags.
-    platform/vpn-egress/*)     echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
-    # Keycloak is the whole SSO surface: the realm import (declarative identity config),
-    # the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
-    # the client-secret incident, in which editing platform/keycloak/ attributed NO
-    # requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
-    # subset — the change that could break every OIDC login was gated by nothing.
-    # R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
-    # its off-cluster backup credential, not only the realm.
-    platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
-    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;

clusters/base/backup.yaml

@@ -34,9 +34,6 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
-  # and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
-  # overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
-  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
-  # clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
-  # velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
+  # SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
+  # environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
+  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.

clusters/base/cluster-vars.yaml

@@ -26,36 +26,49 @@ data:
   # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
   # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
   # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
-  # subdomain, so the two clusters can no longer describe the same name.
+  # suffix, so the two clusters can no longer describe the same name.
   #
-  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
-  #   prod  -> prod.home.keithjackson.dev
+  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
   #
-  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
-  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
-  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
-  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
-  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
-  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
-  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
-  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
-  BASE_DOMAIN: "home.keithjackson.dev"
+  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # string dev holds.
+  #
+  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
+  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
+  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
+  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
+  # infra/PROD-PROVISIONING.md §10.
+  #
+  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
+  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
+  BASE_DOMAIN: "dev.home.keithjackson.dev"
 
   # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
   # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
   # so it cannot be created, updated or deleted regardless of TXT ownership.
   #
-  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
-  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
-  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
-  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
-  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
   #
-  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
-  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
-  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
-  # — it can never delete another environment's records — and the fallback is to set prod's value
-  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # it — the change is not defence in depth, it is a total outage of DNS publication.
+  #
+  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
+  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
+  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
+  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
+  # both environments live in the one zone either way.
   EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
 
   # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
@@ -64,14 +77,35 @@ data:
   # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
   # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
   #
-  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
-  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
-  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
   #
-  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
-  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
-  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
-  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
+  # is now free to be chosen on its merits. Two reasons to take `dev`:
+  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
+  #     clusters silently shared it, which IS the incident). An environment whose identity is the
+  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
+  #     hold it.
+  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
+  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
+  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
+  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
+  #     race between two reconcile loops.
+  #
+  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
+  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
+  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
+  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
+  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
+  # environment's id. A new environment that forgets this knob must not silently inherit another's
+  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
+  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
+  # instead of destroying a live environment's DNS.
   EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
 
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
@@ -79,7 +113,7 @@ data:
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                             # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                             # takes the pool's first free IP, and coredns-custom answers
-                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
+                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
                                             # hairpins through the Gateway (correct SNI -> wildcard cert).
                                             # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                             # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
@@ -96,22 +130,9 @@ data:
   #
   # Substitution coverage (verify before adding a knob here): every path that consumes these declares
   # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
-  # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
-  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
-  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
-  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
-  # UNSUBSTITUTED path would ship the literal string into the cluster.
-  #
-  # ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
-  # UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
-  # live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
-  # `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
-  # tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
-  # the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
-  # `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
-  # ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
-  # the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
+  # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
+  # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
+  # ship the literal string into the cluster.
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
   #
@@ -160,23 +181,29 @@ data:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
 
-  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
-  # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
-  # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
-  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
-  # and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
-  # network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
-  # `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
-  S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
+  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
+  # Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
+  # backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
+  #
+  # OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
+  # being until a better on-prem backup solution is created". Two reasons beyond cost:
+  #   1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
+  #      pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
+  #   2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
+  #      physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
+  #      genuinely off-site.
+  # Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
+  # 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
+  # stay free and R-STATE-03 has no cost disincentive to stay proven.
+  #
+  # The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
+  # means pointing this knob back at http://192.168.1.9:9000 temporarily.
+  S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
 
-  # S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
-  # exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
-  # string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
-  # accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
-  # is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
-  # against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
-  # value. Prod overrides to `auto`.
-  S3_BACKUP_REGION: "us-east-1"
+  # S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
+  # This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
+  # MinIO and wrong for R2, which is why it is a knob now.
+  S3_BACKUP_REGION: "auto"
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
@@ -230,98 +257,38 @@ data:
   VELERO_BACKUP_SCHEDULE: "0 3 * * *"        # cron; overlays may differ
   VELERO_RETENTION: "168h"                   # dev short | prod long
 
-  # ==========================================================================================
-  # OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
-  # writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
-  # ==========================================================================================
-  # Nothing about the object layout was per-environment before this: Velero's bucket was the
-  # hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
-  # `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
-  # have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
-  # that layout and:
-  #   (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
-  #   (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
-  #       mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
-  #       dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
+  # BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
+  # other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
   #
-  # WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
-  # the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
-  # barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
-  # CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
-  # hold the same bucket name a second time would be a rotation waiting to drift — the exact class
-  # of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
-  # no two of them can contradict each other.
+  # Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
   #
-  # BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
-  # `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
-  # a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
-  # the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
-  # /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
-  # from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
-  # BSL is unchanged by this parameterization rather than merely equivalent to it.
+  #   VELERO  — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
+  #             `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
+  #             itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
+  #             therefore adopt dev's backups and delete them on schedule.
+  #   CNPG    — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
+  #             CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
+  #             so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
+  #             interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
+  #             dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
   #
-  # PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
+  # Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
+  # worth explaining, because the earlier reasoning was the opposite:
   #
-  #     s3://k3s-prod-backup/velero/…                 Velero (kopia repo + backup metadata)
-  #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG WAL + base backups
+  #   While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
+  #   backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
+  #   restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
   #
-  # The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
-  # prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
-  # renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
-  # real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
-  # bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
-  VELERO_BACKUP_BUCKET: "velero"             # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
-  VELERO_BACKUP_PREFIX: ""                   # dev: bucket root (live layout) | prod: "velero"
-  CNPG_BACKUP_PATH: "s3://cnpg"              # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
-
-  # ------------------------------------------------------------------------------------------------
-  # WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
-  # ------------------------------------------------------------------------------------------------
-  # The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
-  # it exists because the endpoint moved per-environment while the credential did not.
+  #   Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
+  #   reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
+  #   free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
+  #   colliding with a real environment's backups.
   #
-  # THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
-  # <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
-  # by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
-  # differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
-  # credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
-  # signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
-  # because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
-  # so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
-  # spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
-  # either environment's key silently breaks the other; that is the structural reason for the split,
-  # independent of dev's eventual decommission.
-  #
-  # THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
-  # base resources; nothing here forks a manifest per environment). The consumer names one:
-  #
-  #     velero:   credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
-  #     CNPG:     s3Credentials.*.name:  <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
-  #
-  # ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
-  # quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
-  # rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
-  # UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
-  # or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
-  # names are spelled out in full in each kustomization.yaml beside the resources.
-  #
-  # THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
-  # which are the objects that already exist and that the live dev cluster is running. A default
-  # naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
-  # Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
-  # CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
-  # again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
-  # Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
-  #
-  # An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
-  # not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
-  # gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
-  # "-prod" — so neither inherits it silently.
-  #
-  # ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
-  # `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
-  # `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
-  BACKUP_CREDENTIALS_SUFFIX: ""              # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
+  # *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
+  # the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
+  # one change; do not split the commit.
+  VELERO_BACKUP_PREFIX: "dev"                # dev: "dev" | prod: "prod"
+  CNPG_BACKUP_PATH: "s3://cnpg/dev"          # dev: s3://cnpg/dev | prod: s3://cnpg/prod
 
   # --- placement ---
   PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os"   # base no-op selector; prod may pin roles

clusters/dev/kustomization.yaml

@@ -5,6 +5,20 @@
 # reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
 # It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
 # on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
+#
+# ##############################################################################################
+# ###  *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) ***                 ###
+# ###                                                                                        ###
+# ###  This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex   ###
+# ###  `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url       ###
+# ###  moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a   ###
+# ###  merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
+# ###  Dev's Gitea is both its GitOps source and the push target: if the new host is not up    ###
+# ###  with a Ready cert first, dev stops reconciling and pushing cannot repair it.            ###
+# ###                                                                                        ###
+# ###  Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator        ###
+# ###  present. Merging is step D3, not step one.                                             ###
+# ##############################################################################################
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -106,11 +120,8 @@ patches:
           secretRef:
             name: sops-age
   # velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
-  # It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
-  # MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
-  # stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
-  # BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
-  # out-of-band (name only).
+  # It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
+  # decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
   - patch: |-
       apiVersion: kustomize.toolkit.fluxcd.io/v1
       kind: Kustomization
@@ -134,7 +145,24 @@ patches:
         # via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
         # secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
         # revert this url (docs/05).
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        #
+        # *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
+        # It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
+        # it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
+        # postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
+        # break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
+        # So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
+        #
+        # It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
+        # makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
+        # (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
+        # DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
+        # resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
+        # BY PUSHING — the broken thing is the path Flux uses to read pushes.
+        # ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
+        # Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
+        # the migration window, and only restore the in-cluster url after the new host is proven.
+        url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   # --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -151,33 +179,60 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
-      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
-      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
-      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
-      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
-      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
-      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
-      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
-      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
-      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      # ==========================================================================================
+      # DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
+      #
+      #   dev   -> dev.home.keithjackson.dev      (this knob; = the base default, restated)
+      #   prod  -> home.keithjackson.dev          (THE APEX — prod IS the apex, no `prod.` label,
+      #                                            and therefore no apex CNAME layer at all)
+      #
+      # *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
+      # now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
+      # an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
+      #
+      # WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
+      # lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
+      # source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
+      # DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
+      # not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
+      # BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
+      # is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
+      # in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
+      # FIRST, or park the source on the bootstrap Gitea for the window.
+      #
+      # ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
+      # ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
+      # server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
+      # split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
       - op: replace
         path: /data/BASE_DOMAIN
-        value: "home.keithjackson.dev"
-      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
-      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
-      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+        value: "dev.home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
+      # F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
+      # strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
+      # record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
+      # newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
       - op: replace
         path: /data/EXTERNALDNS_DOMAIN_FILTER
         value: "keithjackson.dev"
-      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
-      # registry under owner `default` (the external-dns chart default, which is what both clusters
-      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
-      # BY this string: any other value and it stops recognising every record it manages, then
-      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
-      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      # OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
+      # records were already in Cloudflare's TXT registry under owner `default`, so renaming would
+      # have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
+      # dev publishes a brand-new record set with no prior registry entries to recognise, so the id
+      # is chosen on its merits, and `default` has none — it is the value external-dns uses when you
+      # FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
+      # deliberate:
+      #   - dev's OLD apex records remain owned by `default`, which no environment now claims, so
+      #     nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
+      #     publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
+      #   - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
+      #     cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
+      # dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
       - op: replace
         path: /data/EXTERNALDNS_TXT_OWNER_ID
-        value: "default"
+        value: "dev"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -186,7 +241,7 @@ patches:
       #
       # Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
       # (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
-      # *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
+      # *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
@@ -225,58 +280,20 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
-      # barmanObjectStore and Velero's backupStorageLocation write to.
+      # OFF-CLUSTER BACKUP TARGET — no op here any more, deliberately.
       #
-      # DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
-      # Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
-      # operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
-      # a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
-      # would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
-      # (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
-      - op: replace
-        path: /data/S3_BACKUP_ENDPOINT
-        value: "http://192.168.1.9:9000"
-      # DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
-      # reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
-      # defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
-      # its proven restore depends on — without any manifest looking wrong.
+      # This used to pin the on-Proxmox MinIO (http://192.168.1.9:9000). As of 2026-09-05 the operator
+      # moved backups to Cloudflare R2 "for the time being until a better on-prem backup solution is
+      # created", and R2 is a single environment-INDEPENDENT endpoint: both clusters write to the same
+      # buckets and are isolated by VELERO_BACKUP_PREFIX / CNPG_BACKUP_PATH (dev vs prod), not by URL.
+      # Under R-ENV-02 a value that does not differ per environment belongs in base and must NOT be
+      # restated here — two copies of one endpoint is just a rotation waiting to drift.
       #
-      # `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
-      # ignores the value, but changing it is a needless risk against a live, working backup path.
-      # An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
-      # are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
-      # against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
-      - op: replace
-        path: /data/S3_BACKUP_REGION
-        value: "us-east-1"
-      - op: replace
-        path: /data/VELERO_BACKUP_BUCKET
-        value: "velero"
-      - op: replace
-        path: /data/VELERO_BACKUP_PREFIX
-        value: ""
-      - op: replace
-        path: /data/CNPG_BACKUP_PATH
-        value: "s3://cnpg"
-      # WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
-      # (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
-      # its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
-      # renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
-      #
-      # RESTATED EXPLICITLY even though it equals the base default, for the same reason as
-      # EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
-      # base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
-      # signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
-      # turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
-      # restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
-      #
-      # DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
-      # 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
-      # backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
-      - op: replace
-        path: /data/BACKUP_CREDENTIALS_SUFFIX
-        value: ""
+      # *** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts EMPTY, so Velero writes a
+      # fresh kopia repository and CNPG takes new base backups; the 30-day Barman window rebuilds from
+      # zero. The existing MinIO copy is NOT deleted and stays restorable — point S3_BACKUP_ENDPOINT
+      # back at http://192.168.1.9:9000 (and VELERO_BACKUP_PREFIX back to "") to reach it. Do not merge
+      # this while dev's proven restore is the only recovery path you are relying on.
       # dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT

clusters/prod/kustomization.yaml

@@ -129,14 +129,23 @@ patches:
   #
   # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
   # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
-  # Two independent reasons, both consequences of the split-horizon design:
-  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
-  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
-  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
-  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
-  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
-  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
-  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
+  # which changes WHY the blocker holds but not THAT it holds:
+  #   1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
+  #      pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
+  #      inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
+  #      BEFORE dev is retired, or prod loses its source of truth.
+  #      NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
+  #      gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
+  #      the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
+  #      cutover below a one-line change once prod actually runs Gitea.
+  #   2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
+  #      easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
+  #      ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
+  #      prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
+  #      listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
+  #      to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
+  #      remains is purely reason 1 — the Gitea is not there YET.
   #
   # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
   #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
@@ -185,25 +194,9 @@ patches:
         name: flux-system
         namespace: flux-system
       spec:
-        # CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
-        # from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
-        # (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
-        # operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
-        # laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
-        #
-        # THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
-        # renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
-        # GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
-        # too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
-        # its `main` to the exact commit prod was already running so the source swap changed nothing
-        # else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
-        # the authored value agrees. Dev hit the identical trap on 2026-09-05
-        # (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
-        #
-        # Verified before the switch: a throwaway pod INSIDE prod resolved this host through
-        # in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
-        # supplied. The workstation being able to reach it proves nothing about pod egress.
-        url: https://gitea.prod.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   - target:
@@ -218,28 +211,41 @@ patches:
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
       # ==========================================================================================
-      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
       # ==========================================================================================
-      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
-      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
-      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
-      # of which already happened:
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
+      # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
+      #
+      #   dev   -> dev.home.keithjackson.dev     (dev carries the label; clusters/dev)
+      #   prod  -> home.keithjackson.dev         (THE APEX — this knob)
+      #
+      # There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
+      # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
+      # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
+      #
+      # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
+      # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
+      # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
+      # consequences, one of which already happened:
       #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
       #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
       #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
       #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
       #      have raced each other's `_acme-challenge` TXT records on that one zone.
-      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
-      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
       #
-      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
-      #   prod  -> prod.home.keithjackson.dev    (this knob)
-      #
-      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
-      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
-      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
-      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
-      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
+      # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
+      # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
+      # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
+      #   (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
+      #   (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
+      #       (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public
+      #       DNS; that mistake was made on 2026-09-04), AND
+      #   (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
+      #       that same name set while dev's still exists counts against LE's duplicate-certificate
+      #       limit (5 per week per identical name set).
+      # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
+      # the 2026-09-04 incident with the roles reversed.
       #
       # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
       # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
@@ -247,30 +253,33 @@ patches:
       # trusts a different Keycloak than the one prod actually runs.
       - op: replace
         path: /data/BASE_DOMAIN
-        value: "prod.home.keithjackson.dev"
-      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
-      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
-      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
-      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
-      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
-      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
-      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
-      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+        value: "home.keithjackson.dev"
       - op: replace
         path: /data/EXTERNALDNS_DOMAIN_FILTER
-        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
-        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
-        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
-        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
-        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
-        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
+        # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
+        # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
+        # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
+        # record DNS Name", so prod publishes NOTHING. An agent narrowed this to
+        # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
+        # Isolation from dev does NOT come from this filter — it comes from
+        # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
+        # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
         # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
         value: "keithjackson.dev"
       # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
       # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
-      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
-      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
-      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      # its own and, under policy: sync, deleted the records behind them.
+      #
+      # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
+      # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
+      # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
+      # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
+      # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
+      # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
+      # simply fails to publish until those records are retired, which is the correct behaviour.
+      # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
+      # non-colliding sentinel. All three differ.
       - op: replace
         path: /data/EXTERNALDNS_TXT_OWNER_ID
         value: "prod"
@@ -392,91 +401,38 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
-      # ==========================================================================================
-      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
-      # ==========================================================================================
-      # Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
-      # option for prod, for two independent reasons:
-      #   1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
-      #      MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
-      #      nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
-      #      timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
-      #   2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
-      #      a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
-      #      restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
-      #
-      # THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
-      # `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
-      # exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
-      # 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
-      # ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
-      # 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
-      # cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
-      # forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
-      #
-      # ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
-      # address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
-      # (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
-      # It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
-      # there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
-      # restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
-      - op: replace
-        path: /data/S3_BACKUP_ENDPOINT
-        value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
-      # R2 requires the region string to be exactly `auto`. (`head-bucket` answers
-      # `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
-      # here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
-      # has always been signed with.
-      - op: replace
-        path: /data/S3_BACKUP_REGION
-        value: "auto"
       # ------------------------------------------------------------------------------------------
-      # OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
-      # barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
-      # They are separated by disjoint TOP-LEVEL PREFIXES instead:
+      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05).
       #
-      #     s3://k3s-prod-backup/velero/…                 Velero: kopia repo + backup metadata
-      #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG: WAL + base backups
+      # THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule
+      # stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev
+      # address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it
+      # is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and
+      # overriding it with an identical value would only create two places to change on a rotation —
+      # the exact drift this repo has been bitten by before. Isolation does NOT come from the
+      # endpoint; it comes from the two path knobs below.
       #
-      # The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
-      # obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
-      # deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
-      # per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
-      # matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
-      # Dev cannot be reached from any of these: different endpoint AND different bucket names.
+      # WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network
+      # 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN
+      # 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod
+      # egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s.
+      # Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and
+      # unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely
+      # off-site. Verify prod can reach the new endpoint before resuming velero:
+      #   bootstrap/standup.sh ... --s3-endpoint <the R2 endpoint> s3check
+      #
+      # THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was
+      # per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster
+      # used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have
+      # keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and
+      # TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman
+      # stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml.
       # ------------------------------------------------------------------------------------------
-      - op: replace
-        path: /data/VELERO_BACKUP_BUCKET
-        value: "k3s-prod-backup"
       - op: replace
         path: /data/VELERO_BACKUP_PREFIX
-        value: "velero"
+        value: "prod"
+      # Barman objects land at s3://cnpg/prod/<clusterName>/; dev's at s3://cnpg/dev/<clusterName>/.
       - op: replace
         path: /data/CNPG_BACKUP_PATH
-        value: "s3://k3s-prod-backup/barman"
-      # ------------------------------------------------------------------------------------------
-      # WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
-      # ------------------------------------------------------------------------------------------
-      # The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
-      # not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
-      # manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
-      # gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
-      # R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
-      #
-      # So each credential is now two objects, and this suffix selects prod's:
-      #     velero-cloud-credentials-prod           the R2 token in the AWS INI `cloud` shape
-      #     <db>-backup-credentials-prod            the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
-      # for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
-      # base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
-      # than four secret-name knobs deliberately — four could be updated three-quarters of the way,
-      # leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
-      # substitution cannot catch a defined-but-stale variable.
-      #
-      # THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
-      # k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
-      # touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
-      # the structural reason for the split, independent of dev's eventual decommission.
-      - op: replace
-        path: /data/BACKUP_CREDENTIALS_SUFFIX
-        value: "-prod"
+        value: "s3://cnpg/prod"
       # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.

conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh

@@ -140,16 +140,8 @@ def docs(path: Path):
         return []
 
 
-# Skip dot-DIRECTORIES (.git, .claude, .cache) while still matching dot-FILES such as
-# .sops.yaml. A bare rglob from ROOT also descends into NESTED WORKTREES —
-# .claude/worktrees/agent-*/ created for isolated agent runs — which made this check see
-# 5 CiliumEgressGatewayPolicy objects instead of the 1 authored, failing R-NET-04 for a
-# reason that has nothing to do with egress. .claude/ is gitignored (.gitignore:43), so
-# nothing under it is part of the authored surface by definition.
 TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
-                      if "_artifacts" not in p.parts
-                      and not any(part.startswith(".")
-                                  for part in p.relative_to(ROOT).parent.parts))
+                      if ".git" not in p.parts and "_artifacts" not in p.parts)
 SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
 RENDER = ROOT / "catalog" / "render.py"
 VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"

conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml

@@ -1,63 +0,0 @@
-# =============================================================================
-# L0 STATIC — Keycloak client secrets are declared consistently, ordered by a REAL
-# Flux edge, and healed by a reconciler whose behaviour is proven offline
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# R-IDP-03 (identity config is declarative) and R-IDP-04 (IdP secrets SOPS-encrypted)
-# are both implemented through ONE mechanism that FAILS OPEN: the KeycloakRealmImport
-# resolves each client's secret from a `(env:NAME)` placeholder mapped to a key of the
-# SOPS-encrypted keycloak-realm-clients Secret. When the operator cannot resolve a
-# placeholder it stores the LITERAL placeholder text as the client secret AND THE IMPORT
-# JOB STILL REPORTS SUCCESS — then rests at Done=True / Started=False and never re-runs.
-#
-# Measured 2026-09-06: prod had all 12 clients holding the literal placeholder (every
-# OIDC login failed with "Invalid client or Invalid client credentials"); dev had exactly
-# one (oauth2-proxy), because that Secret key was added AFTER dev's import completed.
-# Declarative config that silently does not apply is not declarative config.
-#
-# GIVEN the repository as authored,
-# THEN  every confidential client declares a placeholder, every placeholder is defined,
-#       resolves to the keycloak-realm-clients Secret, and names a key that EXISTS in the
-#       SOPS file under the `<clientId>-client-secret` contract the reconciler derives
-#       from; public clients declare no secret; the Flux Kustomization applying the Secret
-#       is a transitive `dependsOn` ancestor (wait: true) of the ones applying the import
-#       and the reconciler — a real edge, not a comment; the reconciler is reconciled from
-#       BOTH cluster entrypoints, mounts only what it needs, carries no ServiceAccount
-#       token and contains no dollar sign; and its own `--selftest` (extracted from the
-#       shipped ConfigMap and executed) proves compare-then-write, idempotence,
-#       no-secret-logging, the endpoint allow-list and fail-closed behaviour.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
-# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
-# Test wraps the SAME script.
-#
-# NOT proven here: that the LIVE realm's stored secrets match git (that is
-# tests/l2-behavioral/keycloak-client-secrets-match-git), that the SOPS ciphertext
-# decrypts to a usable value (key PRESENCE is offline-checkable; plaintext is not), or
-# that the CronJob's schedule fires.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: keycloak-client-secret-reconciler
-  labels:
-    conformance.home.k3s/R-IDP-03: "true"
-    conformance.home.k3s/R-IDP-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
-    conformance.home.k3s/layer: "L0"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Every OIDC client's secret is declared once, resolvable, ordered behind a real Flux
-      dependency, and continuously reconciled — so an unresolved import placeholder can
-      neither be merged nor silently persist in the realm.
-spec:
-  steps:
-    - name: audit-declaration-ordering-and-reconciler
-      description: >-
-        Assert the realm import, the SOPS Secret and the reconciler agree; that the Flux
-        ordering is a real dependsOn edge; and run the reconciler's own selftest.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh

@@ -1,529 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — keycloak-client-secret-reconciler
-# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
-#                   — spec docs/15-identity-and-access.md)
-#         R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
-#                   — spec docs/15-identity-and-access.md)
-#
-# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
-# ---------------------------------------------------------------
-# platform/keycloak/keycloak-realm-import.yaml declares 12 confidential OIDC clients
-# whose secrets are `(env:NAME)` import placeholders, mapped by spec.placeholders to
-# keys of the SOPS-encrypted keycloak-realm-clients Secret. Measured on both clusters
-# on 2026-09-06:
-#
-#   * when the Keycloak Operator cannot resolve a placeholder it stores the LITERAL
-#     placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS;
-#   * the KeycloakRealmImport then rests at Done=True / Started=False and NEVER
-#     re-runs, so it never self-heals.
-#
-# Consequence: on prod all 12 clients held the literal placeholder and every OIDC
-# login on the cluster failed ("Invalid client or Invalid client credentials"); on dev
-# exactly one (oauth2-proxy) was broken, because that Secret key was added AFTER dev's
-# import had already completed and a completed import is never re-run.
-#
-# Nothing in the repo asserted the declaration was self-consistent, and nothing
-# corrected the realm afterwards. This check is the static half of the fix (the
-# reconciler CronJob is the runtime half, and this check proves that too).
-#
-# GIVEN the repository as authored,
-# THEN  (1) DECLARATION IS COMPLETE: every confidential client in the realm import
-#           declares a secret placeholder; every placeholder it uses is defined in
-#           spec.placeholders; every placeholder resolves to the keycloak-realm-clients
-#           Secret; and every referenced key EXISTS in the SOPS file. A client whose
-#           key is missing is exactly the dev failure, and it is now unmergeable.
-#       (2) THE NAMING CONTRACT HOLDS: each key is `<clientId>-client-secret`. That is
-#           what lets the reconciler derive the client set from the mounted Secret with
-#           no second declaration — so it must be enforced, not assumed. No orphan
-#           placeholder, and no orphan Secret key (an orphan key would make every
-#           reconciler run fail with NOT FOUND).
-#       (3) PUBLIC CLIENTS DECLARE NO SECRET (the `kubernetes` client is PKCE-public;
-#           giving it a secret would be a silent downgrade).
-#       (4) ORDERING IS A REAL FLUX EDGE, NOT A COMMENT: the Flux Kustomization that
-#           renders the Secret is a transitive `dependsOn` ancestor — with `wait: true`
-#           — of the one that renders the realm import and the reconciler. The repo's
-#           hard-won rule: a dependency that is only DOCUMENTED is not a dependency.
-#       (5) THE RECONCILER IS WIRED AND SAFE: it is reconciled from BOTH cluster
-#           entrypoints, mounts exactly the two Secrets it needs at the paths its code
-#           defaults to, carries no ServiceAccount token, and contains no dollar sign
-#           (it ships inside a substituted Flux path, where an unescaped dollar-brace
-#           expansion is rewritten before any container runs).
-#       (6) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
-#           script from the ConfigMap and executes its `--selftest`, which drives the
-#           real code paths against an in-process stub Keycloak and asserts
-#           compare-then-write, idempotence (zero writes on a converged realm),
-#           no-secret-logging, the endpoint allow-list (users/groups/roles are
-#           unreachable) and fail-closed behaviour. What CI proves is byte-for-byte
-#           what ships.
-#   AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
-#           blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
-#           platform/keycloak/ to these requirement IDs, and an L2 test exists that
-#           audits the LIVE realm. An unreferenced test is precisely the "looks applied
-#           and isn't" failure this repo keeps hitting.
-#
-# NOT proven here (named honestly, in the style of the sibling L0 checks):
-#   - that the LIVE realm's stored secrets match git. That is cluster state; it is
-#     conformance/tests/l2-behavioral/keycloak-client-secrets-match-git.
-#   - that the SOPS ciphertext decrypts to a usable secret. Key PRESENCE is checkable
-#     offline; the plaintext is not, and must not be.
-#   - that the CronJob's image can be pulled or its schedule fires.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
-SOPS_FILE="platform/keycloak/keycloak-realm-clients.sops.yaml"
-RECON_FILE="platform/keycloak/client-secret-reconciler.yaml"
-
-# Honest SKIP (never green) if the thing under test is absent entirely.
-for f in "$IMPORT_FILE" "$SOPS_FILE" "$RECON_FILE"; do
-  if [ ! -f "${ROOT}/${f}" ]; then
-    echo "check SKIPPED: ${f} absent — nothing to audit" >&2
-    exit 3
-  fi
-done
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-"""Static audit of the Keycloak client-secret declaration, its Flux ordering, and the
-reconciler that heals it. PyYAML (the same dependency tests/l0-static/no-plaintext-secrets
-and least-privilege-rbac already take) parses the manifests; lib/status.py is IMPORTED for
-the Flux graph walk rather than forked, so there is one definition of "what a Flux path
-renders" in the harness."""
-from __future__ import annotations
-
-import re
-import subprocess
-import sys
-import tempfile
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
-sys.path.insert(0, str(ROOT / "conformance" / "lib"))
-import status as st                     # noqa: E402 — reuse the walk, don't fork it
-
-IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
-SOPS_FILE = "platform/keycloak/keycloak-realm-clients.sops.yaml"
-RECON_FILE = "platform/keycloak/client-secret-reconciler.yaml"
-SECRET_NAME = "keycloak-realm-clients"
-ADMIN_SECRET = "keycloak-admin-credentials"
-KEY_SUFFIX = "-client-secret"
-SECRETS_MOUNT = "/etc/keycloak-realm-clients"
-DOLLAR = chr(36)
-
-FAILURES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def load(rel):
-    return list(yaml.safe_load_all((ROOT / rel).read_text()))
-
-
-# ================================================================= (1)-(3) declaration
-realm_import = next((d for d in load(IMPORT_FILE)
-                     if d and d.get("kind") == "KeycloakRealmImport"), None)
-check(realm_import is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document found")
-if realm_import is None:
-    print("CHECK FAIL: cannot parse the realm import", file=sys.stderr)
-    raise SystemExit(1)
-
-placeholders = realm_import["spec"].get("placeholders") or {}
-clients = realm_import["spec"]["realm"].get("clients") or []
-check(len(clients) > 0, f"{IMPORT_FILE}: parsed zero clients — the file shape changed and "
-                        f"this check would silently audit nothing")
-
-# The operator's placeholder syntax, spelled without a literal dollar so this file is
-# itself safe to grep for one. Matches `<dollar>(env:NAME)`.
-PLACEHOLDER_RE = re.compile(re.escape(DOLLAR) + r"\(env:([A-Za-z_][A-Za-z0-9_]*)\)")
-
-used_placeholders: set[str] = set()
-declared_pairs: dict[str, str] = {}          # clientId -> secret key in the Secret
-
-for c in clients:
-    cid = c.get("clientId", "<unnamed>")
-    secret = c.get("secret")
-    public = bool(c.get("publicClient"))
-    if public:
-        # (3) a PKCE public client must not carry a secret at all.
-        check(secret is None,
-              f"{IMPORT_FILE}: client '{cid}' is publicClient but declares a secret — a "
-              f"public client holds no secret; declaring one is a silent downgrade")
-        continue
-    # (1) every confidential client must declare one.
-    check(isinstance(secret, str) and secret,
-          f"{IMPORT_FILE}: confidential client '{cid}' declares NO secret. The operator "
-          f"would generate a random one that exists nowhere in git, so nothing could "
-          f"ever reconcile it and the consuming app's mirrored credential would be wrong")
-    if not isinstance(secret, str):
-        continue
-    m = PLACEHOLDER_RE.fullmatch(secret.strip())
-    check(m is not None,
-          f"{IMPORT_FILE}: client '{cid}' has a secret that is not an (env:NAME) "
-          f"placeholder. A literal secret in git violates R-IDP-04/R-SEC-01; the only "
-          f"permitted form is a placeholder resolved from the {SECRET_NAME} Secret")
-    if m is None:
-        continue
-    name = m.group(1)
-    used_placeholders.add(name)
-    # (1) the placeholder must be DEFINED. An undefined one is stored literally by the
-    #     operator, with a successful-looking import — the whole incident.
-    ph = placeholders.get(name)
-    check(ph is not None,
-          f"{IMPORT_FILE}: client '{cid}' uses placeholder '{name}' but spec.placeholders "
-          f"does not define it. The operator stores the LITERAL placeholder as the client "
-          f"secret and STILL reports the import successful (measured 2026-09-06)")
-    if ph is None:
-        continue
-    ref = (ph or {}).get("secret") or {}
-    check(ref.get("name") == SECRET_NAME,
-          f"{IMPORT_FILE}: placeholder '{name}' reads Secret '{ref.get('name')}', not "
-          f"'{SECRET_NAME}'. The reconciler mounts only {SECRET_NAME}, so a second source "
-          f"would be declared but never healed")
-    key = ref.get("key")
-    check(isinstance(key, str) and key,
-          f"{IMPORT_FILE}: placeholder '{name}' names no Secret key")
-    if not isinstance(key, str):
-        continue
-    # (2) the naming contract the reconciler derives its client set from.
-    check(key == cid + KEY_SUFFIX,
-          f"{IMPORT_FILE}: client '{cid}' reads Secret key '{key}', but the reconciler "
-          f"derives the clientId from the key name, so it MUST be "
-          f"'{cid}{KEY_SUFFIX}'. Rename the key (and the SOPS entry) or the reconciler "
-          f"would look up a client that does not exist")
-    declared_pairs[cid] = key
-
-# (2) no orphan placeholder: a defined-but-unused placeholder is dead config that will
-#     quietly rot out of sync with the Secret.
-for name in sorted(set(placeholders) - used_placeholders):
-    check(False,
-          f"{IMPORT_FILE}: spec.placeholders defines '{name}' but no client uses it — "
-          f"dead declaration; remove it or wire it to the client that needs it")
-
-# ---- the SOPS Secret: keys are cleartext in a SOPS-encrypted file; values are not.
-sops_doc = next((d for d in load(SOPS_FILE) if d and d.get("kind") == "Secret"), None)
-check(sops_doc is not None, f"{SOPS_FILE}: no Secret document found")
-sops_keys = set((sops_doc or {}).get("stringData") or {})
-check(len(sops_keys) > 0,
-      f"{SOPS_FILE}: parsed zero stringData keys — the file shape changed and this check "
-      f"would silently audit nothing")
-check((sops_doc or {}).get("metadata", {}).get("name") == SECRET_NAME,
-      f"{SOPS_FILE}: the Secret is not named {SECRET_NAME}")
-
-for cid, key in sorted(declared_pairs.items()):
-    # (1) THE DEV FAILURE, made unmergeable: a client declared in the import whose key
-    #     is absent from the Secret imports "successfully" with a literal placeholder.
-    check(key in sops_keys,
-          f"{SOPS_FILE}: client '{cid}' needs key '{key}' and it is NOT in the Secret. "
-          f"The realm import would still report SUCCESS and store the literal placeholder "
-          f"as that client's secret — this is exactly how dev's oauth2-proxy broke")
-
-# (2) no orphan Secret key: the reconciler treats every `<x>-client-secret` key as a
-#     client that MUST exist, and fails loudly if it does not. Catch it here instead.
-for key in sorted(k for k in sops_keys if k.endswith(KEY_SUFFIX)):
-    cid = key[: -len(KEY_SUFFIX)]
-    check(cid in declared_pairs,
-          f"{SOPS_FILE}: key '{key}' has no matching client in {IMPORT_FILE}. The "
-          f"reconciler would look up client '{cid}', not find it, and fail every run")
-
-# ===================================================== (4) ordering is a real Flux edge
-# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`; a doc
-# is an entrypoint only if it carries the kustomize.toolkit apiVersion, `kind:
-# Kustomization` and a spec-level `path:`. Overlay `patch: |-` bodies carry Kustomizations
-# WITHOUT a path and are correctly skipped (they are patches, not entrypoints).
-_DEPENDSON_RE = re.compile(r"^(\s*)dependsOn:\s*$", re.M)
-_DEP_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
-_WAIT_RE = re.compile(r"^\s{2,6}wait:\s*true\s*(?:#.*)?$", re.M)
-
-
-def depends_on(doc: str) -> list[str]:
-    m = _DEPENDSON_RE.search(doc)
-    if not m:
-        return []
-    indent = len(m.group(1))
-    out: list[str] = []
-    for raw in doc[m.end():].splitlines():
-        if raw.strip() == "" or raw.lstrip().startswith("#"):
-            continue
-        if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
-            break                                    # dedented out of the list
-        nm = _DEP_NAME_RE.match(raw)
-        if nm:
-            out.append(nm.group(1))
-    return out
-
-
-kustomizations: dict[str, dict] = {}
-for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
-    text = yfile.read_text()
-    if st.FLUX_API not in text:
-        continue
-    for doc in st._DOC_SPLIT_RE.split(text):
-        if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
-            continue
-        pm = st._PATH_RE.search(doc)
-        if not pm:
-            continue
-        nm = st._NAME_RE.search(doc)
-        name = nm.group(1) if nm else "?"
-        kustomizations[name] = {
-            "file": str(yfile.relative_to(ROOT)),
-            "path": pm.group(1),
-            "dependsOn": depends_on(doc),
-            "wait": bool(_WAIT_RE.search(doc)),
-        }
-
-check(len(kustomizations) > 0,
-      "no Flux Kustomization with a spec.path found under clusters/ — the parser or the "
-      "repo layout changed; this check would silently audit nothing")
-
-
-def direct_render(path_spec: str) -> set[str]:
-    """Files a Flux `spec.path` renders ITSELF — a kustomize walk that deliberately does
-    NOT expand nested Flux Kustomizations.
-
-    lib/status.py's reconciled_files() follows every `spec.path` it discovers, which is
-    right for "is this file applied to the cluster at all" but wrong for OWNERSHIP: the
-    root Kustomization's path is clusters/<env>, so a transitive walk makes the root the
-    owner of every file in the repo and the ordering assertion below becomes vacuous
-    (measured: it reported the root as an owner of all three keycloak files). `resources:`
-    parsing itself is st.kustomize_resources, not forked.
-    """
-    rel = path_spec.lstrip("./") or "."
-    out: set[str] = set()
-    seen: set[str] = set()
-
-    def walk(d: str) -> None:
-        if d in seen:
-            return
-        seen.add(d)
-        kfile = ROOT / d / "kustomization.yaml"
-        if not kfile.is_file():
-            return
-        out.add(str(kfile.relative_to(ROOT)))
-        for item in st.kustomize_resources(kfile):
-            if "://" in item:
-                continue                       # remote base: not repo state
-            target = (ROOT / d / item).resolve()
-            try:
-                trel = target.relative_to(ROOT).as_posix()
-            except ValueError:
-                continue                       # outside the repo
-            if target.is_dir():
-                walk(trel)
-            elif target.is_file():
-                out.add(trel)
-
-    if (ROOT / rel).is_dir():
-        walk(rel)
-    return out
-
-
-rendered_by = {n: direct_render(k["path"]) for n, k in kustomizations.items()}
-
-
-def owners(rel_file: str) -> list[str]:
-    return sorted(n for n, files in rendered_by.items() if rel_file in files)
-
-
-def ancestors(name: str, seen=None) -> set[str]:
-    """Transitive dependsOn closure of `name` (its dependencies, direct and indirect)."""
-    seen = seen if seen is not None else set()
-    for dep in kustomizations.get(name, {}).get("dependsOn", []):
-        if dep in seen:
-            continue
-        seen.add(dep)
-        ancestors(dep, seen)
-    return seen
-
-
-secret_owners = owners(SOPS_FILE)
-import_owners = owners(IMPORT_FILE)
-recon_owners = owners(RECON_FILE)
-
-check(bool(secret_owners),
-      f"{SOPS_FILE} is rendered by NO Flux Kustomization — the client secrets are in git "
-      f"but never applied")
-check(bool(import_owners),
-      f"{IMPORT_FILE} is rendered by NO Flux Kustomization")
-# (5) the reconciler must actually be reconciled, or it is decoration.
-check(bool(recon_owners),
-      f"{RECON_FILE} is rendered by NO Flux Kustomization — the healer would sit in git "
-      f"and never run, which is the 'looks applied and isn't' failure it exists to end")
-
-for consumer_file, consumer_owners in ((IMPORT_FILE, import_owners),
-                                       (RECON_FILE, recon_owners)):
-    for co in consumer_owners:
-        for so in secret_owners:
-            if so == co:
-                continue          # same Kustomization: one atomic apply, ordering moot
-            anc = ancestors(co)
-            check(so in anc,
-                  f"ORDERING IS ONLY DOCUMENTED: '{co}' renders {consumer_file} but does "
-                  f"not (transitively) dependsOn '{so}', which renders the "
-                  f"{SECRET_NAME} Secret. Without that edge the consumer can reconcile "
-                  f"before the secret exists, and the Keycloak Operator's response to a "
-                  f"missing placeholder is to store it LITERALLY and report success")
-            if so in anc:
-                check(kustomizations[so]["wait"],
-                      f"'{so}' (which renders the {SECRET_NAME} Secret) does not set "
-                      f"wait: true, so '{co}' can proceed before it is applied")
-
-# ------ both environments must reconcile the reconciler, not just dev.
-for env in ("dev", "prod"):
-    entry = f"clusters/{env}"
-    if not (ROOT / entry).is_dir():
-        continue
-    applied, _ = st.reconciled_files(ROOT, entry)
-    check(RECON_FILE in applied,
-          f"{RECON_FILE} is not reachable from the {entry} entrypoint — the healer would "
-          f"exist on one cluster only, and prod is where all 12 clients were broken")
-    check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
-
-# ============================================ (5) the reconciler manifest is safe + wired
-recon_docs = [d for d in load(RECON_FILE) if d]
-cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
-cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
-check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
-check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
-
-script = ""
-if cm is not None:
-    data = cm.get("data") or {}
-    check(len(data) == 1,
-          f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
-          f"(found {sorted(data)}) — a second copy would be untested by this gate")
-    script = next(iter(data.values()), "")
-    check(len(script.splitlines()) > 50,
-          f"{RECON_FILE}: the extracted script is implausibly short "
-          f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
-          f"selftest below would prove nothing")
-    # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
-    # postBuild.substituteFrom, so envsubst rewrites it before any container runs. The
-    # script is written with NO dollar sign at all, so the hazard is structural rather
-    # than escaped and no future edit has to reason about escaping.
-    check(DOLLAR not in script,
-          f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside a "
-          f"SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
-          f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
-          f"never needs one; chr(36) spells it where a literal is genuinely required")
-
-if cj is not None:
-    pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
-    ctr = pod["containers"][0]
-    vols = {v["name"]: v for v in pod.get("volumes", [])}
-    mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
-    secret_vols = [v for v in vols.values()
-                   if (v.get("secret") or {}).get("secretName") == SECRET_NAME]
-    check(len(secret_vols) == 1,
-          f"{RECON_FILE}: the CronJob must mount the {SECRET_NAME} Secret — that mount IS "
-          f"the desired state it reconciles towards")
-    if secret_vols:
-        mp = mounts.get(secret_vols[0]["name"], {}).get("mountPath")
-        check(mp == SECRETS_MOUNT,
-              f"{RECON_FILE}: {SECRET_NAME} is mounted at {mp}, but the script defaults "
-              f"to {SECRETS_MOUNT} and no env var overrides it")
-    cm_vols = [v for v in vols.values()
-               if (v.get("configMap") or {}).get("name") == (cm or {}).get(
-                   "metadata", {}).get("name")]
-    check(len(cm_vols) == 1,
-          f"{RECON_FILE}: the CronJob does not mount the reconciler ConfigMap, so it "
-          f"would run nothing")
-    env_srcs = {e["name"]: e for e in ctr.get("env", [])}
-    for var in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
-        ref = (env_srcs.get(var, {}).get("valueFrom") or {}).get("secretKeyRef") or {}
-        check(ref.get("name") == ADMIN_SECRET,
-              f"{RECON_FILE}: {var} must come from the {ADMIN_SECRET} Secret "
-              f"(got {ref.get('name')!r}) — the admin credential is never inlined")
-    for var in ("KEYCLOAK_BASE_URL", "KEYCLOAK_REALM"):
-        check(env_srcs.get(var, {}).get("value"),
-              f"{RECON_FILE}: {var} is not set; the script fails closed without it, but "
-              f"the CronJob should not ship broken")
-    # R-SEC-02: it talks only to Keycloak's REST API and needs no Kubernetes API rights.
-    check(pod.get("automountServiceAccountToken") is False,
-          f"{RECON_FILE}: the reconciler needs NO Kubernetes API access (its inputs are "
-          f"mounted Secrets) — set automountServiceAccountToken: false")
-    check(cj["spec"].get("concurrencyPolicy") == "Forbid",
-          f"{RECON_FILE}: set concurrencyPolicy: Forbid so a run stalled on an "
-          f"unavailable Keycloak is not lapped by the next tick")
-    img = ctr.get("image", "")
-    check(re.search(r":[0-9]", img),
-          f"{RECON_FILE}: container image {img!r} is not pinned to an explicit version")
-
-# ============================== (6) the reconciler's BEHAVIOUR, proven by its own selftest
-if script:
-    with tempfile.TemporaryDirectory() as td:
-        p = Path(td) / "reconcile.py"
-        p.write_text(script)
-        proc = subprocess.run([sys.executable, str(p), "--selftest"],
-                              capture_output=True, text=True, timeout=180)
-        tail = (proc.stdout + proc.stderr).strip().splitlines()
-        check(proc.returncode == 0,
-              "the reconciler's --selftest FAILED (compare-then-write / idempotence / "
-              "no-secret-logging / endpoint allow-list / fail-closed):\n        "
-              + "\n        ".join(tail[-12:]))
-        if proc.returncode == 0:
-            print("  reconciler --selftest: " + (tail[-1] if tail else "(no output)"))
-
-# ======================================================= (7) this check is load-bearing
-L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
-GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
-           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
-check("keycloak-client-secret-reconciler/check.sh" in L0,
-      "ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
-      "requirement-scoped conformance selection can filter this check out of a gate run, "
-      "so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
-      "it. An unreferenced test is the failure this whole change exists to end")
-check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
-      "ci/gate.sh does not set the merge block when the l0 stage fails")
-check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
-      "ci/touched-requirements.sh has no platform/keycloak/* entry in its path->requirement "
-      "map, so editing the realm import or this reconciler attributes NO requirement and "
-      "the L2 keycloak tests are filtered out of the merge gate's touched subset")
-l2 = ROOT / "conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml"
-check(l2.is_file(),
-      "the LIVE-realm counterpart (conformance/tests/l2-behavioral/"
-      "keycloak-client-secrets-match-git) is missing — this static check cannot see a "
-      "realm that drifted after apply, which is the whole failure mode")
-if l2.is_file():
-    check("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in l2.read_text(),
-          "the L2 test does not assert the reconciler's audit sentinel, so it could pass "
-          "on a Job that never ran the audit")
-
-# ------------------------------------------------------------------------------ report
-print(f"realm import: {len(clients)} client(s), {len(declared_pairs)} confidential with a "
-      f"declared secret, {len(placeholders)} placeholder(s)")
-print(f"{SECRET_NAME} Secret: {len(sops_keys)} key(s) in git (names only; values stay "
-      f"SOPS-encrypted)")
-print("flux ordering: " + ", ".join(
-    f"{f.split('/')[-1]} <- {owners(f)}" for f in (SOPS_FILE, IMPORT_FILE, RECON_FILE)))
-
-print()
-if not FAILURES:
-    print(f"check OK: keycloak-client-secret-reconciler — {CHECKS} assertions "
-          f"(R-IDP-03, R-IDP-04)")
-    raise SystemExit(0)
-print(f"CHECK FAIL (keycloak-client-secret-reconciler): {len(FAILURES)} failed, "
-      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
-for f in FAILURES:
-    print(f"  FAIL {f}", file=sys.stderr)
-raise SystemExit(1)
-PY

conformance/tests/l2-behavioral/keycloak-client-secrets-match-git/chainsaw-test.yaml

@@ -1,212 +0,0 @@
-# =============================================================================
-# L2 BEHAVIORAL — every OIDC client's STORED secret equals the one git declares
-#                                                   (R-IDP-02, R-IDP-04)   [RUNNABLE]
-# Spec: docs/15 identity-and-access; ADR-0014.
-# Impl: platform/keycloak/client-secret-reconciler.yaml (the CronJob + its script).
-#
-# WHY THIS TEST EXISTS
-# --------------------
-# The KeycloakRealmImport resolves each confidential client's secret from a `(env:NAME)`
-# placeholder mapped to a key of the SOPS-encrypted keycloak-realm-clients Secret. When
-# the Keycloak Operator cannot resolve a placeholder it stores the LITERAL placeholder
-# text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; the CR then rests
-# at Done=True / Started=False and never re-runs. Measured 2026-09-06: prod had all 12
-# clients holding the literal placeholder — every OIDC login on the cluster failed with
-# "Invalid client or Invalid client credentials" — and dev had one (oauth2-proxy), whose
-# Secret key was added after dev's import had already completed.
-#
-# Every SIGNAL the cluster offered was green: the import Job succeeded, the CR read
-# Done=True, the Keycloak pods were healthy, and the existing keycloak-config-as-code
-# test (which asserts Done=True) PASSED throughout. The only observable was a human
-# failing to log in. This test is that observable, automated: it compares STORED against
-# DECLARED for every client and fails on any difference, so the class cannot sit
-# undetected for months again.
-#
-# GIVEN the `homelab` realm and the SOPS-encrypted keycloak-realm-clients Secret,
-# WHEN  the reconciler's script is run in --audit mode (READ-ONLY: it cannot write, and
-#       its endpoint allow-list makes users/groups/roles unreachable),
-#   ->  THEN every declared client's stored secret matches git, and none holds an
-#       unresolved import placeholder. Emits KEYCLOAK_CLIENT_SECRETS_AUDIT_OK.
-#
-# WHY IT RUNS THE SHIPPED SCRIPT rather than re-implementing the comparison: a detector
-# that duplicates the healer's logic drifts from it. This audits with the same code, in
-# the same mode the L0 gate selftests, so a passing audit is evidence about the healer
-# too. It runs as a Job IN the keycloak namespace (not in the ephemeral chainsaw
-# namespace) because the desired state is the mounted keycloak-realm-clients Secret and
-# the admin credential is keycloak-admin-credentials — both namespace-local, which is
-# also why neither this Job nor the CronJob needs any Kubernetes API access.
-#
-# chainsaw v0.2.12: Job + status assert; the leftover-Job delete keeps a re-run from
-# colliding with an immutable Job left by a previous failure.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: keycloak-client-secrets-match-git
-  labels:
-    conformance.home.k3s/R-IDP-02: "true"
-    conformance.home.k3s/R-IDP-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-04"
-    conformance.home.k3s/layer: "L2"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Every OIDC client's stored secret equals the SOPS-declared one and none holds an
-      unresolved import placeholder — the observable that was missing when all 12 prod
-      clients were broken behind a green realm import.
-spec:
-  timeouts:
-    apply: 60s
-    assert: 5m
-    exec: 5m
-  steps:
-    - name: given-the-reconciler-is-reconciled-onto-the-cluster
-      description: >-
-        The audit Job runs the CronJob's own script from its ConfigMap; if Flux has not
-        applied the reconciler, there is nothing to audit WITH and that is the failure.
-      try:
-        - assert:
-            resource:
-              apiVersion: v1
-              kind: ConfigMap
-              metadata:
-                name: keycloak-client-secret-reconciler
-                namespace: keycloak
-        - assert:
-            resource:
-              apiVersion: batch/v1
-              kind: CronJob
-              metadata:
-                name: keycloak-client-secret-reconciler
-                namespace: keycloak
-              spec:
-                concurrencyPolicy: Forbid
-    - name: then-every-stored-client-secret-matches-git
-      try:
-        - script:
-            content: |
-              kubectl -n keycloak delete job keycloak-client-secret-audit \
-                --ignore-not-found --wait=true
-        - apply:
-            resource:
-              apiVersion: batch/v1
-              kind: Job
-              metadata:
-                name: keycloak-client-secret-audit
-                namespace: keycloak
-                labels:
-                  app.kubernetes.io/name: keycloak-client-secret-audit
-                  app.kubernetes.io/part-of: conformance
-              spec:
-                backoffLimit: 0
-                activeDeadlineSeconds: 240
-                template:
-                  metadata:
-                    labels:
-                      app.kubernetes.io/name: keycloak-client-secret-audit
-                  spec:
-                    restartPolicy: Never
-                    automountServiceAccountToken: false
-                    securityContext:
-                      runAsNonRoot: true
-                      runAsUser: 65532
-                      runAsGroup: 65532
-                      seccompProfile:
-                        type: RuntimeDefault
-                    containers:
-                      - name: audit
-                        image: python:3.13.15-alpine3.23
-                        command:
-                          - python3
-                          - /opt/reconciler/reconcile-client-secrets.py
-                          # READ-ONLY. --audit never issues a write on any path; the
-                          # script's --selftest proves that (L0 gate).
-                          - --audit
-                        env:
-                          - name: KEYCLOAK_BASE_URL
-                            value: http://keycloak-service.keycloak.svc.cluster.local:8080
-                          - name: KEYCLOAK_REALM
-                            value: homelab
-                          - name: PYTHONDONTWRITEBYTECODE
-                            value: "1"
-                          - name: PYTHONUNBUFFERED
-                            value: "1"
-                          - name: KEYCLOAK_ADMIN_USER
-                            valueFrom:
-                              secretKeyRef:
-                                name: keycloak-admin-credentials
-                                key: admin-user
-                          - name: KEYCLOAK_ADMIN_PASSWORD
-                            valueFrom:
-                              secretKeyRef:
-                                name: keycloak-admin-credentials
-                                key: admin-password
-                        volumeMounts:
-                          - name: script
-                            mountPath: /opt/reconciler
-                            readOnly: true
-                          - name: client-secrets
-                            mountPath: /etc/keycloak-realm-clients
-                            readOnly: true
-                        securityContext:
-                          allowPrivilegeEscalation: false
-                          readOnlyRootFilesystem: true
-                          capabilities:
-                            drop:
-                              - ALL
-                        resources:
-                          requests:
-                            cpu: 10m
-                            memory: 48Mi
-                          limits:
-                            memory: 128Mi
-                    volumes:
-                      - name: script
-                        configMap:
-                          name: keycloak-client-secret-reconciler
-                          defaultMode: 0444
-                      - name: client-secrets
-                        secret:
-                          secretName: keycloak-realm-clients
-                          defaultMode: 0400
-        - assert:
-            resource:
-              apiVersion: batch/v1
-              kind: Job
-              metadata:
-                name: keycloak-client-secret-audit
-                namespace: keycloak
-              status:
-                succeeded: 1
-        # The Job's exit status already encodes the verdict, but the sentinel is what the
-        # run LEDGER records (conformance/runs/*/run.log): a JUnit alone cannot tell a real
-        # audit from a Job that succeeded without auditing anything.
-        - script:
-            content: |
-              set -u
-              out="$(kubectl -n keycloak logs job/keycloak-client-secret-audit \
-                       --tail=-1 2>&1 || true)"
-              echo "$out"
-              echo "$out" | grep -q 'KEYCLOAK_CLIENT_SECRETS_AUDIT_OK' \
-                || { echo "FAIL: the audit did not report every client secret matching git"; exit 1; }
-              echo "$out" | grep -q 'UNRESOLVED IMPORT PLACEHOLDER' \
-                && { echo "FAIL: a client still holds the literal import placeholder"; exit 1; }
-              echo KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK
-            check:
-              (contains($stdout, 'KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK')): true
-      catch:
-        - describe:
-            apiVersion: batch/v1
-            kind: Job
-            namespace: keycloak
-            name: keycloak-client-secret-audit
-        - podLogs:
-            namespace: keycloak
-            selector: app.kubernetes.io/name=keycloak-client-secret-audit
-            tail: 60
-      finally:
-        - script:
-            content: |
-              kubectl -n keycloak delete job keycloak-client-secret-audit \
-                --ignore-not-found --wait=false

conformance/tests/l2-behavioral/keycloak-config-as-code/chainsaw-test.yaml

@@ -11,18 +11,6 @@
 #
 # Goes GREEN once Keycloak is up + the realm client-secret hand-off (keycloak-realm-clients) is filled
 # so the placeholder resolves and the import completes. chainsaw v0.2.12: script + block contains check.
-#
-# NOT PROVEN HERE — READ THIS BEFORE TRUSTING A GREEN RUN (added 2026-09-06).
-# Done=True means the import JOB EXITED 0. It does NOT mean the placeholders resolved: when the
-# operator cannot resolve one it stores the LITERAL `(env:...)` text as the client secret and the
-# import still reports success, then rests at Done=True / Started=False and never re-runs. THIS
-# TEST PASSED THROUGHOUT the 2026-09-06 outage in which all 12 prod clients held the literal
-# placeholder and every OIDC login on the cluster failed. Done=True is a necessary, badly
-# insufficient condition.
-# The sufficient check is tests/l2-behavioral/keycloak-client-secrets-match-git, which compares
-# each client's STORED secret against the SOPS-declared one; the static half is
-# tests/l0-static/keycloak-client-secret-reconciler. Do not "simplify" by folding those into this
-# test: the whole lesson is that the import's own success signal is not evidence.
 # =============================================================================
 apiVersion: chainsaw.kyverno.io/v1alpha1
 kind: Test

infra/BACKUP-R2.md

@@ -0,0 +1,315 @@
+# Off-cluster backup on Cloudflare R2 — operator procedure
+
+**Owner:** WS-BACKUP · **Serves:** R-STATE-03 (DR restore), R-IDP-05 (Keycloak DB), R-ENV-02 (one
+environment-agnostic path) · **Status:** authored and staged, **NOT switched on**.
+
+> **Read this first.** Velero-against-R2 and CNPG/barman-against-R2 are **UNPROVEN in this repo**.
+> Nothing here claims they work. What *is* proven is in [§7](#7-what-is-proven-and-what-is-not).
+> R-STATE-03 is currently PROVEN **against MinIO only** and must be re-proven against R2 before this
+> migration can be called done.
+
+## Why R2 at all
+
+The off-cluster target used to be a MinIO on the Proxmox host (`http://192.168.1.9:9000`). Two things
+killed it, both recorded on 2026-09-05:
+
+1. **Prod physically cannot reach it.** Prod is on the SOHO network `192.168.2.0/24`, which is
+   pfSense's **WAN**; the MinIO sits inside pfSense's LAN `192.168.1.0/24`. Measured *from a pod inside
+   prod* — the workstation's own reachability proves nothing about pod egress and has misled this
+   project once already — `curl http://192.168.1.9:9000/minio/health/live` timed out after 15 s.
+   Reaching it would need a WAN→LAN rule that does not exist.
+2. **Dev's "off-cluster" backup was not off-cluster.** That MinIO runs on the *same Proxmox host* as
+   dev's VMs, so it shared a failure domain with the thing it was backing up.
+
+R2 needs no firewall change, is genuinely off-site, and is already trusted with the OpenTofu remote
+state (`infra/tofu/REMOTE-STATE.md`) — the same account, the same kind of token. It is explicitly an
+interim decision: *"that's where backups will go for the time being until a better on-prem backup
+solution is created."*
+
+---
+
+## 1. Cloudflare: buckets, token, account id
+
+All of this is in the Cloudflare dashboard; none of it touches the cluster.
+
+1. **R2 → Create bucket** → `velero`. Location hint: whatever is nearest; jurisdiction default.
+2. **R2 → Create bucket** → `cnpg`.
+   Both names are fixed by the manifests: `bucket: velero` in `platform/velero/helmrelease.yaml`, and
+   `cnpg` via `CNPG_BACKUP_PATH` (`s3://cnpg/dev`, `s3://cnpg/prod`). Use exactly those names or change
+   the manifests — do not improvise.
+3. **R2 → Manage API Tokens → Create API Token**
+   - Permission: **Object Read & Write** (not "Admin Read & Write" — the cluster never needs to create
+     or delete buckets, and a token that can is a token that can delete your backups).
+   - **Specify bucket(s):** `velero` **and** `cnpg`. One token scoped to both — the manifests already
+     assume one shared key, and `s3check` proves both buckets with it.
+   - Copy the **Access Key ID** and **Secret Access Key**. They are shown **once**. Put them in the
+     password manager before you do anything else.
+4. **Account id** — 32 hex characters, visible in the dashboard URL and in the R2 S3 endpoint the
+   token page shows you.
+
+   **This repo already knows it.** `infra/tofu/backend.r2.hcl` commits
+   `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com` for the OpenTofu state bucket,
+   with the (correct) note that an account id is not a credential. **If the backup buckets live in the
+   same Cloudflare account as the tofu state — they should; there is no reason for a second account —
+   then the endpoint is already known and you do not need to look anything up:**
+
+   ```
+   https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+   ```
+
+   Confirm it matches the endpoint on the token page. If it does not, you have two accounts and the
+   `velero`/`cnpg` buckets are in the other one — use that one's id, and say so in the DECISIONS-LOG.
+
+The endpoint form is always `https://<account-id>.r2.cloudflarestorage.com` — **no bucket in the
+hostname** (path-style; see [§6](#6-r2-specific-gotchas)).
+
+---
+
+## 2. Repo values to set
+
+Only **one** non-secret value is missing, and it is not in a secret file:
+
+| File | Key | From | To |
+|---|---|---|---|
+| `clusters/base/cluster-vars.yaml` | `S3_BACKUP_ENDPOINT` | `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` | `https://<account-id>.r2.cloudflarestorage.com` |
+
+Already authored, **do not change**: `S3_BACKUP_REGION: "auto"`, `VELERO_BACKUP_PREFIX` (dev `dev` /
+prod `prod`), `CNPG_BACKUP_PATH` (`s3://cnpg/dev` / `s3://cnpg/prod`), and `checksumAlgorithm: ""` on
+the Velero BackupStorageLocation.
+
+The placeholder is deliberately loud, and the guards match it as a **substring** — `bootstrap/standup.sh`
+refuses to probe it and refuses to resume `velero` while it is there.
+
+---
+
+## 3. Credentials: four Secrets, one command
+
+The R2 key is read by **four** committed SOPS Secrets, in **two** different shapes. No manifest changes
+are needed — only the encrypted values change:
+
+| File | Consumer | Shape |
+|---|---|---|
+| `platform/velero/velero-cloud.sops.yaml` | Velero (`velero-cloud-credentials`, ns `velero`) | `stringData.cloud` — an **AWS INI file**: `[default]` + `aws_access_key_id=` / `aws_secret_access_key=` |
+| `platform/keycloak/keycloak-db-backup-credentials.sops.yaml` | CNPG `keycloak-db` (ns `keycloak`) | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+| `apps/wave-0/nextcloud-db-backup-credentials.sops.yaml` | CNPG `nextcloud-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+| `apps/wave-0/semaphore-db-backup-credentials.sops.yaml` | CNPG `semaphore-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
+
+A Secret is namespace-scoped and a Pod reads only its own namespace's, which is why there are four
+copies of one key rather than one.
+
+### The one command
+
+```bash
+./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
+```
+
+It prompts for the two values (never echoed, never in shell history, never written to a file), writes
+them into all four Secrets with `sops set` — which encrypts in place, so **no decrypted copy of any
+Secret is ever created** — then decrypts each file back and proves all four now hold the same key by
+comparing 32-bit fingerprints. It never prints a credential.
+
+There is deliberately **no plaintext template to fill in and encrypt**. The
+`cp x.example x.yaml && $EDITOR && sops -e -i` pattern leaves a plaintext Kubernetes Secret sitting in
+the working tree, one `git add -A` away from being in git forever; `sops set` removes that window
+entirely. If you would rather edit by hand, `sops platform/velero/velero-cloud.sops.yaml` decrypts to
+`$EDITOR` and re-encrypts on save — also with no plaintext on disk — but then you must repeat it four
+times and keep them identical yourself.
+
+To check the four files agree without changing anything:
+
+```bash
+./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
+```
+
+### What is NOT changed
+
+`platform/backup/minio-credentials.sops.yaml` keeps the **MinIO** key. That is not an oversight — it
+is what you need to read the pre-migration backups ([§8](#8-going-back-to-minio)). Its
+`.yaml.example` sibling documents the MinIO shape and stays accurate for the MinIO target.
+
+Review the diff before committing: you should see **only** `ENC[...]` blobs and `lastmodified`/`mac`
+change in four files. If a key name or a `metadata.name` moved, stop.
+
+---
+
+## 4. Prove the target BEFORE merging
+
+```bash
+K=infra/ansible/.cache/dev.kubeconfig        # or prod.kubeconfig
+./infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig $K --expect-server https://192.168.2.50:6443 \
+  --s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
+```
+
+`--s3-endpoint` probes a **candidate** endpoint instead of the live `cluster-vars` value, so the target
+is proven before it is authored. Run it against **both** clusters — prod's egress is the whole reason
+this migration exists, and it is a different network position from dev's.
+
+`s3check` runs a throwaway pod per credential and, for each, proves:
+
+1. the committed credential **loads and authenticates**, in the exact shape its consumer reads it
+   (Velero's INI file mounted from the `cloud` key; CNPG's discrete `ACCESS_KEY_ID`/`ACCESS_SECRET_KEY`);
+2. the bucket **exists and is listable** under this environment's own prefix (`ListObjectsV2`, the
+   operation Velero's BSL sync and Barman's WAL scan both perform);
+3. **PUT → GET → byte-compare (sha256 + length) → DELETE → confirm-gone** of one throwaway object at
+   `<prefix>/.s3check/<pod>.probe` — a genuine write, not a HEAD. A HEAD passes with a read-only token
+   and with R2 rejecting every write;
+4. whether the **R2 checksum trap** ([§6](#6-r2-specific-gotchas)) is present, probed on purpose by
+   repeating the PUT with checksums *enabled*, and named explicitly either way.
+
+It **fails closed**: a pod that cannot be created, an empty log, a missing verdict line, a short
+assertion count, an OK-line count that disagrees with the claimed count, or any single failed
+assertion all exit non-zero. There is no "reachability unknown, carrying on". Each failure names its
+own cause and fix — wrong key, unscoped token, missing bucket, no pod egress, checksum rejection.
+
+It writes and deletes exactly one object per bucket and leaves no pod behind. It applies nothing and
+does not enable `velero`.
+
+---
+
+## 5. The verification sequence — and it ends in a RESTORE
+
+**A backup that has never been restored is not a backup.** R-STATE-03 is PROVEN against MinIO; that
+proof does not transfer to a different object store with a different S3 implementation.
+
+1. **`s3check` on dev, then on prod.** Both must pass. (§4)
+2. Merge the endpoint + the four re-encrypted Secrets. Let Flux reconcile.
+3. **Resume `velero`** — `bootstrap/standup.sh … resume` does it once the endpoint is no longer a
+   placeholder, or `flux resume kustomization velero`.
+4. `velero backup-location get` → `default` **Available**. *Available only means the BSL could be
+   read;* it is not a backup.
+5. `velero backup create --from-schedule daily` → wait for **Completed**, then
+   `velero backup logs <name>`. Watch specifically for `PodVolumeBackup` failures: the PVC-data path
+   is kopia, a **different S3 client** from the one the BSL uses (see §7), so metadata can succeed
+   while volume data fails.
+6. **Restore.** `velero restore create --from-backup <name> --include-namespaces <scratch-ns>` into a
+   *scratch* namespace, and verify the PVC contents actually arrived — not just that the Restore says
+   Completed. This is the R-STATE-03 re-proof.
+7. **CNPG PITR.** Confirm WAL is arriving:
+   `kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.conditions}'` (look for
+   `ContinuousArchiving`), and that objects exist under `s3://cnpg/<env>/keycloak-db/`. Then do the
+   restore that matters: a `Cluster` with `bootstrap.recovery` from the barman object store into a
+   scratch namespace, and check the recovered database actually contains data. The existing
+   `conformance/tests/l2-behavioral/keycloak-db-dr` and `l3-fault-injection/dr-restore` tests are the
+   harness for this; they must be re-run **against R2**, and their previous PROVEN state records MinIO,
+   not R2.
+8. Only after 6 and 7 pass may this migration be described as done, and the ledger updated. Until
+   then the honest status is "R2 accepts objects; recovery from R2 is untested".
+
+---
+
+## 6. R2-specific gotchas
+
+**`region: auto`.** R2 accepts nothing else, and S3 clients require *some* region. Authored as
+`S3_BACKUP_REGION: "auto"`. This repo already learned it — `infra/tofu/backend.tf` carries
+`region = "auto"  # R2 ignores region, but the S3 backend requires a value.`
+
+**`checksumAlgorithm: ""` — do not delete this line.** `velero-plugin-for-aws`'s own
+`backupstoragelocation.md` for **v1.14.2** (the pinned version) documents:
+
+> `checksumAlgorithm` — The checksum algorithm to use for uploading objects to S3. Supported values
+> "CRC32", "CRC32C", "SHA1", "SHA256". If the value is set as empty string "", no checksum will be
+> calculated and attached to the request headers. **Optional (defaults to "CRC32")**
+
+So *omitting* the key means CRC32 **is** sent: `x-amz-sdk-checksum-algorithm` +
+`x-amz-checksum-crc32`. This repo has already measured that R2 rejects AWS checksums — that is exactly
+why `infra/tofu/backend.tf` needs `skip_s3_checksum = true  # R2 rejects AWS trailing (streaming)
+checksums.` for the state bucket. The plugin's own compatibility table lists `checksumAlgorithm=""` as
+the remedy for the same failure on Ceph S3, Backblaze B2 and IBM COS. If backups start failing with
+**HTTP 400 `InvalidArgument`** after a plugin bump, look here first. `s3check` probes for this on
+purpose and names it.
+
+**Path-style.** `s3ForcePathStyle: "true"` is authored and correct for R2 as well as MinIO. Never put
+the bucket in the hostname.
+
+**No bucket auto-creation.** Velero does not create buckets and the token deliberately cannot. A
+missing bucket is a `NoSuchBucket` at runtime; `s3check` catches it first.
+
+**Token scope is per-bucket.** A token scoped only to `velero` authenticates fine and then 403s on
+`cnpg` — the two consumers fail independently and quietly. `s3check` proves both.
+
+**Both clusters share one endpoint.** Isolation is by path only (`VELERO_BACKUP_PREFIX`,
+`CNPG_BACKUP_PATH`), because R2 is one environment-independent URL. Those two knobs are load-bearing:
+without them prod would adopt and TTL-delete dev's Velero backups, and both clusters' identically-named
+CNPG Clusters would interleave their WAL into one Barman stream, destroying PITR for both.
+
+---
+
+## 7. What is proven, and what is not
+
+**Proven (2026-09-05, WS-BACKUP, against the live dev cluster and its MinIO at
+`http://192.168.1.9:9000`):**
+
+- the rewritten `s3check` itself: 7 assertions per credential path, both credential shapes, from
+  in-cluster pods — credential load, bucket list, PUT/GET/byte-compare/DELETE/confirm-gone, no object
+  and no pod left behind;
+- its failure paths: the `TBD` placeholder is refused; an unreachable endpoint (`:9099`) produces a
+  named no-egress failure and exit 1, not a pass;
+- MinIO accepts a checksum-bearing PUT, i.e. the trap is a *non-*MinIO problem, and
+  `checksumAlgorithm: ""` is harmless there.
+
+**Established from the pinned versions (documentation and image inspection, not measured against R2):**
+
+- `velero-plugin-for-aws` **v1.14.2** pairs with Velero **v1.18.x** (upstream compatibility table) and
+  defaults `checksumAlgorithm` to `CRC32` — so the authored `""` is load-bearing, not decoration.
+- That plugin's "Non-AWS S3 compatible provider known issues with plugin v1.14.x" table **does not
+  mention Cloudflare R2 at all** — neither as working nor as broken. There is no upstream statement to
+  lean on in either direction.
+- **The PVC-data path is a different S3 client.** `checksumAlgorithm` is a BackupStorageLocation
+  *object-store plugin* setting. Velero's File System Backup uploads through the unified repository's
+  kopia S3 backend (`pkg/repository/udmrepo/kopialib/backend/s3.go` in Velero v1.18.0), which
+  references only endpoint/region-style options and has **no checksum knob at all**. So R2
+  compatibility for backup *metadata* and for *volume data* are two separate questions, and only step
+  5 of §5 answers the second.
+- **CNPG/barman is likely clear of the checksum trap, by version.** Inspected inside the live
+  `ghcr.io/cloudnative-pg/postgresql:16.4` image on dev: `barman-cloud-wal-archive 3.11.1`,
+  `botocore 1.35.60`, `boto3 1.35.60`. That botocore predates the 1.36 line in which botocore began
+  attaching CRC32 request checksums by default (the change that introduced
+  `AWS_REQUEST_CHECKSUM_CALCULATION`), so this image should not send them.
+  **Forward risk:** barman-cloud exposes **no** `checksumAlgorithm` equivalent. If the CNPG image is
+  ever bumped to one carrying botocore ≥ 1.36, WAL archiving to R2 could start failing with 400
+  `InvalidArgument` and there is no config knob to turn it off — only `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`
+  in the Pod environment, which CNPG does not currently set. Treat a CNPG image bump as a change that
+  must be re-verified against R2.
+
+**Not proven, at all:**
+
+- that Velero can write a usable backup to R2 (metadata or kopia volume data);
+- that CNPG/barman can archive WAL to R2 and that a `bootstrap.recovery` from R2 succeeds;
+- that a restore from R2 yields correct data. R-STATE-03's PROVEN state is **MinIO-only**.
+
+---
+
+## 8. Going back to MinIO
+
+The MinIO copy is **not deleted** and stays restorable. To read pre-migration backups:
+
+1. `clusters/base/cluster-vars.yaml` → `S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"`
+2. `VELERO_BACKUP_PREFIX` → `""` (the MinIO backups are at the **bucket root**, not under `dev/`) and
+   `CNPG_BACKUP_PATH` → `s3://cnpg` (Barman objects are at `s3://cnpg/<clusterName>/`).
+3. Restore the MinIO credential into the four Secrets. `platform/backup/minio-credentials.sops.yaml`
+   still holds it: `sops -d --extract '["stringData"]["ACCESS_KEY_ID"]'` it and feed the two values to
+   `platform/backup/set-r2-credentials.sh` (the script cares about the *shape*, not the provider).
+4. `S3_BACKUP_REGION` may stay `auto` — MinIO ignores the region.
+5. **Dev only.** Prod cannot reach that MinIO at all (§Why R2); pointing prod back is not an option.
+
+Reverting for a restore and then returning to R2 costs another PITR reset. Prefer restoring from MinIO
+into a *scratch* namespace on dev with a second, temporary BackupStorageLocation over flipping the
+whole cluster's endpoint back and forth.
+
+---
+
+## 9. Merging this resets dev's PITR window
+
+Say it out loud before merging, because it is not recoverable by retrying:
+
+- Dev's R2 prefix starts **empty**. Velero writes a **fresh kopia repository** — the first backup is a
+  full upload of every included PVC, and no earlier restore point exists in R2.
+- Every CNPG Cluster takes **new base backups** (`ScheduledBackup … immediate: true`), and the 30-day
+  Barman retention window (`retentionPolicy: "30d"`) rebuilds **from zero**. For roughly a month, dev's
+  point-in-time recovery reaches back only as far as the merge.
+- The MinIO copy still holds the old window (§8), but reaching it means reverting the endpoint — so it
+  is a deliberate excursion, not a fallback that happens automatically.
+
+**Do not merge this while dev's proven MinIO restore is the only recovery path you are relying on.**
+The same applies to prod the day it first backs up anywhere.

infra/PHASE1-STATUS.md

@@ -102,12 +102,10 @@ check runs instead of silently skipping.
 > `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
 > `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
 > **dev remains the default**, so every existing invocation means what it always meant. Inside the
-> container `KUBECONFIG` is a chain of a tiny per-invocation selector file over a merged config
-> carrying both clusters as contexts **`k3s-dev`** and **`k3s-prod`** (never `default`), regenerated
-> each run by [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
-> `infra/ansible/.cache/clusters.kubeconfig`. The merged file deliberately carries **no**
-> `current-context`: it is shared by every container, and stamping the current cluster into it made
-> concurrent `CLUSTER=dev` and `CLUSTER=prod` runs silently swap clusters mid-command. Naming matters: k3s writes *both* source kubeconfigs
+> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
+> **`k3s-prod`** (never `default`), regenerated each run by
+> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
+> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
 > with context, cluster and user all called `default`, and kubectl's merge keeps the first
 > duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
 > believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never

infra/PROD-PROVISIONING.md

@@ -10,6 +10,11 @@ hardware until the blockers in §7 are closed and `make -C ci gate` is green.
 > **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
 > knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).
 
+> **Looking for the commands?** This document is the *decision record* — why prod is built the way it
+> is, what was measured, and which hypotheses are dead. The **executable sequence** lives in
+> [`PROD-STANDUP.md`](PROD-STANDUP.md): every command, in order, driven by `bootstrap/standup.sh`.
+> Start there; come back here for the reasoning behind any step.
+
 Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
 The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
 call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
@@ -1172,7 +1177,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
 | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
 | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
 | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
-| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
+| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real. Prod's value was inverted the same day to the APEX `home.keithjackson.dev`, with dev moving to `dev.home.keithjackson.dev` — see §10a, which is authored but NOT yet applied. |
 
 - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
       overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1260,6 +1265,45 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
             `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
             mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
             override anywhere, so dev resolves `clusters/dev` exactly as before.
+- [x] **The Kyverno CRDs had to be applied BY HAND, and nothing in the repo did it** — AUTHORED
+      2026-09-04 (`bootstrap/flux/preapply-crds.sh`), **not yet run against prod**. This is a hard
+      **prerequisite of a reconcilable cluster**, not a nice-to-have: `platform/kyverno/admission/
+      helmrelease.yaml` sets `crds: {install: false}` — deliberately, because the CRD bundle is
+      **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured 2026-09-04 with `helm template
+      kyverno kyverno --repo https://kyverno.github.io/kyverno/ --version 3.9.x --include-crds`), far
+      past kubectl's 256 KiB `last-applied-configuration` annotation limit, so Helm's client-side apply
+      fails `metadata.annotations: Too long` and **stalls** the HelmRelease. The consequence nobody
+      had automated: with CRD install off, **nothing installs those CRDs at all**. On a cold cluster
+      Kyverno's Deployments come up with no API types, `platform` never goes Ready, and all 9
+      downstream Kustomizations stall behind it. Dev only ever had them because the command was once
+      run in somebody's shell — the exact undocumented manual step this project exists to delete.
+      - **The fix, in git:** `bootstrap/flux/bootstrap.sh` now runs `./preapply-crds.sh` as step
+        **[2/4]** — after `flux install`, before `kubectl apply -k ${FLUX_ENV_PATH}`. It renders the
+        chart, keeps only `kind: CustomResourceDefinition` documents, applies them
+        `--server-side --force-conflicts` (no annotation, so no size limit), then waits for
+        `condition=Established`. The chart name, version constraint, repo URL and namespace are
+        **derived from that same HelmRelease/HelmRepository** — there is no second copy of the version
+        to drift, and a Renovate bump of the one pinned line carries this step with it.
+      - **No environment fork (R-ENV-02):** the script names no environment and takes no overlay
+        input; it reads repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step, so
+        `roles/flux_bootstrap` needed **no change** — it already calls `bootstrap.sh`, the single
+        entrypoint.
+      - **It cannot silently skip.** Three guards, and deliberately no opt-out flag: a derived
+        coordinate that is empty/`null` aborts; a render that yields **0** CRDs aborts (an empty
+        stream would otherwise `kubectl apply` cleanly and exit 0 — the silent-success failure mode
+        that has bitten this repo repeatedly); and every applied CRD must reach `Established`.
+      - **Toolchain:** needs `helm` and `yq` on PATH in addition to `flux`/`kubectl` — both are baked
+        into `infra/devcontainer` (`HELM_VERSION=3.16.0`, `YQ_VERSION=4.44.6`), and `bootstrap.sh`
+        now fails at step 0 if either is missing. It also needs **outbound network to
+        `kyverno.github.io`** from wherever the play runs (the Ansible role delegates to localhost).
+      - **PM to verify in the container** (the authoring agent ran it only with a stubbed `kubectl`,
+        never against a cluster): pinned `helm` 3.16.0 renders the same 22 CRDs, and the run is a
+        clean no-op the second time.
+      - **Unblocking the already-bootstrapped prod cluster** does not need a full re-bootstrap:
+        ```bash
+        # inside infra/devcontainer, KUBECONFIG=.cache/prod.kubeconfig
+        bash bootstrap/flux/preapply-crds.sh
+        ```
 - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
       the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
 
@@ -1272,7 +1316,11 @@ single-server-etcd window, and avoids a later member-add dance.
 1. Per-node §1–§5 on each of the three boxes.
 2. §7 repo work lands on `main` with the gate green.
 3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
-4. `flux bootstrap ... --path=clusters/prod`.
+4. `flux bootstrap ... --path=clusters/prod` — i.e. `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod`,
+   which runs `bootstrap/flux/bootstrap.sh` with `FLUX_ENV_PATH=clusters/prod`. That script's step
+   **[2/4]** pre-applies the 22 Kyverno CRDs server-side; **without it `platform` cannot go Ready**
+   and the 9 tiers below it never reconcile (§7). It needs `helm` + `yq` on PATH and outbound network
+   to the chart repo.
 5. Conformance against prod (§9).
 
 ## 9. Exit criteria (Phase 7)
@@ -1283,12 +1331,117 @@ single-server-etcd window, and avoids a later member-add dance.
       specifically.
 - [ ] Monitoring and alerts live on prod.
 - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
-      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
-- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
-      distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
-      prod's own Keycloak realm + per-service client secrets.
+      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10a
+      "The L2 tests hardcode the apex".
+- [ ] **Prod serves the apex DNS/TLS identity** and dev's records are provably untouched (§10/§10a):
+      distinct `EXTERNALDNS_TXT_OWNER_ID` (`prod` vs `dev`), prod wildcard cert
+      `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
+      records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.
 
-## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
+## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator
+
+Prod's `platform` tier was unsuspended after the Kyverno CRDs were pre-applied (see §7). Result:
+**8 of 9 HelmReleases Ready, 76/76 pods Running** — cert-manager, CNPG, Cilium, Longhorn,
+kube-prometheus-stack, Loki, Alloy, external-dns. `platform` itself is **not** Ready, and the 10
+downstream tiers remain blocked behind it. Two distinct problems, both needing an operator decision.
+
+### The CRD pre-apply itself: WORKED, and is now proven against a live API server
+
+22 CRDs applied server-side and all reached `Established`; a second run was a clean no-op (`rc=0`),
+so the idempotency claim is proven, not asserted. Reproduced under the **pinned** toolchain
+(helm 3.16.0, yq 4.44.6) matching an independent helm 3.20.0 run byte-for-byte: `3.9.x` → 3.9.0,
+22 CRDs, 5,636,155 bytes.
+
+**The CRDs were the real cause of a 4½-hour crash loop.** Kyverno's four Deployments had **55
+restarts each** and stabilised within minutes of the CRD apply — its controllers were coming up with
+no API types to watch. That is the failure mode §7 predicted, observed.
+
+### BLOCKER 1 — Kyverno's Helm release is stuck in `uninstalling`
+
+Diagnosis chain, from the HelmRelease conditions:
+
+```
+Released:   InstallFailed   — Helm install failed ...: context deadline exceeded
+Remediated: UninstallFailed — Helm uninstall remediation ... timed out waiting for the condition
+Ready:      StateError      — unable to determine state for release with status 'uninstalling'
+```
+
+The install timed out **because** the pods were crash-looping without CRDs. `install.remediation.
+retries: 3` then triggered an uninstall, and that uninstall itself timed out — leaving
+`sh.helm.release.v1.kyverno.v1` with `status=uninstalling`. helm-controller cannot proceed from that
+state, so `platform` (which has `wait: true`) never goes Ready and all 10 tiers below it stay blocked.
+
+**The Deployments are healthy and at the right chart version** (kyverno-3.9.0), so this is purely a
+stale Helm bookkeeping record, not a broken workload.
+
+**Remediation requires a destructive command and was deliberately NOT run unattended:**
+
+```
+helm uninstall kyverno -n kyverno --no-hooks --wait=false   # clear the stuck release
+flux reconcile helmrelease kyverno -n kyverno --force        # let Flux install cold, CRDs now present
+```
+
+This is safe on prod specifically because prod carries no real workloads yet, and a clean cold
+install is the stronger outcome: it *proves* the bootstrap path this repo now automates. Note the
+`failurePolicy: Fail` webhooks (`kyverno-policy-*`, `kyverno-exception-*`, `kyverno-cleanup-*`,
+`kyverno-global-context-*`) gate only **Kyverno's own** resource types, so a brief absence does not
+block ordinary workload admission. The pre-applied CRDs live **outside** the Helm release and
+survive the uninstall, which is exactly why the pre-apply had to be a separate step.
+
+### BLOCKER 2 — prod's external-dns is an ARMED footgun: the installed release predates the fix
+
+**This is the more dangerous of the two.** Prod's live external-dns Deployment renders:
+
+```
+--policy=sync  --registry=txt  --domain-filter=keithjackson.dev  --provider=cloudflare
+```
+
+**and NO `--txt-owner-id`.** Absent that flag external-dns falls back to the chart default `default`
+— which is precisely the owner under which **all 36 of dev's TXT registry entries are recorded**.
+Scaling this Deployment to 1 would make prod claim dev's records and, under `policy: sync`,
+**delete them: the 2026-09-04 incident, exactly.**
+
+The repo is CORRECT and is not the problem. `platform/externaldns/helmrelease.yaml` sets
+`txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"`, prod's applied `cluster-vars` resolves it to `prod`, and
+the **live HelmRelease `spec.values` carries `txtOwnerId: "prod"`**. Verified against the real chart:
+`helm template external-dns --version 1.21.1` with `txtOwnerId: prod` renders `--txt-owner-id=prod`,
+so the values key is right and an upgrade *would* fix it.
+
+**The fault is Helm/Flux state, not authoring.** `helm get values external-dns` shows the INSTALLED
+release carries `policy` and `domainFilters` but **no `txtOwnerId`** — it predates `c8c1d29`. Yet
+helm-controller reports `Ready=True`, `observedGeneration=3`, and a history entry whose
+`configDigest` **equals** `lastAttemptedConfigDigest`, i.e. **it believes it has already deployed the
+current config.** So it will never upgrade on its own, and the stale release persists indefinitely.
+
+**The only thing preventing the incident was `replicas=0` — manual drift from the incident recovery,
+which any Helm upgrade would reset to the chart default of 1.** That is not a guard; it is luck.
+
+Containment applied (reversible, and it costs nothing because prod publishes no hostnames yet — the
+app tiers are suspended): **`flux suspend helmrelease external-dns -n external-dns`**, with
+`replicas` left at 0. Flux can now neither upgrade the release nor reset the replica count.
+
+**Verified remediation, in this order — the ordering is the whole point:**
+
+1. Force the upgrade so the release picks up `txtOwnerId`, but **do not let the pod run before the
+   flag is confirmed present.** The naive `resume` + `reconcile --force` resets `replicas` to 1 and
+   starts the pod in the same operation, so there is no window in which to check — a race against a
+   process whose failure mode is deleting dev's DNS.
+2. Confirm the live Deployment args contain **`--txt-owner-id=prod`**.
+3. Only then allow it to scale to 1, and immediately diff the Cloudflare zone against a
+   pre-recorded snapshot.
+
+**Dev was verified unharmed throughout this work:** the Cloudflare zone was snapshotted before
+unsuspending `platform` and re-queried after — **57 records, byte-for-byte identical, all 36 TXT
+entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local
+resolver (a pfSense override produced a false healthy reading during the incident).
+
+## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**
+
+> **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's
+> `platform` tier and its 10 downstream tiers are SUSPENDED. Dev is **live on the apex**. The
+> ordered cutover is **§10a**, and merging the authored change to `main` is step **D3** of it,
+> not step one — dev self-reconciles `main`, so a merge outside the runbook window executes
+> the migration unattended.
 
 ### What happened
 
@@ -1320,43 +1473,82 @@ values; for DNS and TLS that promise was false, so `clusters/prod` was *authored
 public identity. Two clusters that describe the same DNS name will fight over it no matter how the
 DNS controller is configured.
 
-### The architecture decision (operator)
+### The architecture decision (operator) — **INVERTED 2026-09-04, this is the current target**
 
 ```
-dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
-prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
-apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev
+dev   ->  *.dev.home.keithjackson.dev      gitea.dev.home.keithjackson.dev
+prod  ->  *.home.keithjackson.dev          gitea.home.keithjackson.dev     (THE APEX)
 ```
 
-### PHASE 1 — **prod only. Dev did not move.** (this change)
+**Prod *is* the apex. There is no `prod.` label and no CNAME layer at all** —
+`<svc>.home.keithjackson.dev` is prod's real name, not an alias for one. This **supersedes** the
+interim split committed earlier the same day (`c8c1d29`), which put prod on
+`*.prod.home.keithjackson.dev` behind apex CNAMEs pointing at it. That layer is deleted: fewer
+moving parts, no CNAME chain to keep in step, and prod ends up on the names users already type.
 
-`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
+The cost of the inversion is that **dev must move**, and dev is the live cluster. The interim split
+was chosen precisely because it moved only the unbuilt environment; this one does not have that
+property, so the whole of §10's remaining risk now sits on dev's side, and it is sequenced below.
 
-| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
-|---|---|---|---|
-| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
-| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
-| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
+### The values, per environment
 
-**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
-alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
-while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
-`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
-by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
-both overlays and diffing.
+| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` | Ansible `k3s_base_domain` |
+|---|---|---|---|---|
+| base default | `dev.home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` | `dev.home.keithjackson.dev` (role default) |
+| **dev** | `dev.home.keithjackson.dev` | `dev` | `keithjackson.dev` | (inherits the role default) |
+| **prod** | `home.keithjackson.dev` | `prod` | `keithjackson.dev` | `home.keithjackson.dev` (`group_vars/k3s_prod`) |
 
-**Two independent guards now, either sufficient alone:**
+`[parity]`: `k3s_base_domain` must equal the same cluster's `BASE_DOMAIN`. The kube-apiserver's
+`oidc-issuer-url` is a k3s **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux
+post-build substitution can never reach it — hence one fact in two places. If they disagree the
+apiserver validates tokens against a different Keycloak than the cluster runs: kubectl SSO fails
+while the cert-based admin kubeconfig keeps working, so it presents as "SSO is broken", not as skew.
 
-1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
-   the direct fix for the incident. Dev **must** render exactly `default`: its live records are
-   already registered under that owner, and any other value would make dev's own external-dns stop
-   recognising them and re-create duplicates beside the orphans.
-2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
-   Defence in depth, added because the first guard is a single string in a single file.
+**`EXTERNALDNS_DOMAIN_FILTER` is the Cloudflare ZONE for BOTH, and must stay that way.** FINDING
+F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for **zone selection** as
+well as record matching, so a filter that is a strict subdomain of the hosted zone matches **no
+zone** — external-dns generates the record and then skips it with *"no hosted zone matching record
+DNS Name"*, publishing **nothing**. This has been "hardened" and reverted more than once. It is not
+defence in depth; it is a total outage of DNS publication. Isolation comes from the two guards below,
+which a domain filter cannot provide anyway: both environments live in the one zone regardless.
 
-The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
-future environment that forgets the knob must not silently inherit dev's identity — that is exactly
-what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
+**The two guards that actually work:**
+
+1. **`txtOwnerId`, per-environment identity** — a cluster only ever reconciles records its own owner
+   id claims in the TXT registry. This is the direct fix for the incident.
+2. **Disjoint hostname sets** — after the migration the two clusters never describe the same record,
+   so there is nothing to fight over. *This guard does not exist during the migration window*, which
+   is the entire reason the ordering below is mandatory.
+
+#### Why dev's owner id changed from `default` to `dev`
+
+Until this migration dev **had** to render `default`: its live apex records were already registered
+in Cloudflare's TXT registry under that owner, and renaming would have orphaned every one of them.
+**The migration dissolves that constraint** — on `*.dev.home.keithjackson.dev` dev publishes an
+entirely new record set with no prior registry entries to recognise, so the id is free and is chosen
+on its merits. `default` has none: it is the value external-dns uses when you **forget** the flag,
+which is exactly how both clusters came to share it, and an environment whose identity is the
+forget-value can never be told apart from a misconfigured newcomer. Consequences, both deliberate:
+
+- Dev's **old apex records stay owned by `default`**, which no environment now claims, so nothing
+  deletes them implicitly on some future reconcile. Retiring them is an explicit step (**D7**) and
+  prod's first publish is gated on that retirement being *confirmed* — not on two reconcile loops
+  racing each other.
+- **Prod stays `prod` and must never be given `default`.** Prod is taking over the apex *names*, and
+  dev's existing apex records are registered under `default` — so `default` is the one value that
+  would let prod adopt them, and under `policy: sync` delete the ones no prod source claims, on a
+  cluster dev is still serving. That is the 2026-09-04 incident re-armed with the roles swapped.
+  Prod inherits the apex **names**; it must never inherit the apex **owner id**.
+
+All three values differ, and the base default remains a sentinel so an environment that forgets the
+knob owns nothing and can therefore delete nothing.
+
+### PHASE 1 — prod tokenized and moved off dev's names (`c8c1d29`, done; suffix since inverted)
+
+`BASE_DOMAIN` was a **dead knob** — declared in `clusters/base/cluster-vars.yaml` and referenced by
+no reconciled manifest, because every hostname was a hardcoded literal. `c8c1d29` made it real (141
+references across the reconciled set) and gave prod its own suffix. That work stands; only the
+*value* prod holds has changed, from `prod.home.keithjackson.dev` to the apex.
 
 ### What could NOT be tokenized, and why
 
@@ -1365,70 +1557,249 @@ what happened here. With the sentinel such a cluster owns nothing, so it can del
   Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
   **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
   would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
-  **Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
-  Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
-  secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
-  is per-realm.
+  **Both environments now need action:** dev's copy must move to `keycloak.dev.home.keithjackson.dev`
+  as part of step **D5**, and prod needs its own copy with `keycloak.home.keithjackson.dev` and a
+  client secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client
+  secret is realm-scoped.
 - **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
-  **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
-  can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
-  `k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
-  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
-  `[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
-  validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
-  cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
+  server flag, not a manifest. Handled on the **Ansible** seam instead: role default
+  `k3s_base_domain` (dev's value) + derived `k3s_oidc_issuer_url`, overridden for prod in
+  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`,
+  so a fourth prod server inherits it from group membership. Verified per host by a read-only probe
+  (`connection: local`, zero SSH, `changed=0`): all three dev servers resolve
+  `https://keycloak.dev.home.keithjackson.dev/realms/homelab`, all three prod servers resolve
+  `https://keycloak.home.keithjackson.dev/realms/homelab`.
+- **`spec.url` of dev's `GitRepository`** (`clusters/dev/kustomization.yaml`) — see the strand below.
+  It *cannot* be tokenized: the `flux-system` root Kustomization that renders `./clusters/dev`
+  declares no `postBuild.substituteFrom`, so `gitea.${BASE_DOMAIN}` there would ship the **literal
+  token** and break the source outright (the L0 `flux-postbuild-substitution` check asserts exactly
+  this). It is a hand-maintained literal that must move in lockstep with `BASE_DOMAIN`.
 
-### A pre-existing prod problem this incidentally FIXED
+---
 
-§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
-issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
-making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
-its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
-**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
-and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
-external-dns has actually published the record; until then use the break-glass cert kubeconfig.
+## 10a. THE MIGRATION RUNBOOK — ordering is load-bearing
 
-### Verify on prod's first reconcile
+> **Prod's `platform` tier and its 10 downstream tiers are SUSPENDED and must stay suspended until
+> step P1.** Nothing below is safe to run out of order, and nothing below should be run while the
+> operator is away.
 
-- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
-      as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
-      ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
-      will create **nothing** — fail-closed, never destructive. Fallback: set prod's
-      `EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
-      sufficient on its own.
-- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
-      dev's `_acme-challenge` records.
-- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
-      entries still read owner `default` and that prod's read owner `prod`.
-- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
-      secret is realm-scoped; prod cannot reuse dev's).
+### The two ways this goes wrong
 
-### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
+**(1) BOTH CLUSTERS ON THE APEX — the 2026-09-04 incident, recreated.** If prod publishes
+`*.home.keithjackson.dev` before dev has vacated it, two clusters describe the same records in the
+same zone again. Distinct `txtOwnerId`s stop prod from *deleting* dev's records — that guard holds —
+but they do not stop the second failure mode: both clusters requesting a `letsencrypt-prod` wildcard
+for the **same name set** and racing each other's `_acme-challenge` TXT records, so neither reliably
+solves DNS-01. **Rule: dev vacates the apex first, and the vacate is verified against Cloudflare
+before prod is unsuspended.**
 
-Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
-Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
-Required, roughly in order:
+**(2) THE STRAND — dev's GitOps source is the thing that moves.** `spec.url` of dev's
+`GitRepository` is `https://gitea.home.keithjackson.dev/...`, which is **both** dev's GitOps source
+**and** the PM's push target. Flipping dev's `BASE_DOMAIN` moves Gitea's HTTPRoute, its public DNS
+record and the wildcard cert covering it to `gitea.dev.home.keithjackson.dev` all at once. If that
+host does not already resolve and serve a valid, Ready cert at that instant, **dev stops reconciling
+and cannot be fixed by pushing** — the broken thing *is* the path Flux uses to read pushes. Recovery
+is then hands-on and needs LAN access to the cluster.
 
-1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
-   dev inventory group.
-2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
-3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
-   (`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
-   `qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
-   updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
-   **rolling k3s restart** on the dev servers.
-4. **kubectl SSO re-issue** for every user (issuer changed).
-5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
-   `clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
-   the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
-   Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
-6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
-   records are for names dev no longer serves; retire them deliberately rather than letting a
-   renamed owner orphan them.
-7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
-   actually **moves user traffic to prod**, and it must not land before prod serves those services.
-8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
-   `dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
-   `trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
-   — but they will fail against prod and must be parameterized before §9's "suite passes green
-   against the prod cluster" can be claimed.
+**(2b) AND THE MERGE ITSELF IS THE TRIGGER.** `clusters/dev/sync.yaml` is a self-managing root
+Kustomization reconciling `./clusters/dev` from **`main`** every 10 minutes, and prod tracks `main`
+too. **A merge to `main` therefore *executes* this migration on the live dev cluster, unattended.**
+The authored change is a *target*, not a deployment. **Do not merge these files to `main` outside the
+runbook window.** Park them on a branch until the operator is at the keyboard and step **D1** has
+been done.
+
+### D — dev vacates the apex (operator present, in this order)
+
+- [ ] **D0. Snapshot the escape routes before touching anything.** Confirm a working break-glass
+      **cert-based** admin kubeconfig for dev (not the OIDC one — OIDC breaks in D6). Confirm the
+      retained bootstrap Gitea can be started (`bootstrap/gitea/bootstrap-gitea.sh`; dev's historical
+      recovery host was `192.168.2.103:3000`, prod's current bootstrap is `192.168.2.132:3000`).
+      Record dev's current Cloudflare record set (names + TXT registry owners) so D7 has a checklist
+      and so an accidental deletion is detectable.
+- [ ] **D1. Break the automatic trigger.** Either `flux suspend kustomization flux-system -n
+      flux-system` on dev (stops the root from self-applying the merge), **or** re-point dev's
+      `GitRepository` url at the bootstrap Gitea for the window. Do this **before** the merge, not
+      after. This is what converts "merging strands dev" into "merging is inert until you say go".
+- [ ] **D2. Publish and prove the new host set *while dev still serves the apex*.** The new names
+      must exist before anything depends on them:
+      - a `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev` **wildcard Certificate issued
+        and `Ready`** (LE prod, DNS-01) — see the ACME note below before ordering it;
+      - public DNS for `gitea.dev.home.keithjackson.dev` resolving to dev's Gateway VIP
+        (192.168.1.53), **checked against Cloudflare / a public resolver, never through the pfSense
+        override** — that mistake was made on 2026-09-04 and produced a false healthy reading;
+      - `curl -sS https://gitea.dev.home.keithjackson.dev/api/v1/version` succeeding with a
+        publicly-trusted cert from a host on the dev LAN.
+      Until D2 is green, D3 is not safe.
+- [ ] **D3. Merge the authored change to `main`** (`clusters/base/cluster-vars.yaml`,
+      `clusters/dev/`, `clusters/prod/`, the two Ansible files, `platform/externaldns`,
+      `catalog/render.py`). With D1 in place nothing moves yet.
+- [ ] **D4. Release dev's reconciliation deliberately.** Resume the root Kustomization (or restore
+      the in-cluster url, now pointing at `gitea.dev.home.keithjackson.dev`) and watch
+      `flux get kustomizations -A` converge. Dev's Gateway listeners, HTTPRoutes, coredns-custom zone
+      and Homepage links all move here. Dev's external-dns publishes the new record set under owner
+      `dev`.
+- [ ] **D5. Keycloak: issuer + every client.** Dev's realm issuer becomes
+      `https://keycloak.dev.home.keithjackson.dev/realms/homelab`. The issuer is baked into every
+      token, and **~12 OIDC clients** need their redirect URIs moved in the same change: `grafana`,
+      `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`, `qbittorrent`,
+      `radarr`, `sonarr`, plus the `kubernetes` public client. Most are reconciled from
+      `platform/keycloak/keycloak-realm-import.yaml` and move with D4; **headlamp's is the exception**
+      — its issuer/callback live in a SOPS-encrypted Secret and must be re-encrypted by hand.
+- [ ] **D6. Rolling k3s restart + kubectl SSO re-issue.** `k3s_base_domain` now renders a different
+      `oidc-issuer-url` into `/etc/rancher/k3s/config.yaml`, so the `k3s_server` role is **no longer
+      a no-op on dev**: it rewrites the file and fires the Restart handler on all three servers. Run
+      it **one server at a time**, confirming `kubectl get nodes` between each. Every user's OIDC
+      kubeconfig is invalidated and must be re-issued; the cert-based admin kubeconfig keeps working
+      throughout, which is what makes this step recoverable. **Do not reboot nodes** — restart the
+      `k3s` service only.
+- [ ] **D7. Retire dev's old apex records.** They are owned by `default`, which no environment now
+      claims, so nothing removes them automatically. Delete the `<svc>.home.keithjackson.dev` A
+      records, the apex record, **and their TXT registry entries** (`external-dns-*` TXT), plus any
+      stale `_acme-challenge.home.keithjackson.dev`. *Alternative, if you prefer external-dns to do
+      its own cleanup:* set dev's `EXTERNALDNS_TXT_OWNER_ID` back to `default` for exactly one
+      reconcile after D4 — `policy: sync` will then garbage-collect the apex set it no longer has
+      sources for — and only then switch it to `dev`. That is two commits and needs watching; the
+      direct deletion is preferred because it is verifiable in one place.
+- [ ] **D8. Verify the apex is EMPTY, against Cloudflare.** `dig +short <svc>.home.keithjackson.dev
+      @1.1.1.1` returns nothing for every service name, and no TXT registry entry for those names
+      remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not
+      evidence about public DNS.
+
+### D0/D1 AS EXECUTED (2026-09-04) — and the self-reference that would have undone it
+
+D1 was taken via the **repoint** option, not the suspend option, because it is strictly stronger:
+suspending only defers the strand, while repointing removes the dependency that causes it. Dev's
+`GitRepository` now reads **`http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git`,
+branch `dev-live`** — IP-addressed, plain HTTP, so it depends on **neither DNS nor TLS**, which is
+exactly the property needed during a DNS/TLS migration. **A merge to `main` is now inert for dev.**
+
+**THE TRAP, and it is not obvious: a bare copy of `main@4809199` on the bootstrap Gitea would NOT
+have held.** The root Kustomization renders `./clusters/dev`, and `clusters/dev/kustomization.yaml`
+*contains the patch that sets `GitRepository.spec.url`*. So Flux would have fetched from the
+bootstrap Gitea, applied the manifests it found there, and those manifests would have set the url
+straight back to `gitea.home.keithjackson.dev` — reverting the repoint on the next reconcile and
+re-arming the strand. **The branch must declare its own source.** `dev-live` (`a4a3c2e`) therefore
+carries exactly one content change on top of `4809199`: `clusters/dev/kustomization.yaml`'s url +
+`ref.branch`. Proven by `kubectl kustomize clusters/dev` at both revisions differing in **exactly
+two lines** out of 357. The same property makes the revert self-sustaining in reverse, since
+`main@4809199` carries the `gitea.home…` url. Revert command: see `REVERT.txt` in the run scratchpad.
+
+Verified after the cutover: **13/13 Kustomizations Ready** on `dev-live@a4a3c2e`, `cluster-vars`
+byte-identical to the D0 snapshot (27/27 keys, `BASE_DOMAIN` still `home.keithjackson.dev`),
+**18/18 Longhorn volumes attached+healthy**, all 16 HTTPRoute hostnames still on the apex,
+`wildcard-home` still Ready, and the Cloudflare record set re-queried and diffed **byte-for-byte
+against the D0 snapshot — identical** (57 records, 36 TXT all `owner=default`).
+
+> **NEW DEPENDENCY, AND IT IS THE WEAK LINK.** `192.168.2.132` is a **DHCP lease on the
+> workstation's wifi** (`wlp0s20f3`, NetworkManager `ipv4.method=auto` — confirmed, not assumed). A
+> lease change, a wifi drop, or the workstation going offline **stalls dev's fetching**. The cluster
+> keeps running and serving; only reconciliation pauses, and the fix is a `kubectl patch` of the url.
+> Two consequences: (a) a **static reservation** for this host is worth having before any repeat of
+> this procedure, and (b) **keep the window short** — the exit is D4, which returns dev to its own
+> in-cluster Gitea at `gitea.dev.home.keithjackson.dev`.
+>
+> Pointing Flux at dev's Gitea **Service** instead (`gitea.gitea.svc.cluster.local:3000`) would drop
+> the DNS, TLS *and* workstation dependencies all at once, and is the better long-term shape. It was
+> **not** done here: namespace `gitea` carries a `default-deny` NetworkPolicy plus a
+> `gitea-allow-gateway-ingress` CiliumNetworkPolicy admitting **only** the Gateway, so
+> source-controller cannot reach the Service without a new authored policy. Worth doing deliberately,
+> not mid-migration.
+
+### P — prod takes the apex
+
+- [ ] **P1. Unsuspend prod**, in dependency order: `flux resume kustomization platform`, then the
+      10 downstream tiers.
+      **CORRECTION (2026-09-04) — the D8 gate is on prod taking the APEX, not on prod coming up.**
+      This step originally read "only now unsuspend prod", which silently assumed prod would be
+      serving `*.home.keithjackson.dev`. It is not: prod runs revision `c8c1d29`, where
+      `BASE_DOMAIN=prod.home.keithjackson.dev` and `EXTERNALDNS_TXT_OWNER_ID=prod`. Prod's names are
+      therefore a **disjoint subtree** under a **distinct registry owner**, so neither incident mode
+      of §10a applies: it cannot claim dev's `default`-owned records, and its wildcard
+      `*.prod.home.keithjackson.dev` is a **different name set** from dev's, so it has its own
+      Let's Encrypt duplicate-certificate bucket and races no `_acme-challenge` with dev.
+      **Prod may therefore be brought up in full while dev still holds the apex** — and should be,
+      because it proves the entire platform stack end-to-end *before* the apex flip, turning the
+      cutover into a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
+      once. What D8 gates is **only** the later move of prod's `BASE_DOMAIN` to the bare apex.
+      Verify the two guard values live in prod's applied `cluster-vars` before unsuspending; if
+      `EXTERNALDNS_TXT_OWNER_ID` is not `prod`, STOP — that was the root cause of the incident.
+- [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted
+      zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means
+      the filter was narrowed against instruction). Confirm the new TXT registry entries read owner
+      `prod` and that no record owned by `dev` was touched.
+- [ ] **P3. Prod's wildcard cert `*.home.keithjackson.dev` + `home.keithjackson.dev` is `Ready`** and
+      solved DNS-01 without colliding with any `_acme-challenge` dev still holds.
+- [ ] **P4. Prod's kubectl SSO works without an `/etc/hosts` pin.** Prod nodes resolve
+      `keycloak.home.keithjackson.dev` through **public** upstreams (§4), so this only becomes true
+      once P2 has published that name at prod's Gateway VIP (192.168.2.20). Before then a prod node
+      resolving its own issuer reaches **dev**. Use the break-glass cert kubeconfig until P2 is done.
+- [ ] **P5. Cut prod's GitOps source over** from the workstation bootstrap Gitea
+      (`http://192.168.2.132:3000/...`) to `https://gitea.home.keithjackson.dev/...` once prod runs
+      its own Gitea — the block in `clusters/prod/kustomization.yaml` owns that procedure. Note the
+      inversion **simplified** this: prod's apex Gitea name and dev's `gitea.dev.home...` are now
+      different hosts, so prod pointing at the apex means "prod's own Gitea", never dev's.
+
+### ACME: the duplicate-certificate limit
+
+Let's Encrypt rate-limits **duplicate certificates — 5 per week for an identical set of names.**
+Prod's wildcard is `*.home.keithjackson.dev` + `home.keithjackson.dev`, which is **exactly the name
+set dev holds today**. Every prod order therefore counts against the same bucket as dev's historical
+renewals of that pair, and a retry loop during P3 can exhaust it and leave prod with **no trusted
+cert for up to a week**. Mitigations:
+
+- Order prod's cert **after** D8, so dev is no longer renewing the same pair.
+- Let dev's old `*.home.keithjackson.dev` `Certificate`/Secret be **deleted** in D4/D7 rather than
+  left to renew in the background.
+- Rehearse against **letsencrypt-staging** first (`ACME_CLUSTER_ISSUER`) — staging has its own,
+  far looser limits — and only switch prod to `letsencrypt-prod` when the DNS-01 solve is proven.
+- Do **not** delete-and-recreate prod's `Certificate` to "retry"; each recreation is a fresh order.
+
+### The L2 tests hardcode the apex
+
+Six behavioural (L2) chainsaw tests embed `home.keithjackson.dev` as a literal:
+
+| test | file |
+|---|---|
+| `dns-exposure` | `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
+| `dns-internal-resolution` | `conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml` |
+| `gateway-ingress-reachable` | `conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml` (+ `fixtures/app-and-probe.yaml`) |
+| `oidc-sso-grafana` | `conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml` |
+| `sso-forward-auth` | `conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml` |
+| `trusted-tls` | `conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml` |
+
+They are **test inputs, not reconciled manifests**, so they are deliberately left alone by the
+authoring change: they are the only thing still asserting the apex, and while dev is live on the apex
+they are *correct*. Under the new direction the literal happens to become right for **prod** and
+wrong for **dev**, so they must be **parameterized per environment** (not merely edited) before
+either §9's *"suite passes green against the prod cluster"* or a green L2 run against migrated dev
+can be claimed. Until then, expect the dev L2 suite to fail between D4 and that parameterization —
+a known, scheduled failure, not a regression.
+
+The L1 *rendering* tests need no change: their goldens carry `${BASE_DOMAIN}` as a token, and the
+literal hostnames in them come from each test's own fixture `ingressRoutes[].host`, which is
+environment-independent by construction.
+
+### Offline proof carried out for the authoring change (no cluster touched)
+
+All 12 `substituteFrom` paths were rendered for **both** overlays through the real toolchain
+(`kubectl kustomize --load-restrictor LoadRestrictionsNone <spec.path> | flux envsubst --strict`,
+with each overlay's effective `cluster-vars` taken from `kubectl kustomize clusters/<env>`), before
+and after the change.
+
+- **Dev's diff is *only* the hostname move.** Proven by inverse substitution rather than by reading
+  the diff: applying `s/dev.home.keithjackson.dev/home.keithjackson.dev/` to the new render restores
+  the baseline **byte-for-byte** in all 12 paths, plus 2 declared `txtOwnerId: default → dev` lines.
+  145 changed line pairs, all classified: 143 hostname-move, 2 declared owner-id, **0 unexplained**,
+  and **0 bare-apex remnants**. The 2 paths with no change (`cilium`, `velero`) provably contain no
+  hostname at all (`grep -c keithjackson.dev` = 0).
+- **Prod lands on the bare apex:** **0** occurrences of `prod.home.keithjackson.dev` and **0** of
+  `dev.home.keithjackson.dev` anywhere in prod's rendered output; 143 apex names.
+- **Certs:** dev `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev`; prod
+  `*.home.keithjackson.dev` + `home.keithjackson.dev`.
+- **Positive controls** (the harness is provably not blind): perturbing an unrelated knob
+  (`GATEWAY_VIP`) surfaces 1 unexplained line; perturbing `BASE_DOMAIN` to a wrong value fails 10 of
+  12 paths; injecting a single un-moved hostname is caught by the independent bare-apex scan (which
+  the byte-identity check alone cannot see — which is why both exist).
+- Gates: `ci/l0.sh`, `ci/l1.sh`, `ci/render-golden.sh`, `ci/lint.sh`,
+  `flux-postbuild-substitution/check.sh` (254 assertions), `catalog/render.py --selftest`,
+  `ci/spec-immutability.sh` — all **PASS**.

infra/PROD-STANDUP.md

@@ -0,0 +1,519 @@
+# Prod cluster standup — the complete command sequence
+
+**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
+container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
+this file it is not part of standup.
+
+State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s
+v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready**
+(`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and
+**all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is
+confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not
+biting on the backports kernel).
+
+Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory
+upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** —
+though only after a leftover cordon was cleared; read
+[§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing)
+before doing node work, that cordon silently prevents Longhorn from rebuilding.
+
+What remains is in [§Blockers](#blockers), and every item there needs an operator decision:
+no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found
+2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod.
+
+> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
+> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
+> therefore **requires** `--expect-server` and compares it against the live API server before running
+> anything. There is no override flag. Do not add one.
+
+## Constants
+
+| | |
+|---|---|
+| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
+| Gateway / ingress VIP | `192.168.2.20` |
+| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
+| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
+| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
+| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |
+
+Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
+**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).
+
+---
+
+## Phase A — bare metal to a joined cluster (Ansible)
+
+Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
+network (§4), and SSH + the vaulted `ansible_become_password` in
+`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
+file. See `infra/PROD-PROVISIONING.md`.
+
+**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
+publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
+validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
+from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
+```
+
+Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
+required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/site.yml \
+  -i /work/infra/ansible/inventory/hosts.ini \
+  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
+```
+
+kube-vip is a separate play (not imported by `site.yml`):
+
+```bash
+timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
+```
+
+Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).
+
+> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
+> target var so they stay independently pointable while a bring-up run is scoped elsewhere
+> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
+> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
+> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
+> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
+> caught only by `--list-hosts`. **Always dry-run the target first:**
+>
+> ```bash
+> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+>   ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
+>   -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
+> ```
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
+  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
+  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
+```
+
+## Phase B — Flux bootstrap
+
+`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
+documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
+no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
+and refuses to continue rather than silently skipping decryption
+(`bootstrap/sops/age-key-secret.md`).
+
+## Phase C — bring the reconciliation graph up
+
+One script, six steps, each independently runnable and idempotent. **Run them in this order.**
+
+```bash
+K=infra/ansible/.cache/prod.kubeconfig
+S=https://192.168.2.9:6443
+```
+
+### C1 — preflight (read-only, changes nothing)
+
+Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
+closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).
+
+```bash
+timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
+```
+
+### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)
+
+```bash
+timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
+```
+
+Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization
+before applying any of them, so one unknown `kind` aborts the entire tier.**
+
+1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets
+   `crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them
+   server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).
+
+2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and*
+   `platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does
+   not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind →
+   the CR is never applied → the CRD is never created → `platform-config` can never converge, and
+   every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been
+   there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator`
+   Available for 12h with **no CDI CR** and no `storageprofiles` CRD.
+
+   The step breaks it by applying **only the CR** — the minimum — and leaving everything else to
+   Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
+   The CRD appeared in 10s.
+
+   **REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier,
+   `./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is
+   **revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy*
+   — which is precisely the event that creates the `StorageProfile` CRD — before the profiles are
+   applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster
+   already stuck in the broken state, and because it is idempotent.
+
+   **The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a
+   *controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers
+   exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and
+   `dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of
+   `platform-config` found `StorageProfile` is the only second-order kind there.
+
+   `apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic:
+   `apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment
+   previously claimed `platform-config` guaranteed the profiles, which the move would have silently
+   falsified.
+
+#### Merge precondition and the one-time transient
+
+`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both
+StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi`
+re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).
+
+**Measured on dev, so the exposure is known rather than assumed** — comparing our override against
+what CDI infers unaided for the same Longhorn provisioner:
+
+| profile | `spec` (ours) | CDI-derived `status` |
+|---|---|---|
+| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` |
+| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` |
+
+So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes`
+is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window*
+would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that
+(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand
+during the window.**
+
+- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If
+      it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in
+      exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with
+      `platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in
+      a second commit.
+- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries
+      `claimPropertySets` RWX/Block, and both are labelled
+      `kustomize.toolkit.fluxcd.io/name=platform-config-cdi`.
+
+### C3 — unwedge stuck Helm releases
+
+A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
+`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
+**Dry run by default** — it prints what it would do and changes nothing:
+
+```bash
+timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
+```
+
+To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
+CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
+  --confirm-destructive unwedge
+```
+
+### C4 — external-dns safe upgrade
+
+**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
+resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
+window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
+`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
+them**.
+
+So this step **proves the outcome before causing it**: it renders the chart with the *live*
+HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
+deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
+Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
+and diffs the zone.
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
+```
+
+Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
+`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
+regression — stop and investigate.
+
+### C5 — resume the tiers, in dependency order
+
+`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
+apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
+partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
+a placeholder (see §Blockers). Long-running; expect it to background.
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
+```
+
+### C6 — verify
+
+```bash
+timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
+```
+
+### Everything at once
+
+Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
+destructive step by design:
+
+```bash
+timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
+  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
+```
+
+---
+
+## VERIFIED WORKING — 2026-09-05, standup complete
+
+| | |
+|---|---|
+| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
+| Pods | **113 total, 0 not Running/Completed** |
+| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
+| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
+| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
+| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |
+
+End-to-end HTTPS **through the workstation's own resolver — no `--resolve` override, no `-k`** (after
+installing the dnsmasq profile below). Every request landed on `192.168.2.20`:
+
+```
+prod.home.keithjackson.dev  200   headlamp.  200   keycloak.  302 (login)   nextcloud. 302
+gitea.       200            semaphore. 200        grafana.   302 (SSO)      pihole.    302
+```
+
+The served certificate is prod's own, confirmed by SNI — **not** dev's wildcard, which is what the
+resolver bug previously made it look like:
+
+```
+subject = CN=*.prod.home.keithjackson.dev
+issuer  = C=US, O=Let's Encrypt, CN=YR1          (LE production)
+SAN     = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
+valid   = 2026-09-05 .. 2026-12-04
+```
+
+Dev, checked in the same pass, still serves `CN=*.home.keithjackson.dev` — **unchanged**.
+
+> Ignore `curl: (91) No OCSP response received` if you pass `--cert-status`: Let's Encrypt retired
+> OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain
+> requests above all validated normally.
+
+### Workstation DNS — install the prod dnsmasq profile
+
+**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
+notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.
+
+`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
+
+```
+address=/.home.keithjackson.dev/192.168.2.50
+```
+
+dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
+the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
+correctly returns prod's `192.168.2.20`.
+
+**Fix — one command, already authored in this repo:**
+
+```bash
+sudo infra/workstation/dnsmasq/install.sh
+```
+
+It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
+`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
+Verified against a throwaway dnsmasq loaded with both profiles:
+
+| query | answer |
+|---|---|
+| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
+| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
+| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
+| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
+
+> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
+> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
+> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
+> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
+
+**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
+convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
+`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
+a broken prod certificate. It is the wrong server answering.
+
+```bash
+curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
+  https://gitea.prod.home.keithjackson.dev/
+```
+
+## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing
+
+Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade
+on prod-2) produced one finding worth more than either incident.
+
+### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing
+
+`kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods,
+DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.**
+Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica
+scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with
+prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod
+was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged
+17/17 healthy in ~6 minutes.
+
+**So: after any drain — successful or not — `kubectl uncordon <node>` is not tidying up, it is what
+re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\
+U:.spec.unschedulable` before believing a green pod list.
+
+### `drain` cannot complete on this cluster, and the two usual flags do not fix it
+
+`--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs
+indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and
+`node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an
+instance-manager be evicted while it runs volume engines. That is data protection, not a bug.
+
+**Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no
+fourth node to move a replica to, so eviction can never complete. This is structural, not a setting.
+
+Options, in order of preference:
+
+1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on
+   return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10
+   minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel
+   work.
+2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it —
+   that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with:
+   `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`.
+3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades
+   the protection for exactly the data risk the default exists to prevent.
+
+### Check the kube-vip leader before rebooting a server
+
+`kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so
+`--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the
+lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault.
+Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`.
+
+### Post-maintenance checklist
+
+- [ ] `kubectl uncordon <node>` — **and confirm** `.spec.unschedulable` is clear on all three.
+- [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**.
+      Rebuilds are serial, so allow minutes, and watch rather than assume.
+- [ ] Replica symmetry: each node runs the same count
+      (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`).
+- [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and
+      a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`.
+- [ ] Pods: none `Running`-but-not-`Ready`.
+
+> **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread
+> was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA
+> apps here a spread constraint would change nothing (one replica lives on one node either way). It
+> self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing
+> **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and
+> Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the
+> operator owns — see §Blockers.
+
+## Blockers
+
+### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
+
+**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
+`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.
+
+**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
+from the workstation, which proves nothing about pod egress:
+
+```
+curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
+```
+
+**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
+**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
+**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
+`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
+pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
+a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
+
+Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
+
+> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
+> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
+> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
+> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
+
+Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
+token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
+**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
+coverage.**
+
+#### Sharing one MinIO is only safe because of two new knobs
+
+Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
+MinIO without these would have been the external-dns incident again, in a worse place:
+
+| | before | after |
+|---|---|---|
+| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
+| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |
+
+- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
+  when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
+- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
+  so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
+  WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
+  **proven** (R-STATE-03).
+
+**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
+backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
+backups its proven restore depends on** — the same reasoning that keeps dev's
+`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.
+
+> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
+> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
+> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
+> commits. No urgency: velero cannot be enabled until the routing above is fixed.
+
+### 2. Prod is on `prod.home.keithjackson.dev`, not the apex
+
+The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
+Prod cannot take the apex until dev vacates it, because both clusters would request a
+`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
+records, so neither reliably solves DNS-01.
+
+**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
+nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
+the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
+once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
+
+> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
+> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
+> production environment**, which the operator requires to stay live until the switch is deliberately
+> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
+> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
+> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
+> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
+>
+> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
+> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
+
+---
+
+## Not covered here
+
+- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
+  redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
+- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
+  `clusters/prod/kustomization.yaml` owns that procedure.
+- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
+  §9's "suite passes green against the prod cluster" can be claimed.

infra/ansible/collections/requirements.yml

@@ -2,7 +2,8 @@
 # Ansible Galaxy collections required by infra/ansible. Install with:
 #   ansible-galaxy collection install -r collections/requirements.yml
 collections:
-  # node_baseline role: community.general.modprobe, community.general.filesystem
+  # node_baseline role: community.general.modprobe, community.general.filesystem,
+  #                      community.general.timezone
   - name: community.general
   # node_baseline role: ansible.posix.mount
   - name: ansible.posix

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -306,35 +306,47 @@ node_baseline_dns_servers:
   # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
   # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
-  #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
+  # ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
+  # name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
+  # That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
+  # points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
+  # node resolving its own issuer through these public upstreams therefore reaches dev until:
+  #   (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
+  #   (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
+  #       Gateway VIP (192.168.2.20).
+  # Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
+  # /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
+  # admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
+  # the ordered steps are infra/PROD-PROVISIONING.md §10.
 node_baseline_dns_search: []
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
-# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
-# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
-# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
-# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
-# postBuild.substituteFrom never sees it. Full rationale in
-# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
+# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
+# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
+# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
+# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
+# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
 #
-# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
-# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
-# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
-# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
-# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
-# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
-k3s_base_domain: "prod.home.keithjackson.dev"
+# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
+# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records.
+#
+# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
+# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
+#   prod -> home.keithjackson.dev       (THE APEX; this variable — no `prod.` label, no CNAMEs)
+#   dev  -> dev.home.keithjackson.dev   (roles/k3s_server/defaults/main.yml)
+# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
+# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
+# against.
+#
+# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
+# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
+# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
+# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
+# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
+# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
+# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
+k3s_base_domain: "home.keithjackson.dev"
 
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE

infra/ansible/inventory/hosts.ini

@@ -31,17 +31,10 @@ k3s_secondary
 [hypervisor]
 localhost ansible_connection=local
 
-# VPN egress-gateway node — runs the node-level split-tunnel OpenVPN + kill-switch
-# (ADR-0015, playbooks/vpn.egress.gateway.yml). EXACTLY ONE host belongs here.
-#
-# THIS GROUP IS SINGLETON BY DESIGN — a second host does NOT buy failover. Cilium OSS has no
-# egress-gateway failover: the CRD's own `egressGateway` description says "In case multiple nodes
-# are a match for the given set of labels, the first node in lexical ordering based on their name
-# will be selected". Two labelled nodes therefore means one silently wins by NAME ORDER (a labelled
-# `k3s-prod-1` beats a labelled `k3s-prod-3`) and the other carries no traffic. See
-# platform/vpn-egress/README.md "Resilience — there is no gateway failover" for the full statement
-# and the ADR-0016 escalation. Members are already defined above with their ansible_host under
-# k3s_secondary; this group only tags the egress role target.
+# VPN egress-gateway node(s) — run the node-level split-tunnel OpenVPN + kill-switch
+# (ADR-0015, playbooks/vpn.egress.gateway.yml). Initially just k3s-dev-3 (single gateway;
+# add a second host here for Cilium Egress Gateway HA). Members are already defined above
+# with their ansible_host under k3s_secondary; this group only tags the egress role target.
 [vpn_egress_gateway]
 k3s-dev-3
 

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -95,15 +95,26 @@ k3s_tls_sans: >-
 # while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
 # not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
 #
-# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
-# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
-# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
-# what it already is and the Restart handler does NOT fire on a re-run.
+# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
+# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
+# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
+# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
+# from anyone remembering).
 #
-# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
-# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
-# from group membership rather than from anyone remembering.
-k3s_base_domain: "home.keithjackson.dev"
+# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
+# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
+# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
+# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
+#
+# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
+# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
+# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
+# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
+# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
+# work: DO NOT run the k3s_server role against dev until step D6 of
+# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
+# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
+k3s_base_domain: "dev.home.keithjackson.dev"
 
 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -81,6 +81,45 @@ node_baseline_kernel_modules:
   - dm_crypt
   - nfs
 
+# --- NODE CLOCK / TIMEZONE (tag `system`) ---------------------------------------------
+# EVERY NODE IS UTC. Measured drift that made this a knob rather than an assumption: the
+# three prod nodes disagreed on their local zone (read off k3s's own etcd snapshot
+# annotations, 2026-09-05) —
+#   k3s-prod-1  2026-09-05T16:31:28-05:00   (Central)
+#   k3s-prod-2  2026-09-05T16:29:07-05:00   (Central)
+#   k3s-prod-3  2026-09-05T17:27:35-04:00   (Eastern)
+# In UTC those three are ~4 minutes apart, which is the CORRECT staggering: nothing was
+# functionally wrong, because Kubernetes, etcd and the container runtime are UTC internally.
+# What IS wrong is that any timestamp a node renders in LOCAL time — journald's default
+# output, `ls -l`, a shell prompt, an etcd snapshot filename — cannot be compared across
+# nodes without knowing each node's zone. Correlating three nodes' logs during an incident
+# is exactly when nobody has that knowledge to hand, and this cluster took two unplanned
+# node reboots in the 24h before this was written.
+#
+# WHY UTC AND NOT "STANDARDISE ON CENTRAL". Picking one local zone leaves the class of
+# problem intact (a fourth node, or a DST boundary, re-creates it) and adds a translation
+# step to every log read, because every timestamp the PLATFORM emits — Kubernetes events,
+# Prometheus, Loki, Velero, CNPG WAL, the etcd snapshot annotations above — is already UTC.
+# UTC removes the class: node-local time and platform time become the same clock, and there
+# is no DST discontinuity in which a repeated or skipped hour breaks log ordering. This also
+# matches the only timezone convention the repo already had — catalog/plex.yaml records
+# "no other service in the repo sets a timezone ... so use the safe default" and both it and
+# apps/wave-0/plex.yaml set TZ=Etc/UTC. (platform/crafty sets America/Chicago, deliberately:
+# that is a USER-FACING game server's schedule, not a node's log clock.)
+#
+# WHY A KNOB AT ALL (R-ENV-02). The value is a node-level Ansible fact, so its override seam
+# is inventory/group_vars/<env>, NOT clusters/base/cluster-vars.yaml (a Flux postBuild
+# ConfigMap no Ansible code reads) — the same seam node_baseline_backports_kernel and
+# node_baseline_dns_servers use. NO ENVIRONMENT OVERRIDES IT TODAY, and one should not have
+# to: dev and prod both want UTC and there is nothing environment-specific about a node
+# clock. It is a variable rather than a literal in the task so a genuinely
+# operator-facing one-off host (never a k3s node) can differ without forking the role.
+#
+# Must be a zoneinfo name present on the node (`timedatectl list-timezones`); the assert in
+# tasks/main.yml rejects an empty or obviously-malformed value before the module runs, and
+# community.general.timezone itself refuses a name /usr/share/zoneinfo does not have.
+node_baseline_timezone: Etc/UTC
+
 # --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
 # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
 # a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -262,6 +262,52 @@
       vars:
         newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
 
+# =====================================================================================
+# NODE CLOCK — tag `system`: every node renders time in the SAME zone, and that zone is UTC
+# =====================================================================================
+# MEASURED DRIFT (2026-09-05), read off k3s's own etcd snapshot annotations on live prod:
+#   k3s-prod-1  2026-09-05T16:31:28-05:00   Central
+#   k3s-prod-2  2026-09-05T16:29:07-05:00   Central
+#   k3s-prod-3  2026-09-05T17:27:35-04:00   Eastern  <-- a different UTC offset
+# Functionally harmless (in UTC those three are ~4 minutes apart, which is the correct
+# staggering — Kubernetes, etcd and containerd are UTC internally). Operationally it is a
+# trap: any timestamp a node renders in LOCAL time (journald's default output, `ls -l`, an
+# etcd snapshot filename) is not comparable across nodes without knowing each node's zone,
+# and cross-node log correlation during an incident is precisely when nobody knows that.
+#
+# Rationale for UTC rather than standardising on one local zone, and for this being a knob
+# whose override seam is inventory/group_vars/<env>: see node_baseline_timezone in
+# defaults/main.yml. NOT a cluster-vars knob — no Ansible code reads that ConfigMap.
+#
+# IDEMPOTENCE / BLAST RADIUS. community.general.timezone is a converge-to-state module: it
+# reports `changed` only when /etc/localtime + /etc/timezone do not already name this zone,
+# and it does NOT step the clock (no time jump, no NTP interaction) — only the RENDERING of
+# an unchanged monotonic clock moves. No reboot is required. Already-running services that
+# render local time keep the old zone until they are next restarted (the module restarts
+# cron itself, which is the one scheduler on these nodes that cares); k3s, etcd and the
+# kubelet are unaffected either way because they emit UTC regardless.
+- name: Assert this node's timezone has actually been decided
+  ansible.builtin.assert:
+    that:
+      - node_baseline_timezone is string
+      - node_baseline_timezone | length > 0
+      - "'BLOCKED' not in node_baseline_timezone"
+      - node_baseline_timezone is match('^[A-Za-z][A-Za-z0-9+_-]*(/[A-Za-z0-9+_.-]+)*$')
+    fail_msg: >-
+      node_baseline_timezone is not a usable zoneinfo name (got:
+      '{{ node_baseline_timezone | default("<undefined>") }}'). It must be a name
+      /usr/share/zoneinfo actually carries — `timedatectl list-timezones` on the node — and
+      the repo default (Etc/UTC) is almost certainly what you want; see the rationale in
+      roles/node_baseline/defaults/main.yml. Empty is NOT "leave it alone": a node whose zone
+      nobody has decided is the config drift this task exists to remove.
+    quiet: true
+  tags: [system]
+
+- name: Pin the node timezone (cross-node log correlation; UTC by default)
+  community.general.timezone:
+    name: "{{ node_baseline_timezone }}"
+  tags: [system]
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf

infra/devcontainer/README.md

@@ -45,15 +45,8 @@ An env-var **prefix**, so the target sits on the same line as the command. Every
 a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
 from an exported shell variable.
 
-Inside the container `KUBECONFIG` is a two-file **chain**,
-`<per-invocation selector>:<shared merged config>`:
-
-| file | shared? | holds |
-|---|---|---|
-| selector, in a private `mktemp` dir | no — one per `dev.sh` invocation | only `current-context: k3s-$CLUSTER`; no credentials |
-| `infra/ansible/.cache/clusters.kubeconfig` | yes — bind-mounted, read by every container | both clusters under unambiguous names; **no `current-context`** |
-
-kubectl merges the chain first-wins, so the selector supplies this invocation's default:
+Inside the container `KUBECONFIG` points at a **merged** config holding both clusters under
+unambiguous names, with `CLUSTER`'s context current:
 
 ```
 CURRENT   NAME       CLUSTER    AUTHINFO
@@ -64,23 +57,6 @@ CURRENT   NAME       CLUSTER    AUTHINFO
 So `kubectl config current-context` answers "which cluster am I on?" honestly, and
 `kubectl --context k3s-prod …` works from a dev-defaulted shell.
 
-**Why the default context is not stored in the merged file.** It used to be, and that was a
-cross-cluster data-corruption bug. The merged file lives in the repo's `.cache`, which every
-container bind-mounts, so `current-context` was **shared mutable state**: two concurrent `dev.sh`
-runs with different `CLUSTER` values overwrote each other's default *mid-command*. A
-`CLUSTER=prod` script read prod for its first half and **dev** for its second; it was caught only
-because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file
-removes the shared field rather than locking it — there is no window left to lose. Read on its own
-the merged file now resolves to nothing (`error: current-context is not set`), which is loud and
-safe; `--context k3s-dev` / `--context k3s-prod` against it are unaffected and remain the pattern
-to prefer in scripts. Concurrent `CLUSTER=dev` and `CLUSTER=prod` invocations are now safe.
-
-Consequences worth knowing:
-
-- Export what `kubeconfig.sh` prints **verbatim** — it is a colon-joined chain, not one path.
-- `kubectl config use-context …` in an interactive toolbox shell writes to the **first** file in
-  the chain (the selector), so a manual switch stays local to that container.
-
 **Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
 the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
 occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
@@ -92,9 +68,6 @@ these files because the cluster and the user entry are both `- name: default` at
 - Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
   mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
   committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
-  Its content no longer depends on which cluster was asked for, so concurrent regenerations write
-  identical bytes; the write is published by atomic rename so a concurrent reader can never load a
-  half-written kubeconfig.
 - A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
   only `dev.kubeconfig` still works.
 - **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host

infra/devcontainer/dev.sh

@@ -21,23 +21,11 @@
 #
 # The selector is an env-var PREFIX so the target is visible on the same line as the command, and
 # every invocation prints a banner naming the cluster it is about to talk to. Inside the container
-# KUBECONFIG is a two-file CHAIN, "<per-invocation selector>:<shared merged config>":
-#
-#   * the shared merged config holds BOTH clusters under unambiguous names — contexts `k3s-dev`
-#     and `k3s-prod`, never `default` — so `--context k3s-prod` actually works, and it carries NO
-#     current-context of its own;
-#   * the selector is a private per-invocation file holding only `current-context: k3s-$CLUSTER`.
-#
-# So `kubectl config current-context` still answers the "which cluster am I on?" question honestly,
-# but the answer is now PER INVOCATION. That is the point: the previous design stamped
-# current-context into the one shared merged file, so two concurrent dev.sh runs with different
-# CLUSTER values overwrote each other's default MID-COMMAND — a `CLUSTER=prod` script could read
-# prod for its first half and DEV for its second. See infra/devcontainer/kubeconfig.sh for the full
-# write-up, and for why the naive `KUBECONFIG=dev:prod` merge is dangerous in the first place (both
-# source files name everything `default`).
-#
-# Concurrency note: `CLUSTER=dev ./dev.sh run ...` and `CLUSTER=prod ./dev.sh run ...` may now run
-# at the same time and neither can affect the other's default cluster.
+# KUBECONFIG is a MERGED config holding both clusters under unambiguous names — contexts `k3s-dev`
+# and `k3s-prod`, never `default` — with `CLUSTER`'s context selected as current. So `kubectl
+# config current-context` always answers the "which cluster am I on?" question honestly, and
+# `--context k3s-prod` actually works. See infra/devcontainer/kubeconfig.sh for why the naive
+# `KUBECONFIG=dev:prod` merge is dangerous (both source files name everything `default`).
 #
 # Requesting a cluster whose kubeconfig is missing is a hard ERROR — it never falls back to the
 # other cluster.
@@ -137,10 +125,6 @@ run() {
         # Regenerated every run: cheap, and it means an updated dev/prod.kubeconfig is picked up
         # without anyone remembering to refresh a hand-made merged file. Exits non-zero (set -e)
         # if the requested cluster is missing — no silent fallback.
-        #
-        # This prints a KUBECONFIG CHAIN ("<per-invocation selector>:<shared merged config>"), not a
-        # single path. Export it verbatim; do not split it or take only the last element, or this
-        # invocation loses its private default context and inherits whatever is shared.
         KUBECONFIG="$(/work/infra/devcontainer/kubeconfig.sh "${CLUSTER}")"
         export KUBECONFIG
       fi

infra/devcontainer/kubeconfig.sh

@@ -1,10 +1,9 @@
 #!/usr/bin/env bash
 #
 # infra/devcontainer/kubeconfig.sh — build ONE merged kubeconfig that can address BOTH clusters
-# under unambiguous names, and select which of them is current WITHOUT mutating shared state.
+# under unambiguous names, and select which of them is current.
 #
-#   kubeconfig.sh <dev|prod>      # regenerate the merged file, mint a per-invocation selector,
-#                                 # print the KUBECONFIG *chain* to use (selector:merged)
+#   kubeconfig.sh <dev|prod>      # regenerate the merged file, set its current-context, print its path
 #
 # WHY THIS EXISTS — the name collision that makes the obvious fix dangerous
 #   infra/ansible/.cache/dev.kubeconfig  and  .cache/prod.kubeconfig  each name their context, their
@@ -22,47 +21,17 @@
 #   file. The two renamed copies are then merged by kubectl itself, which is safe once the names are
 #   distinct.
 #
-# WHY THE MERGED FILE HAS NO `current-context` — the concurrency bug this shape fixes
-#   This script used to stamp `current-context: k3s-<target>` into the ONE shared merged file. That
-#   file lives in the repo's .cache, which every dev.sh container bind-mounts, so it is shared
-#   mutable state across concurrent invocations. Two `dev.sh` runs with different CLUSTER values
-#   therefore raced: `CLUSTER=prod dev.sh run <script>` would start on prod, a `CLUSTER=dev` run
-#   would rewrite `current-context` to k3s-dev mid-script, and the second half of the prod script
-#   would silently read the DEV cluster. This was hit live; it was caught only by luck (a PVC name
-#   that could not exist on prod). Silent cross-cluster reads — and, worse, cross-cluster WRITES —
-#   are the exact failure mode this file exists to prevent.
-#
-#   The fix removes the shared mutable field entirely instead of trying to lock it:
-#     * the merged file carries BOTH contexts and NO current-context. Read on its own it resolves
-#       to nothing and kubectl errors with "current-context is not set" — loud, never a wrong
-#       cluster. `--context k3s-dev` / `--context k3s-prod` against it keep working unchanged, and
-#       that remains the pattern to prefer.
-#     * the default context is supplied by a tiny PER-INVOCATION selector file — literally just
-#       `current-context: k3s-<target>`, no credentials — created under a private mktemp dir, and
-#       handed back FIRST in a KUBECONFIG chain: "<selector>:<merged>". kubectl merges the chain
-#       first-wins, so the selector decides this invocation's default and nothing else's.
-#   Nothing in the chain is shared and writable at once, so there is no lock to take and no window
-#   to lose. A concurrent regeneration can only rewrite the merged file with byte-identical
-#   content, and that write is done by atomic rename so a reader never sees a half-written file.
-#   As a bonus, `kubectl config use-context` inside an interactive shell writes to the FIRST file
-#   in the chain — the per-invocation selector — so even a manual context switch stays local to
-#   that container instead of leaking into everyone else's session.
-#
 # OUTPUT
-#   stdout: the KUBECONFIG chain "<selector>:<merged>" — export it verbatim, do not split it.
-#   merged: ${KUBECONFIG_CACHE_DIR:-<repo>/infra/ansible/.cache}/clusters.kubeconfig, mode 0600.
-#     That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is
-#     ignored repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed.
-#     It is a BUILD ARTIFACT: delete it any time, this script rebuilds it.
-#   selector: a ~90-byte file in a 0700 mktemp dir (override the parent with
-#     KUBECONFIG_SELECTOR_DIR). It holds no credentials. It is deliberately NOT cleaned up on exit,
-#     because the caller needs it for the whole life of its shell; inside dev.sh's `--rm` container
-#     it dies with the container. Run standalone on a host it leaves one tiny dir behind.
+#   ${KUBECONFIG_CACHE_DIR:-<repo>/infra/ansible/.cache}/clusters.kubeconfig, mode 0600.
+#   That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is ignored
+#   repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed. It is a
+#   BUILD ARTIFACT: delete it any time, this script rebuilds it.
 #
 # TOLERANCE / FAILURE
 #   A cluster whose source kubeconfig is absent is simply left out of the merged file (so a machine
 #   that only has dev still works). But asking for a cluster whose kubeconfig is absent is a hard
-#   ERROR — never a silent fallback to the other cluster.
+#   ERROR — never a silent fallback to the other cluster. Silently landing on the wrong cluster is
+#   the exact failure this script exists to prevent.
 #
 # Requires: kubectl, yq (both baked into the dev container image). Normally invoked by dev.sh from
 # inside the container; it runs fine anywhere those two binaries exist.
@@ -105,8 +74,7 @@ fi
 mkdir -p "${CACHE_DIR}"
 tmp="$(mktemp -d)"
 chmod 700 "${tmp}"
-staged=""
-trap 'rm -rf "${tmp}"; [ -n "${staged}" ] && rm -f "${staged}"; true' EXIT
+trap 'rm -rf "${tmp}"' EXIT
 
 parts=()
 present=()
@@ -116,16 +84,13 @@ for c in "${CLUSTERS[@]}"; do
   ctx="k3s-${c}"; usr="k3s-${c}-admin"
   # Structural rename. Every kubeconfig k3s emits here has exactly one cluster/user/context, but
   # [] indexing renames all of them consistently rather than assuming index 0 exists.
-  # No `current-context` is set on the parts, and none survives into the merged file: the merged
-  # file must stay identical no matter which cluster was asked for, because it is SHARED across
-  # concurrent invocations. Selection happens in the per-invocation selector below.
   ctx="$ctx" usr="$usr" yq '
       .clusters[].name = env(ctx)
     | .users[].name    = env(usr)
     | .contexts[].name = env(ctx)
     | .contexts[].context.cluster = env(ctx)
     | .contexts[].context.user    = env(usr)
-    | del(."current-context")
+    | ."current-context" = env(ctx)
   ' "$s" > "${tmp}/${c}.yaml"
   chmod 600 "${tmp}/${c}.yaml"
   parts+=("${tmp}/${c}.yaml")
@@ -135,49 +100,19 @@ done
 [ "${#parts[@]}" -gt 0 ] || die "no kubeconfigs found under ${CACHE_DIR}"
 
 # Names are distinct now, so kubectl's merge cannot silently drop one. --raw keeps the embedded
-# credential data (a merged config without it authenticates as nobody). `view` re-emits an empty
-# `current-context: ""` even when no part had one; drop the key so the file is unambiguous.
+# credential data (a merged config without it authenticates as nobody).
 KUBECONFIG="$(IFS=:; echo "${parts[*]}")" kubectl config view --flatten --raw > "${tmp}/merged.yaml"
-yq -i 'del(."current-context")' "${tmp}/merged.yaml"
 chmod 600 "${tmp}/merged.yaml"
+KUBECONFIG="${tmp}/merged.yaml" kubectl config use-context "k3s-${TARGET}" >/dev/null
 
-# Publish the merged file by ATOMIC RENAME, not by writing in place. Concurrent invocations produce
-# byte-identical content, but `install`/`>` truncate first, so an unlucky concurrent reader could
-# otherwise load a half-written kubeconfig. rename(2) within one filesystem swaps it whole.
-staged="$(mktemp "${CACHE_DIR}/.clusters.kubeconfig.XXXXXX")"
-cat "${tmp}/merged.yaml" > "${staged}"
-chmod 600 "${staged}"
-mv -f "${staged}" "${OUT}"
+install -m 600 "${tmp}/merged.yaml" "${OUT}"
 
-# Post-condition on the SHARED file: both renamed contexts present, nothing called `default`, and
-# no current-context to race over.
+# Post-condition: the requested context must exist and be current, and must NOT be `default`.
+cur="$(KUBECONFIG="${OUT}" kubectl config current-context)"
+[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'"
 if KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx default; then
   die "internal: a context named 'default' survived the rename — refusing to hand back an ambiguous config"
 fi
-KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx "k3s-${TARGET}" \
-  || die "internal: context 'k3s-${TARGET}' is missing from ${OUT}"
-if KUBECONFIG="${OUT}" kubectl config current-context >/dev/null 2>&1; then
-  die "internal: ${OUT} carries a current-context — that field is shared mutable state and must stay unset"
-fi
 
-# --- per-invocation selector: this invocation's default context, private to this invocation ------
-# `kubectl config set` (a config subcommand, never sed) writes the single scalar. It does not
-# validate the name, which is why the context's existence was asserted against the merged file
-# above.
-sel_dir="$(mktemp -d "${KUBECONFIG_SELECTOR_DIR:-${TMPDIR:-/tmp}}/kubeconfig-sel.XXXXXX")"
-chmod 700 "${sel_dir}"
-SEL="${sel_dir}/selector.kubeconfig"
-printf 'apiVersion: v1\nkind: Config\n' > "${SEL}"
-chmod 600 "${SEL}"
-KUBECONFIG="${SEL}" kubectl config set current-context "k3s-${TARGET}" >/dev/null
-
-CHAIN="${SEL}:${OUT}"
-
-# Post-condition on the CHAIN — the thing the caller will actually export. This is the end-to-end
-# assertion that this invocation defaults to the cluster that was asked for.
-cur="$(KUBECONFIG="${CHAIN}" kubectl config current-context)"
-[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'"
-
-printf '    kubeconfig.sh: merged [%s] -> %s (no current-context; shared)\n' "${present[*]}" "${OUT}" >&2
-printf '    kubeconfig.sh: this invocation defaults to %s via %s\n' "${cur}" "${SEL}" >&2
-echo "${CHAIN}"
+printf '    kubeconfig.sh: merged [%s] -> %s (current-context: %s)\n' "${present[*]}" "${OUT}" "${cur}" >&2
+echo "${OUT}"

infra/workstation/dnsmasq/20-prod.home.keithjackson.dev.conf

@@ -0,0 +1,49 @@
+# Workstation split-horizon DNS for the k3s PROD cluster.
+#
+# INSTALL TO: /etc/NetworkManager/dnsmasq.d/20-prod.home.keithjackson.dev.conf
+#   NOT /etc/dnsmasq.d/ — the standalone dnsmasq.service is INACTIVE on this workstation. The live
+#   resolver is the one NetworkManager spawns (NetworkManager.conf `dns=dnsmasq`), and it reads
+#   ONLY --conf-dir=/etc/NetworkManager/dnsmasq.d. A file dropped in /etc/dnsmasq.d/ changes nothing
+#   and looks like it should — verify with: pgrep -a dnsmasq
+#
+# WHY THIS FILE EXISTS
+#   The dev profile (10-home.keithjackson.dev.conf) contains:
+#       address=/.home.keithjackson.dev/192.168.2.50
+#   dnsmasq's `address=` matches the named domain AND EVERY SUBDOMAIN, so that one line also
+#   captures the entire `prod.home.keithjackson.dev` tree and answers it with DEV's address. Every
+#   prod hostname therefore resolved to dev from this workstation, while Cloudflare correctly
+#   returned prod's gateway.
+#
+#   That failure is deceptive because it PARTIALLY works: dev serves a `*.home.keithjackson.dev`
+#   wildcard certificate, which matches `prod.home.keithjackson.dev` (one label) — so the prod apex
+#   returns 200 over valid TLS — but NOT `keycloak.prod.home.keithjackson.dev` (two labels), which
+#   fails with a certificate-name error. It reads like a broken prod certificate. It is not; it is
+#   the wrong server answering. NEVER verify prod's DNS or TLS through this workstation's resolver
+#   without confirming these rules are installed.
+#
+#   dnsmasq resolves `address=` by LONGEST MATCH, so the more specific rules below win over the dev
+#   line without editing or reordering it. The dev profile stays untouched.
+#
+# TOPOLOGY (corrected 2026-09-05 by the operator — the earlier "pfSense override" framing was wrong)
+#   The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network,
+#   192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24 (the dev LAN).
+#   This workstation is on 192.168.2.0/24 too, so it reaches prod's VIPs DIRECTLY — no NAT hop,
+#   which is why these addresses are prod's real VIPs and not a forwarded address like dev's
+#   192.168.2.50.
+#
+#   The SOHO router NATs inbound 443 to the pfSense WAN address (192.168.2.102) to serve the
+#   EXISTING, NON-K3S production environment. That NAT is untouched by this file: these rules are
+#   local to this workstation's resolver only and change nothing about inbound traffic. The legacy
+#   prod stays live on 443 until the k3s cutover is deliberately made.
+
+# The prod cluster's shared ingress Gateway (Cilium LB-IPAM; == CILIUM_LB_POOL_START).
+# Covers the apex and every service hostname not overridden below.
+address=/prod.home.keithjackson.dev/192.168.2.20
+
+# L4 (non-HTTP) services get their OWN LoadBalancer VIPs, so they must NOT be sent to the Gateway.
+# These are longer matches than the line above and therefore take precedence.
+address=/unifi.prod.home.keithjackson.dev/192.168.2.21
+address=/crafty-mc.prod.home.keithjackson.dev/192.168.2.22
+
+# NOTE: no `local=` line is needed. The dev profile's `local=/home.keithjackson.dev/` already stops
+# the whole tree — including prod.home.* — from being forwarded upstream.

infra/workstation/dnsmasq/install.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# install.sh — install this repo's workstation dnsmasq profiles and reload the live resolver.
+#
+# Needs root (writes /etc/NetworkManager/dnsmasq.d). Run it as:
+#     sudo infra/workstation/dnsmasq/install.sh
+#
+# WHY A SCRIPT AND NOT A DOC LINE: these rules decide which CLUSTER this workstation talks to. Getting
+# them into the wrong directory is silent — the file exists, looks installed, and changes nothing —
+# so the destination and the reload are pinned here rather than retyped. Idempotent; safe to re-run.
+#
+set -euo pipefail
+
+src_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# THE LIVE RESOLVER IS NETWORKMANAGER'S dnsmasq, not the standalone service. NetworkManager.conf sets
+# `dns=dnsmasq`, and the process it spawns runs with `--conf-file=/dev/null` plus
+# `--conf-dir=/etc/NetworkManager/dnsmasq.d` — so /etc/dnsmasq.d is read ONLY by the standalone
+# dnsmasq.service, which is inactive here. Installing to the wrong one is the failure this pins down.
+dest_dir=/etc/NetworkManager/dnsmasq.d
+
+[ "$(id -u)" = "0" ] || { echo "ERROR: must run as root — try: sudo $0" >&2; exit 1; }
+
+if ! pgrep -af 'dnsmasq.*NetworkManager' >/dev/null 2>&1; then
+  echo "WARN: no NetworkManager-spawned dnsmasq is running." >&2
+  echo "      Check 'dns=dnsmasq' in /etc/NetworkManager/NetworkManager.conf before trusting this." >&2
+fi
+
+install -d -m 0755 "${dest_dir}"
+shopt -s nullglob
+count=0
+for f in "${src_dir}"/*.conf; do
+  install -m 0644 "$f" "${dest_dir}/$(basename "$f")"
+  echo "installed: ${dest_dir}/$(basename "$f")"
+  count=$((count+1))
+done
+[ "${count}" -gt 0 ] || { echo "ERROR: no .conf files found in ${src_dir}" >&2; exit 1; }
+
+# dnsmasq does NOT re-read its config on SIGHUP (SIGHUP only reloads /etc/hosts and the caches), so a
+# reload of NetworkManager — which respawns dnsmasq — is what actually applies these files.
+echo "reloading NetworkManager (respawns dnsmasq with the new config)"
+systemctl reload NetworkManager || systemctl restart NetworkManager
+
+echo
+echo "verify (expect prod's own VIPs, NOT dev's 192.168.2.50):"
+echo "  getent hosts gitea.prod.home.keithjackson.dev     # -> 192.168.2.20"
+echo "  getent hosts unifi.prod.home.keithjackson.dev     # -> 192.168.2.21"
+echo "  getent hosts gitea.home.keithjackson.dev          # -> 192.168.2.50  (dev, unchanged)"

platform-config/kustomization.yaml

@@ -43,11 +43,6 @@ resources:
   - ../platform/longhorn/volumesnapshotclass.yaml
   # monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
   - ../platform/monitoring/alerts.yaml
-  # Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
-  # (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
-  # was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
-  # file header. (2026-09-06, from the CNPG WAL/archiving incident.)
-  - ../platform/monitoring/pvc-capacity-alerts.yaml
   # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
   # operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
   # installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
@@ -78,21 +73,9 @@ resources:
   # keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
   # operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
   # ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
-  # cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
-  # CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
-  # identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
-  # clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
-  - ../platform/cnpg/alerts.yaml
   - ../platform/keycloak/postgres-cluster.yaml
   - ../platform/keycloak/keycloak-instance.yaml
   - ../platform/keycloak/keycloak-realm-import.yaml
-  # The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
-  # stored LITERALLY and the import still reports success, then rests at Done=True and
-  # never re-runs. This CronJob is the level-triggered loop that corrects that drift
-  # (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
-  # so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
-  # alongside the import whose failure mode it covers.
-  - ../platform/keycloak/client-secret-reconciler.yaml
   - ../platform/keycloak/k8s-oidc-rbac.yaml   # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
   - ../platform/keycloak/httproute.yaml
   - ../platform/keycloak/servicemonitor.yaml

platform/backup/minio-credentials.sops.yaml.example

@@ -2,6 +2,20 @@
 # TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
 # auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
 #
+# *** SUPERSEDED AS THE LIVE TARGET (2026-09-05), RETAINED ON PURPOSE. ***
+# Off-cluster DR moved to Cloudflare R2: prod physically cannot reach this MinIO (prod is on the SOHO
+# net 192.168.2.0/24, pfSense's WAN; the MinIO is inside pfSense's LAN — measured from a pod inside
+# prod: timed out), and this MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster"
+# backup shared a failure domain with dev. The R2 procedure is infra/BACKUP-R2.md.
+#
+# This file and its encrypted sibling minio-credentials.sops.yaml are KEPT because the MinIO copy of
+# the backups is NOT deleted and stays restorable: this is the key that reads the pre-migration
+# backups. Do NOT overwrite it with the R2 key — infra/BACKUP-R2.md §8 is how you point back here.
+#
+# DO NOT hand-fill this for R2. Four Secrets in three namespaces hold that one key, and
+# platform/backup/set-r2-credentials.sh writes all four with `sops set` — encrypted in place, so no
+# plaintext Secret ever exists in the working tree to be committed by accident.
+#
 # WHAT THIS IS: the scoped (non-root) MinIO S3 access key for the off-cluster backup target that
 # roles/minio_backup_target stands up on the Proxmox host (playbooks/backup-minio.yml). It is the
 # credential Velero (R-STATE-03) and CloudNativePG / Barman (R-IDP-05) use to write backups. The

platform/backup/set-r2-credentials.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+#
+# set-r2-credentials.sh — put ONE Cloudflare R2 API token into every credential Secret that consumes
+# the off-cluster backup target, without plaintext ever touching the disk.
+#
+# WHY THIS EXISTS
+#   The same S3 key is read by four different Secrets in three namespaces, in TWO different shapes
+#   (Velero wants an AWS INI file; CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys).
+#   Hand-editing four SOPS files is four chances to typo one of them — and a typo'd backup credential
+#   does not fail at apply time, it fails at 03:00 with no backup and a green HelmRelease.
+#
+#   It uses `sops set`, which edits an ALREADY-ENCRYPTED file in place: the new value is encrypted as
+#   it is written and NO decrypted copy of the file is ever created. That is the whole point — the
+#   `cp template plaintext.yaml && edit && sops -e -i` dance leaves a plaintext Secret sitting in the
+#   working tree, one `git add -A` away from being committed forever.
+#
+# USAGE (inside the dev container — it has sops and mounts the age key)
+#   ./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
+#     → prompts for the two values (never echoed, never in shell history, never in a file)
+#
+#   ./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
+#     → touches nothing; just reports whether the four Secrets already agree with each other
+#
+#   R2_ACCESS_KEY_ID=… R2_SECRET_ACCESS_KEY=… …/set-r2-credentials.sh --no-prompt
+#     → non-interactive. Prefer the prompts: an env var is visible in /proc and in `history`.
+#
+# WHAT IT DOES NOT DO
+#   - It does not touch clusters/base/cluster-vars.yaml. The endpoint is not a secret and belongs in
+#     a reviewable plain manifest; the script prints the exact edit to make.
+#   - It does not commit. Review `git diff` (you will see only ENC[...] blobs change) and commit.
+#   - It does not delete or rewrite platform/backup/minio-credentials.sops.yaml. That is the RETAINED
+#     pre-migration MinIO key and it is what you need to read the pre-migration backups.
+#
+# SEE  infra/BACKUP-R2.md for the whole procedure, including the restore that actually proves it.
+set -euo pipefail
+
+# Belt and braces: never let a stray `set -x` in a parent shell echo a credential out of this script.
+set +x
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+root="$(git -C "${here}" rev-parse --show-toplevel 2>/dev/null || (cd "${here}/.." && pwd))"
+cd "${root}"
+
+MODE=set
+PROMPT=1
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --check)     MODE=check; shift ;;
+    --no-prompt) PROMPT=0; shift ;;
+    -h|--help)   sed -n '2,40p' "${BASH_SOURCE[0]}"; exit 0 ;;
+    *)           echo "ERROR: unknown argument '$1'" >&2; exit 2 ;;
+  esac
+done
+
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+say()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
+
+command -v sops >/dev/null 2>&1 || die "'sops' not found — run this inside ./infra/devcontainer/dev.sh."
+
+# ---------------------------------------------------------------- the consumers
+# file|shape.  shape=ini   -> stringData.cloud, an AWS INI credentials file (velero-plugin-for-aws)
+#              shape=discrete -> stringData.ACCESS_KEY_ID + ACCESS_SECRET_KEY (CNPG barmanObjectStore)
+CONSUMERS=(
+  "platform/velero/velero-cloud.sops.yaml|ini"
+  "platform/keycloak/keycloak-db-backup-credentials.sops.yaml|discrete"
+  "apps/wave-0/nextcloud-db-backup-credentials.sops.yaml|discrete"
+  "apps/wave-0/semaphore-db-backup-credentials.sops.yaml|discrete"
+)
+
+for entry in "${CONSUMERS[@]}"; do
+  f="${entry%%|*}"
+  [ -f "${f}" ] || die "missing consumer file ${f} — has the layout changed? Fix this script, do not skip the file."
+  grep -q '^sops:' "${f}" || die "${f} has no top-level sops: block — refusing to touch a file that is not SOPS-encrypted."
+done
+
+# A 32-bit fingerprint of a high-entropy secret: enough to prove two files hold the SAME value,
+# nowhere near enough to recover it. Values themselves are NEVER printed.
+fp() { local h; h="$(printf '%s' "$1" | sha256sum)"; printf '%s' "${h:0:8}"; }
+
+# Read one value out of an encrypted file without ever writing plaintext anywhere.
+xtract() { sops -d --extract "$2" "$1"; }
+
+# Pull the two values out of an AWS INI blob using shell only.
+ini_field() { # <ini-text> <aws_access_key_id|aws_secret_access_key>
+  local line v=""
+  while IFS= read -r line; do
+    case "${line}" in
+      "$2"=*|"$2 "*|*[![:alnum:]_]"$2"=*) v="${line#*=}"; v="${v#"${v%%[![:space:]]*}"}" ;;
+    esac
+  done <<<"$1"
+  printf '%s' "${v}"
+}
+
+report() { # print the fingerprint table; returns non-zero when the four files disagree
+  local entry f shape id sec akid_fp sec_fp first_id="" first_sec="" agree=0
+  printf '    %-58s %-10s %-10s\n' FILE ID-FP SECRET-FP
+  for entry in "${CONSUMERS[@]}"; do
+    f="${entry%%|*}"; shape="${entry##*|}"
+    if [ "${shape}" = ini ]; then
+      local blob; blob="$(xtract "${f}" '["stringData"]["cloud"]' 2>/dev/null || true)"
+      id="$(ini_field "${blob}" aws_access_key_id)"
+      sec="$(ini_field "${blob}" aws_secret_access_key)"
+      blob=""
+    else
+      id="$(xtract "${f}" '["stringData"]["ACCESS_KEY_ID"]' 2>/dev/null || true)"
+      sec="$(xtract "${f}" '["stringData"]["ACCESS_SECRET_KEY"]' 2>/dev/null || true)"
+    fi
+    [ -n "${id}" ]  || { warn "${f}: could not read the access key id (age key available?)"; agree=1; }
+    [ -n "${sec}" ] || { warn "${f}: could not read the secret access key"; agree=1; }
+    akid_fp="$(fp "${id}")"; sec_fp="$(fp "${sec}")"
+    printf '    %-58s %-10s %-10s\n' "${f}" "${akid_fp}" "${sec_fp}"
+    if [ -z "${first_id}" ]; then first_id="${akid_fp}"; first_sec="${sec_fp}"
+    elif [ "${akid_fp}" != "${first_id}" ] || [ "${sec_fp}" != "${first_sec}" ]; then agree=1; fi
+    id=""; sec=""
+  done
+  return "${agree}"
+}
+
+if [ "${MODE}" = check ]; then
+  say "current state of the four backup credential Secrets (values never printed)"
+  if report; then
+    info "ALL FOUR AGREE. (Agreeing is not the same as being right — prove it with bootstrap/standup.sh s3check.)"
+    exit 0
+  fi
+  die "the four Secrets do NOT all hold the same key. Velero and CNPG would be writing to the same
+       target with different credentials — one of them is stale. Re-run this script without --check."
+fi
+
+# ---------------------------------------------------------------- the two values
+AKID="${R2_ACCESS_KEY_ID-}"
+SAK="${R2_SECRET_ACCESS_KEY-}"
+if [ "${PROMPT}" = 1 ]; then
+  say "paste the R2 API token (Cloudflare dashboard -> R2 -> Manage API Tokens)"
+  info "Object Read & Write, scoped to the velero + cnpg buckets. Neither value is echoed or stored."
+  [ -n "${AKID}" ] || { printf '    Access Key ID: ';     IFS= read -rs AKID; printf '\n'; }
+  [ -n "${SAK}" ]  || { printf '    Secret Access Key: '; IFS= read -rs SAK;  printf '\n'; }
+fi
+[ -n "${AKID}" ] || die "no access key id given (prompt, or R2_ACCESS_KEY_ID with --no-prompt)."
+[ -n "${SAK}" ]  || die "no secret access key given (prompt, or R2_SECRET_ACCESS_KEY with --no-prompt)."
+
+# FAIL CLOSED on anything that is not a plain S3 credential. These values are passed to `sops set` as
+# JSON, so a quote, a backslash or a newline would corrupt the Secret rather than error — and the
+# usual cause is a copy-paste that swallowed a newline or a trailing space, which would produce a
+# credential that authenticates nowhere and a backup that silently never runs.
+for pair in "access key id:${AKID}" "secret access key:${SAK}"; do
+  case "${pair#*:}" in
+    *[!A-Za-z0-9._~+/=-]*) die "the ${pair%%:*} contains a character outside [A-Za-z0-9._~+/=-].
+       Almost certainly a stray space, quote or newline from the paste. Nothing was changed." ;;
+  esac
+done
+# R2 issues 32-hex ids and 64-hex secrets today. A mismatch is a warning, not a refusal: the format is
+# Cloudflare's to change, and refusing outright would make this script the thing that blocks a rotation.
+case "${AKID}" in *[!0-9a-f]*|"") warn "the access key id is not 32 lowercase hex chars — unusual for R2. Continuing." ;;
+  *) [ "${#AKID}" = 32 ] || warn "the access key id is ${#AKID} chars, not the usual 32. Continuing." ;;
+esac
+[ "${#SAK}" -ge 40 ] || warn "the secret access key is only ${#SAK} chars — unusual for R2 (64). Continuing."
+
+# ---------------------------------------------------------------- write
+say "writing the credential into ${#CONSUMERS[@]} SOPS Secrets (encrypted in place; no plaintext file)"
+for entry in "${CONSUMERS[@]}"; do
+  f="${entry%%|*}"; shape="${entry##*|}"
+  case "${shape}" in
+    ini)
+      # velero-plugin-for-aws reads this key VERBATIM as an AWS credentials file. The [default]
+      # profile name and the two underscore_names are the plugin's contract, not a style choice.
+      sops set "${f}" '["stringData"]["cloud"]' \
+        "\"[default]\\naws_access_key_id=${AKID}\\naws_secret_access_key=${SAK}\\n\"" ;;
+    discrete)
+      sops set "${f}" '["stringData"]["ACCESS_KEY_ID"]'     "\"${AKID}\""
+      sops set "${f}" '["stringData"]["ACCESS_SECRET_KEY"]' "\"${SAK}\"" ;;
+  esac
+  info "updated ${f}"
+done
+AKID_FP="$(fp "${AKID}")"; SAK_FP="$(fp "${SAK}")"
+AKID=""; SAK=""
+
+say "verifying by decrypting each file back (values never printed)"
+report || die "the files still disagree after writing — do NOT commit. Inspect with 'sops <file>'."
+info "every file now reads id-fp=${AKID_FP} secret-fp=${SAK_FP} — matching what was pasted."
+
+say "NEXT — in this order, and do not skip the restore"
+cat <<'NEXT'
+    1. Set the endpoint (NOT a secret, so it lives in a plain reviewable manifest):
+         clusters/base/cluster-vars.yaml
+           S3_BACKUP_ENDPOINT: "https://<account-id>.r2.cloudflarestorage.com"
+           S3_BACKUP_REGION:   "auto"          # R2 accepts nothing else
+    2. Prove the target BEFORE merging anything (writes and deletes one throwaway object):
+         bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
+           --s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
+    3. Only then commit + merge, and follow infra/BACKUP-R2.md through the RESTORE.
+       A backup that has never been restored is not a backup, and R-STATE-03 is currently PROVEN
+       against MinIO only.
+NEXT

platform/cnpg/alerts.yaml

@@ -1,182 +0,0 @@
----
-# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
-#
-# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
-# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
-# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
-# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
-# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
-#
-# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
-# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
-# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
-# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
-# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
-# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
-# warning for a storage outage. The generic PVC-fill backstop is the separate
-# platform/monitoring/pvc-capacity-alerts.yaml.
-#
-# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
-# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
-# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
-# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
-# operator's Prometheus select these rules (ruleSelector). It IS referenced from
-# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
-# failure this repo keeps hitting.
-#
-# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
-# 2026-09-06). CNPG's exporter publishes TWO independent families:
-#
-#   * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
-#     and the backup catalogue). Present for EVERY instance.
-#   * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
-#
-# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
-# exporter cannot log in to its own database:
-#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)  [user cnpg_metrics_exporter]
-# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
-# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
-# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
-# itself so the blind spot cannot recur unnoticed.
-#
-# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
-#   cnpg_collector_pg_wal_archive_status{value="ready"}   prod: keycloak 17, nextcloud 7, semaphore 3
-#                                                          dev: 0, 0, 0  (min over 24h also 0)
-#   cnpg_collector_first_recoverability_point             prod: 0, 0, 0
-#                                                          dev: 1788018752 / 1788018762 / 1788018770
-#   cnpg_collector_last_available_backup_timestamp        prod: 0, 0, 0   dev: 0.1-3.1h old
-#   cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64  dev: 36 vs 64  (neither fires yet)
-#   cnpg_last_error                                       prod+dev: nextcloud-db = 1
-#
-# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
-# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
-# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
-# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
-# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
-# which fires only when an outage is genuinely imminent.
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: platform-health-cnpg
-  namespace: monitoring
-  labels:
-    release: kube-prometheus-stack
-    platform-health: "true"
-spec:
-  groups:
-    - name: platform-health.cnpg
-      rules:
-        # ---------------------------------------------------------------------------------------
-        # THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
-        # the hour instead of 38 hours later.
-        #
-        # pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
-        # archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
-        # seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
-        # segment is 16 MiB that Postgres may not recycle.
-        #
-        # THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
-        # condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
-        # queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
-        # returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
-        # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
-        # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
-        # backlog sits at 3 because it writes almost nothing.
-        - alert: CNPGWALArchiveBacklog
-          expr: |
-            max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
-          for: 1h
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
-            description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
-        # ---------------------------------------------------------------------------------------
-        # The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
-        # time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
-        # is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
-        # be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
-        # and an old base backup can exist while archiving has just broken (only the other fires).
-        #
-        # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
-        # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
-        # start quiet while still catching a bootstrap that never produced a recoverability point.
-        - alert: CNPGNoRecoverabilityPoint
-          expr: |
-            max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
-        # ---------------------------------------------------------------------------------------
-        # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
-        # platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
-        # databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
-        # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
-        # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
-        # ~57 years and always true). 36h = one missed daily run plus slack.
-        - alert: CNPGNoRecentBaseBackup
-          expr: |
-            (time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
-            and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0
-          for: 2h
-          labels:
-            severity: warning
-          annotations:
-            summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
-        # ---------------------------------------------------------------------------------------
-        # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
-        # which means checkpoints are no longer reclaiming space — the mechanism that actually fills
-        # the PVC. Both series come from the same metric under different `value` labels, so each side
-        # is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
-        # label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
-        # both clusters), `count` = segments currently in pg_wal.
-        #
-        # This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
-        # CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
-        # to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
-        # at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
-        # runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
-        # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
-        # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
-        # therefore NOT usable as a signal.
-        - alert: CNPGWALVolumeNotRecycling
-          expr: |
-            max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
-              > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
-          for: 2h
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
-            description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
-        # ---------------------------------------------------------------------------------------
-        # THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
-        # queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
-        # `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
-        #
-        # THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
-        # 2026-09-06: `cnpg_errors_total` carries
-        #   pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
-        #   database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
-        # i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
-        # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
-        # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
-        # permanently-firing `critical` this repo has been burned by.
-        - alert: CNPGMetricsCollectorErroring
-          expr: |
-            max by (namespace, pod) (cnpg_last_error) == 1
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"

platform/externaldns/helmrelease.yaml

@@ -39,24 +39,33 @@ spec:
     # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
     # Full write-up: infra/PROD-PROVISIONING.md §10.
     #
-    # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
-    #   1. txtOwnerId       — per-environment identity; a cluster only ever touches records it owns.
-    #   2. domainFilters    — per-environment name scope; prod cannot even see names outside its
-    #                         own subtree. Defence in depth, added after the incident.
+    # THE TWO GUARDS THAT ACTUALLY PREVENT A RECURRENCE — note that neither is domainFilters:
+    #   1. txtOwnerId  — per-environment identity; a cluster only ever touches records it owns.
+    #   2. DISJOINT HOSTNAME SETS — the two environments serve different suffixes, so they never
+    #      describe the same record in the first place. dev = *.dev.home.keithjackson.dev,
+    #      prod = *.home.keithjackson.dev (prod IS the apex; operator decision 2026-09-04).
     # ---------------------------------------------------------------------------------------------
-    # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
-    # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
-    # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
-    # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
-    # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
-    # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
-    # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
-    # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
+    # Which names this instance may manage at all. BOTH ENVIRONMENTS RENDER THE CLOUDFLARE **ZONE**
+    # `keithjackson.dev` HERE, and this is NOT an oversight to be tightened.
+    #
+    # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for ZONE
+    # SELECTION as well as record matching, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+    # matches NO zone — external-dns generates the record and then skips it with "no hosted zone
+    # matching record DNS Name", publishing NOTHING. This has been "improved" and reverted more than
+    # once (an agent narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in
+    # review). It is not defence in depth; it is a total outage of DNS publication. Isolation comes
+    # from the two guards above, which a domain filter cannot provide anyway — both environments live
+    # in the ONE zone regardless of what this says. See clusters/base/cluster-vars.yaml.
     domainFilters:
       - "${EXTERNALDNS_DOMAIN_FILTER}"
     # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
-    # caused the incident. Dev renders `default` (its live records are already registered under that
-    # owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
+    # caused the incident. dev renders `dev`, prod renders `prod`, and the BASE default is a
+    # non-colliding sentinel so a new environment that forgets the knob owns nothing and can delete
+    # nothing. NOTHING renders `default` any more (the external-dns chart default, and the value both
+    # clusters silently shared): dev held it only because its live apex records were registered under
+    # it, and dev's move to *.dev.home.keithjackson.dev retired that constraint along with those
+    # records. Prod must NOT be given `default` even though it inherits the apex NAMES — that would
+    # hand prod ownership of dev's old apex records and let `policy: sync` below delete them.
     txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
     policy: sync            # create + update + delete records as services come and go — but ONLY
                             # records this owner id owns AND this domainFilter admits (see above).

platform/keycloak/README.md

@@ -29,7 +29,6 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
 | `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
 | `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
 | `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
-| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
 | `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
 | `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
 
@@ -48,14 +47,10 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
 
 ## Operator hand-offs (fill before / during deploy)
 
-1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
-   `<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
-   import injects them into the Keycloak clients via `spec.placeholders`. Add to
-   `kustomization.yaml` once filled.
-   **The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
-   from it, and CI rejects any other name.
-   *(This step used to read "the realm import stays PENDING until this exists". It does not — see
-   the next section. That false belief is what allowed the 2026-09-06 outage.)*
+1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. Two client
+   secrets (`grafana-client-secret`, `oauth2-proxy-client-secret`), `openssl rand -hex 32` each. The
+   realm import injects them into the Keycloak clients via `spec.placeholders`. Add to
+   `kustomization.yaml` once filled. The realm import stays PENDING until this exists.
 2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
    `client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
 3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
@@ -65,45 +60,6 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
 import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
 exact keys + steps. No plaintext secret is ever committed.
 
-## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
-
-**The realm import is a one-shot that fails open.** Measured on both clusters:
-
-* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
-  text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
-  error condition, and `HasErrors` stays `False`.
-* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
-  added later is never picked up.
-
-Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
-every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
-on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
-signal was green throughout, including the conformance test that asserts `Done=True`.
-
-Three layers now cover it:
-
-| Layer | Where | What it does |
-|---|---|---|
-| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
-| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
-| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
-
-The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
-ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
-no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
-
-**Manual repair / one-off run** (the CronJob normally does this):
-
-```sh
-kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
-kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
-```
-
-**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
-redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
-its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
-landed.
-
 ## Exceptions that keep native auth (do not force SSO) — ADR-0014
 
 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).

platform/keycloak/client-secret-reconciler.yaml

@@ -1,729 +0,0 @@
----
-# =============================================================================
-# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04)
-#
-# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only
-# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients
-# Secret and the Keycloak Operator, and alongside the realm import itself.
-#
-# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06)
-# -------------------------------------------------------------------
-# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are
-# import placeholders resolved from the keycloak-realm-clients Secret via
-# spec.placeholders. When the operator cannot resolve a placeholder it stores the
-# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
-# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and
-# NEVER RE-RUNS, so nothing ever corrects it:
-#
-#   * prod: all 12 clients held the literal placeholder. Every OIDC login on the
-#     cluster failed with "Invalid client or Invalid client credentials".
-#   * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added
-#     AFTER dev's realm import had already completed, and a completed import is
-#     never re-run, so the new key was never picked up.
-#
-# Both were repaired live via the admin API on 2026-09-06 and verified. This
-# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is
-# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same
-# posture Flux gives every other resource in this repo (R-GITOPS-03 drift
-# correction, applied to the one piece of state Flux itself cannot reach --
-# Keycloak's own database).
-#
-# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph
-# already guarantees the Secret is applied before the realm import (the Secret is
-# rendered by ./platform; the import by ./platform-config, which dependsOn
-# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of
-# the Secret lands before revision N of the import). Ordering still cannot help,
-# because the operator will not re-run a completed import when a LATER revision
-# adds a key -- which is precisely how dev broke. Only a reconciler closes that.
-#
-# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state
-# lives in Keycloak's Postgres database, not in a Kubernetes object, so no
-# admission or mutation policy can see it. The admin REST API is the only
-# interface to it.
-#
-# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token,
-# client lookup, client-secret read, minimal client PUT). Users, groups, roles,
-# authentication flows and realm settings are structurally unreachable from it --
-# prod's `homelab` realm carries a real user and the groups media-users /
-# platform-admins / viewers, and they must survive every run. It never logs a
-# secret, and it writes only where the stored value differs from git.
-#
-# PROOF: the script's own `--selftest` drives these real code paths against an
-# in-process stub Keycloak and asserts compare-then-write, idempotence (a second
-# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint
-# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script
-# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships:
-# conformance/tests/l0-static/keycloak-client-secret-reconciler/.
-# =============================================================================
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: keycloak-client-secret-reconciler
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak-client-secret-reconciler
-    app.kubernetes.io/part-of: keycloak
-data:
-  # SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy:
-  # the L0 gate extracts THIS block and executes its --selftest, so a second copy
-  # would be untested. Contains no dollar sign at all, on purpose -- this file is
-  # rendered by a Flux Kustomization with postBuild.substituteFrom, so an
-  # unescaped dollar-brace expansion here would be rewritten before any container
-  # ran (the trap documented at length in apps/wave-1/nextcloud.yaml).
-  reconcile-client-secrets.py: |
-    #!/usr/bin/env python3
-    # =============================================================================
-    # reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret
-    # equal the value git declares for it, idempotently, without touching anything
-    # else in the realm.
-    #
-    # WHY THIS EXISTS (the incident it prevents)
-    # ------------------------------------------
-    # keycloak-realm-import.yaml declares each confidential client's secret as an
-    # `(env:<NAME>_CLIENT_SECRET)` import placeholder and maps <NAME> to a key of the
-    # SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the
-    # Keycloak Operator cannot resolve a placeholder at import time it stores the
-    # LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
-    # SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and
-    # never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12
-    # clients with secrets held the literal placeholder and every OIDC login failed
-    # with "Invalid client or Invalid client credentials"; on dev exactly one
-    # (oauth2-proxy) was broken, because that Secret key was added after dev's import
-    # had already completed.
-    #
-    # So the import is a ONE-SHOT that fails OPEN. This is the level-triggered
-    # reconciler that closes it.
-    #
-    # CONTRACT
-    # --------
-    #   Desired state  files in SECRETS_DIR named `<clientId>-client-secret`, each
-    #                  holding exactly the secret for that clientId (this is the
-    #                  `keycloak-realm-clients` Secret, mounted).
-    #   Actual state   GET /admin/realms/<realm>/clients/<uuid>/client-secret
-    #   Action         only where they differ: PUT /admin/realms/<realm>/clients/<uuid>
-    #                  with the MINIMAL representation {"clientId":..,"secret":..}.
-    #
-    # WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a
-    # two-field PUT rewrites the secret and nothing else. PUTting the full
-    # round-tripped representation back has a far larger blast radius and is in fact
-    # rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the
-    # JSON"} (measured during the live repair). The stub in --selftest models that
-    # rejection, so a regression to the fat PUT fails in CI, not on a live realm.
-    #
-    # SAFETY PROPERTIES, each enforced in code and proved by --selftest:
-    #   1. compare-then-write: a client already correct is SKIPPED, no PUT issued.
-    #   2. idempotent: a second run over a converged realm issues zero writes.
-    #   3. no secret is ever printed, in any mode, on any path.
-    #   4. the ONLY endpoints this program may touch are the master-realm token
-    #      endpoint, the client LOOKUP, the client-secret READ and the minimal client
-    #      PUT. Every other URL is refused by _http() before a socket is opened, so
-    #      users, groups, roles, authentication flows and realm settings are
-    #      structurally unreachable from here -- prod's `homelab` realm carries a
-    #      real user and the groups media-users / platform-admins / viewers, and they
-    #      MUST survive.
-    #   5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped.
-    #
-    # MODES
-    #   (default)    reconcile: compare, and write only what differs.
-    #   --audit      READ-ONLY. Report drift and exit 1 if any client's stored secret
-    #                differs from the declared one. Never writes. This is what the L2
-    #                conformance test runs.
-    #   --selftest   the offline behavioural proof, against an in-process stub
-    #                Keycloak. No cluster; loopback only. This is what the L0 static
-    #                gate runs, so the code the gate proves is the code that ships.
-    #
-    # NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor
-    # (hence re.fullmatch throughout), and not even inside these comments. It ships
-    # inside a ConfigMap that the `platform-config` Flux Kustomization renders with
-    # postBuild.substituteFrom, so Flux envsubst rewrites this text before any
-    # container runs: an unescaped dollar-brace expansion would be silently emptied at
-    # reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's
-    # OIDC guard dead code for months -- read the comment block there. Python needs no
-    # dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and
-    # the character is banned outright so no future edit has to reason about escaping.
-    # Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh.
-    # =============================================================================
-    from __future__ import annotations
-
-    import hmac
-    import json
-    import os
-    import re
-    import sys
-    import time
-    import urllib.error
-    import urllib.parse
-    import urllib.request
-
-    SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients")
-    KEY_SUFFIX = "-client-secret"
-    TIMEOUT = 20
-    READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24"))
-    READY_SLEEP = float(os.environ.get("READY_SLEEP", "5"))
-
-    # Endpoint allow-list (safety property 4). fullmatch-ed against the path of every
-    # URL _http() is asked to open; anything else raises before a socket exists.
-    _ALLOWED = (
-        r"/realms/master/protocol/openid-connect/token",
-        r"/admin/realms/[^/]+/clients",
-        r"/admin/realms/[^/]+/clients/[^/]+",
-        r"/admin/realms/[^/]+/clients/[^/]+/client-secret",
-    )
-
-
-    class Refused(RuntimeError):
-        """A URL outside the allow-list was requested. Never expected in a normal run."""
-
-
-    def _http(method, base, path, query=None, headers=None, form=None, body=None):
-        """Return (status, text). Refuses any path outside the allow-list."""
-        if not any(re.fullmatch(p, path) for p in _ALLOWED):
-            raise Refused("endpoint not on the allow-list: %s %s" % (method, path))
-        url = base.rstrip("/") + path
-        if query:
-            url = url + "?" + urllib.parse.urlencode(query)
-        hdrs = dict(headers or {})
-        data = None
-        if form is not None:
-            data = urllib.parse.urlencode(form).encode()
-            hdrs["Content-Type"] = "application/x-www-form-urlencoded"
-        elif body is not None:
-            data = json.dumps(body).encode()
-            hdrs["Content-Type"] = "application/json"
-        req = urllib.request.Request(url, data=data, headers=hdrs, method=method)
-        try:
-            with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
-                return resp.status, resp.read().decode("utf-8", "replace")
-        except urllib.error.HTTPError as e:
-            return e.code, e.read().decode("utf-8", "replace")
-
-
-    def declared(secrets_dir):
-        """{clientId: secret} from the mounted Secret.
-
-        Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal
-        entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A
-        key must be named `<clientId>-client-secret`: that naming contract is what
-        lets this reconciler derive the client set from git without a second
-        declaration, and the L0 static check enforces it against
-        keycloak-realm-import.yaml so the two can never drift apart.
-        """
-        out = {}
-        for name in sorted(os.listdir(secrets_dir)):
-            if name.startswith(".") or not name.endswith(KEY_SUFFIX):
-                continue
-            path = os.path.join(secrets_dir, name)
-            if not os.path.isfile(path):
-                continue
-            with open(path, "r", encoding="utf-8") as fh:
-                value = fh.read()
-            # A trailing newline in the SOPS source would otherwise become part of the
-            # secret and mismatch forever. Keycloak secrets never contain whitespace.
-            out[name[: -len(KEY_SUFFIX)]] = value.strip()
-        return out
-
-
-    def admin_token(base, user, password, log):
-        """Bearer token for admin-cli in the master realm, with a bounded readiness wait."""
-        last = ""
-        for attempt in range(1, READY_ATTEMPTS + 1):
-            try:
-                st, text = _http(
-                    "POST", base, "/realms/master/protocol/openid-connect/token",
-                    form={"client_id": "admin-cli", "grant_type": "password",
-                          "username": user, "password": password})
-            except OSError as e:
-                st, text = 0, "connection error (%s)" % e.__class__.__name__
-            if st == 200:
-                token = json.loads(text).get("access_token")
-                if token:
-                    return token
-                last = "http 200 but no access_token in the response"
-            else:
-                # Deliberately do NOT echo the response body: it is harmless today, but
-                # the request that produced it carried the admin password, and keeping
-                # that class of mistake impossible by habit is cheaper than reviewing it.
-                last = "http %s" % st
-            log("waiting for the Keycloak admin API (attempt %d/%d): %s"
-                % (attempt, READY_ATTEMPTS, last))
-            if attempt < READY_ATTEMPTS:
-                time.sleep(READY_SLEEP)
-        raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s"
-                           % (READY_ATTEMPTS, last))
-
-
-    def reconcile(base, realm, token, want, audit, log):
-        """Return (changed, problems). Writes only when audit is False."""
-        hdrs = {"Authorization": "Bearer " + token}
-        changed = 0
-        problems = []
-        for cid in sorted(want):
-            st, text = _http("GET", base, "/admin/realms/%s/clients" % realm,
-                             query={"clientId": cid}, headers=hdrs)
-            if st != 200:
-                problems.append("%s: client lookup failed (http %s)" % (cid, st))
-                continue
-            found = json.loads(text)
-            if not isinstance(found, list) or not found:
-                problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it "
-                                "but the realm has no such client (the operator runs the "
-                                "realm import exactly once)" % (cid, realm))
-                continue
-            client = found[0]
-            uuid = client.get("id")
-            if client.get("publicClient"):
-                log("%-14s skipped (public client, holds no secret)" % cid)
-                continue
-            st, text = _http("GET", base,
-                             "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
-                             headers=hdrs)
-            if st != 200:
-                problems.append("%s: could not read the stored secret (http %s)" % (cid, st))
-                continue
-            stored = json.loads(text).get("value") or ""
-            if hmac.compare_digest(stored, want[cid]):
-                log("%-14s ok (already correct, no write)" % cid)
-                continue
-            # Describe the drift WITHOUT quoting either value. The literal-placeholder
-            # case is named because it is the failure this program exists for, and
-            # saying so reveals nothing secret.
-            kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER"
-                    if "env:" in stored else "stored value differs from the declared one")
-            if audit:
-                problems.append("%s: DRIFT -- %s" % (cid, kind))
-                continue
-            log("%-14s DRIFT (%s) -- repairing" % (cid, kind))
-            # MINIMAL representation only -- see the header. Keycloak merges non-null
-            # fields, so protocol mappers, redirect URIs, flows, groups, roles and users
-            # are all untouched by this write.
-            st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid),
-                             headers=hdrs, body={"clientId": client.get("clientId"),
-                                                 "secret": want[cid]})
-            if not 200 <= st < 300:
-                problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160]))
-                continue
-            st, text = _http("GET", base,
-                             "/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
-                             headers=hdrs)
-            readback = (json.loads(text).get("value") or "") if st == 200 else ""
-            if not hmac.compare_digest(readback, want[cid]):
-                problems.append("%s: PUT was accepted but the readback still does not match"
-                                % cid)
-                continue
-            changed += 1
-            log("%-14s REPAIRED and verified by readback" % cid)
-        return changed, problems
-
-
-    def run(argv, log):
-        audit = "--audit" in argv
-        base = os.environ.get("KEYCLOAK_BASE_URL", "").strip()
-        realm = os.environ.get("KEYCLOAK_REALM", "").strip()
-        user = os.environ.get("KEYCLOAK_ADMIN_USER", "")
-        password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")
-        for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm),
-                            ("KEYCLOAK_ADMIN_USER", user),
-                            ("KEYCLOAK_ADMIN_PASSWORD", password)):
-            if not value:
-                log("FATAL: %s is empty" % name)
-                return 1
-        log("mode=%s realm=%s base=%s"
-            % ("audit (read-only)" if audit else "reconcile", realm, base))
-        try:
-            want = declared(SECRETS_DIR)
-        except OSError as e:
-            log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__))
-            return 1
-        if not want:
-            log("FATAL: no `<clientId>%s` keys found in %s -- the keycloak-realm-clients "
-                "Secret is missing or empty, and reporting success here would repeat the "
-                "very fail-open the realm import already has"
-                % (KEY_SUFFIX, SECRETS_DIR))
-            return 1
-        log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want))))
-        try:
-            token = admin_token(base, user, password, log)
-            changed, problems = reconcile(base, realm, token, want, audit, log)
-        except Refused as e:
-            log("FATAL: %s" % e)
-            return 1
-        except (RuntimeError, OSError, ValueError) as e:
-            log("FATAL: %s: %s" % (e.__class__.__name__, e))
-            return 1
-        if problems:
-            log("")
-            for p in problems:
-                log("PROBLEM %s" % p)
-            log("")
-            if audit:
-                log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems))
-            else:
-                log("RECONCILE FAIL: %d client secret(s) could not be reconciled"
-                    % len(problems))
-            return 1
-        if audit:
-            log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match "
-                "git" % len(want))
-        else:
-            log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired"
-                % (len(want), changed))
-        return 0
-
-
-    # =============================================================================
-    # --selftest -- the offline behavioural proof. It drives the REAL code paths above
-    # against an in-process stub Keycloak, so what the gate proves is what ships.
-    # =============================================================================
-    def _refused(path):
-        """True iff _http() refuses `path` outright (allow-list, safety property 4)."""
-        try:
-            _http("PUT", "http://127.0.0.1:1", path, body={})
-        except Refused:
-            return True
-        except OSError:
-            return False
-        return False
-
-
-    def _selftest():
-        import contextlib
-        import http.server
-        import io
-        import tempfile
-        import threading
-
-        global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP
-
-        SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
-        SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
-        SECRET_C = "cccccccccccccccccccccccccccccccc"
-        STALE = "0ld-r0tated-value"
-        ADMIN_PW = "stub-admin-password"
-        # The exact value the broken clusters stored. chr(36) is the dollar sign,
-        # spelled this way so the file stays free of the character envsubst reacts to.
-        LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)"
-
-        state = {"clients": {}, "writes": [], "touched": []}
-
-        class Stub(http.server.BaseHTTPRequestHandler):
-            def log_message(self, *a):
-                pass
-
-            def _send(self, code, obj):
-                payload = json.dumps(obj).encode()
-                self.send_response(code)
-                self.send_header("Content-Type", "application/json")
-                self.send_header("Content-Length", str(len(payload)))
-                self.end_headers()
-                self.wfile.write(payload)
-
-            def _route(self, method):
-                parsed = urllib.parse.urlparse(self.path)
-                path, qs = parsed.path, urllib.parse.parse_qs(parsed.query)
-                state["touched"].append("%s %s" % (method, path))
-                if path == "/realms/master/protocol/openid-connect/token":
-                    return self._send(200, {"access_token": "stub-token"})
-                if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path):
-                    cid = (qs.get("clientId") or [""])[0]
-                    c = state["clients"].get(cid)
-                    return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)])
-                m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret",
-                                 path)
-                if m and method == "GET":
-                    c = state["clients"].get(m.group(1))
-                    if c is None:
-                        return self._send(404, {"error": "not found"})
-                    return self._send(200, {"type": "secret", "value": c.get("secret", "")})
-                m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path)
-                if m and method == "PUT":
-                    raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
-                    rep = json.loads(raw or b"{}")
-                    # The live server rejects a full round-tripped representation with
-                    # 400 "Cannot parse the JSON"; modelled so a regression to the fat
-                    # PUT fails here rather than against a real realm.
-                    if set(rep) != {"clientId", "secret"}:
-                        return self._send(400, {
-                            "error": "unknown_error",
-                            "error_description": "Cannot parse the JSON"})
-                    state["writes"].append(m.group(1))
-                    state["clients"][m.group(1)]["secret"] = rep["secret"]
-                    return self._send(204, {})
-                return self._send(500, {"error": "an endpoint the reconciler must not touch"})
-
-            def do_GET(self):
-                self._route("GET")
-
-            def do_PUT(self):
-                self._route("PUT")
-
-            def do_POST(self):
-                self._route("POST")
-
-        srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub)
-        threading.Thread(target=srv.serve_forever, daemon=True).start()
-        base = "http://127.0.0.1:%d" % srv.server_address[1]
-
-        tmp = tempfile.mkdtemp()
-        for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B),
-                            ("kubernetes", SECRET_C), ("gitea", SECRET_A)):
-            with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh:
-                fh.write(secret + "\n")        # trailing newline: must be tolerated
-        # Kubernetes' own projection entries must be ignored, not read as clients.
-        os.mkdir(os.path.join(tmp, "..data"))
-
-        SECRETS_DIR = tmp
-        READY_ATTEMPTS, READY_SLEEP = 2, 0.01
-        os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab",
-                           "KEYCLOAK_ADMIN_USER": "admin",
-                           "KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW})
-
-        failures = []
-        checks = [0]
-
-        def expect(cond, what):
-            checks[0] += 1
-            if not cond:
-                failures.append(what)
-
-        def invoke(*argv):
-            state["writes"] = []
-            state["touched"] = []
-            buf = io.StringIO()
-            with contextlib.redirect_stdout(buf):
-                rc = run(list(argv), print)
-            return rc, buf.getvalue()
-
-        def reset():
-            state["clients"] = {
-                # already correct
-                "grafana": {"clientId": "grafana", "publicClient": False,
-                            "secret": SECRET_A},
-                # THE INCIDENT: the literal, unresolved import placeholder
-                "pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL},
-                # public client: no secret, must be skipped
-                "kubernetes": {"clientId": "kubernetes", "publicClient": True},
-                # ordinary drift (rotated in git, stale in the realm)
-                "gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE},
-            }
-
-        def leaked(text):
-            return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW)
-                    if s in text]
-
-        # -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING
-        reset()
-        rc, out = invoke("--audit")
-        expect(rc == 1, "audit must exit 1 when a stored secret differs from git")
-        expect(state["writes"] == [],
-               "audit mode must never write (writes=%r)" % state["writes"])
-        expect("UNRESOLVED IMPORT PLACEHOLDER" in out,
-               "audit must name the unresolved-placeholder case explicitly")
-        expect("pihole: DRIFT" in out and "gitea: DRIFT" in out,
-               "audit must report BOTH drifted clients, not stop at the first")
-        expect(re.search(r"grafana\s+ok \(already correct", out),
-               "an already-correct client must be reported as such")
-        expect(re.search(r"kubernetes\s+skipped \(public client", out),
-               "a public client must be skipped, not treated as drift")
-        expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out))
-
-        # -- 2. reconcile: repairs exactly the drifted clients, verified by readback
-        reset()
-        rc, out = invoke()
-        expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:])
-        expect(sorted(state["writes"]) == ["gitea", "pihole"],
-               "reconcile must write exactly the drifted clients, got %r" % state["writes"])
-        expect("grafana" not in state["writes"],
-               "compare-then-write: an already-correct client must not be written")
-        expect("kubernetes" not in state["writes"], "a public client must not be written")
-        expect("REPAIRED and verified by readback" in out,
-               "every repair must be readback-verified")
-        expect("2 repaired" in out, "the summary must count the repairs")
-        expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out))
-
-        # -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing
-        rc, out = invoke()
-        expect(rc == 0, "the second reconcile must exit 0")
-        expect(state["writes"] == [],
-               "IDEMPOTENCE: a converged realm must take ZERO writes, got %r"
-               % state["writes"])
-        expect("0 repaired" in out, "the second run must report 0 repaired")
-        rc, out = invoke("--audit")
-        expect(rc == 0, "audit must pass once the realm has converged")
-        expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out,
-               "a clean audit must emit the sentinel the L2 conformance test greps for")
-
-        # -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched
-        reset()
-        invoke()
-        for touched in state["touched"]:
-            expect(re.fullmatch(
-                r"POST /realms/master/protocol/openid-connect/token"
-                r"|GET /admin/realms/homelab/clients"
-                r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?",
-                touched), "the reconciler touched an off-list endpoint: %s" % touched)
-        expect(not [t for t in state["touched"]
-                    if "/users" in t or "/groups" in t or "/roles" in t
-                    or "/authentication" in t],
-               "the reconciler must never touch users/groups/roles/authentication")
-        expect(_refused("/admin/realms/homelab/users"),
-               "the allow-list must refuse a users endpoint before opening a socket")
-        expect(_refused("/admin/realms/homelab/groups"),
-               "the allow-list must refuse a groups endpoint before opening a socket")
-        expect(_refused("/admin/realms/homelab"),
-               "the allow-list must refuse a whole-realm update endpoint")
-
-        # -- 5. a declared client the realm never got is LOUD, not a silent pass
-        reset()
-        del state["clients"]["gitea"]
-        rc, out = invoke("--audit")
-        expect(rc == 1, "a declared client missing from the realm must fail the audit")
-        expect("NOT FOUND" in out, "the missing client must be named in the output")
-
-        # -- 6. an empty / unmounted Secret must fail closed, never report success
-        SECRETS_DIR = tempfile.mkdtemp()
-        rc, out = invoke()
-        expect(rc == 1, "an empty secrets dir must fail closed")
-        expect("keys found in" in out, "the empty-secret failure must say what is missing")
-        SECRETS_DIR = tmp
-
-        srv.shutdown()
-        print("")
-        if failures:
-            print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0]))
-            for f in failures:
-                print("  FAIL %s" % f)
-            return 1
-        print("SELFTEST OK: %d assertions (compare-then-write, idempotence, "
-              "no-secret-logging, endpoint allow-list, fail-closed)" % checks[0])
-        return 0
-
-
-    def main():
-        argv = sys.argv[1:]
-        if "--selftest" in argv:
-            return _selftest()
-        return run(argv, lambda m: print(m, flush=True))
-
-
-    if __name__ == "__main__":
-        sys.exit(main())
----
-# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key
-# added after the realm import already completed) converges within one interval
-# instead of waiting for a human to notice a failed login. Runs are cheap: a
-# converged realm performs 12 GETs and ZERO writes.
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: keycloak-client-secret-reconciler
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak-client-secret-reconciler
-    app.kubernetes.io/part-of: keycloak
-spec:
-  schedule: "*/15 * * * *"
-  # Forbid: a run that is still waiting for Keycloak must not be lapped by the next
-  # tick. Two concurrent reconcilers would be harmless (both converge to the same
-  # value) but would double the write attempts during an outage recovery.
-  concurrencyPolicy: Forbid
-  startingDeadlineSeconds: 300
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 600
-      ttlSecondsAfterFinished: 3600
-      template:
-        metadata:
-          labels:
-            app.kubernetes.io/name: keycloak-client-secret-reconciler
-            app.kubernetes.io/part-of: keycloak
-        spec:
-          restartPolicy: Never
-          # The reconciler talks ONLY to the Keycloak admin API. It needs no
-          # Kubernetes API access at all — its two inputs are mounted Secrets — so
-          # it gets no ServiceAccount token (R-SEC-02, least privilege).
-          automountServiceAccountToken: false
-          securityContext:
-            runAsNonRoot: true
-            runAsUser: 65532
-            runAsGroup: 65532
-            # fsGroup makes the projected Secret volume GROUP-owned by 65532. Without it the
-            # files are owned by root and mode 0400 (owner-only) is unreadable to runAsUser
-            # 65532 — the container died with "cannot read /etc/keycloak-realm-clients
-            # (PermissionError)" on every run. fsGroup + 0440 keeps the secret unreadable to
-            # other users rather than relaxing it to world-readable 0444.
-            fsGroup: 65532
-            seccompProfile:
-              type: RuntimeDefault
-          containers:
-            - name: reconciler
-              # Stock upstream python; the reconciler is pure standard library, so
-              # nothing is installed at runtime. Pinned exactly, like busybox /
-              # curlimages elsewhere in the repo. Renovate does not manage plain
-              # `image:` tags (renovate.json disables image-tag updates so it never
-              # fights Flux image-automation), so this is bumped by hand.
-              image: python:3.13.15-alpine3.23
-              imagePullPolicy: IfNotPresent
-              command:
-                - python3
-                - /opt/reconciler/reconcile-client-secrets.py
-              env:
-                - name: KEYCLOAK_BASE_URL
-                  value: http://keycloak-service.keycloak.svc.cluster.local:8080
-                - name: KEYCLOAK_REALM
-                  value: homelab
-                # readOnlyRootFilesystem is on; keep CPython from trying to write
-                # bytecode caches beside the read-only ConfigMap mount.
-                - name: PYTHONDONTWRITEBYTECODE
-                  value: "1"
-                - name: PYTHONUNBUFFERED
-                  value: "1"
-                # Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to
-                # bootstrap the admin; keys admin-user / admin-password.
-                - name: KEYCLOAK_ADMIN_USER
-                  valueFrom:
-                    secretKeyRef:
-                      name: keycloak-admin-credentials
-                      key: admin-user
-                - name: KEYCLOAK_ADMIN_PASSWORD
-                  valueFrom:
-                    secretKeyRef:
-                      name: keycloak-admin-credentials
-                      key: admin-password
-              volumeMounts:
-                - name: script
-                  mountPath: /opt/reconciler
-                  readOnly: true
-                # The DESIRED state: one file per declared client, named
-                # `<clientId>-client-secret`. Mounting the Secret (rather than
-                # reading it through the API) is what lets this pod run with no
-                # ServiceAccount token at all.
-                - name: client-secrets
-                  mountPath: /etc/keycloak-realm-clients
-                  readOnly: true
-              securityContext:
-                allowPrivilegeEscalation: false
-                readOnlyRootFilesystem: true
-                capabilities:
-                  drop:
-                    - ALL
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 48Mi
-                limits:
-                  memory: 128Mi
-          volumes:
-            - name: script
-              configMap:
-                name: keycloak-client-secret-reconciler
-                defaultMode: 0444
-            - name: client-secrets
-              secret:
-                secretName: keycloak-realm-clients
-                # 0440 (not 0400): readable by fsGroup 65532, which the pod securityContext
-                # sets. 0400 is owner-only and the owner is root, not the 65532 the container
-                # runs as. See the fsGroup comment above.
-                defaultMode: 0440

platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml

@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: keycloak-db-backup-credentials-prod
-    namespace: keycloak
-type: Opaque
-stringData:
-    ACCESS_KEY_ID: ENC[AES256_GCM,data:kQ5x5fXb6KdVaDsItGug2cpQMZw9tSvbsYDqTMxoP+A=,iv:dCTfwVxlpnjaMhGBIMa18d4yAiQ6SKhb4EMm1p1JyQ8=,tag:sNqMLs+rGQm/1MyV90InDQ==,type:str]
-    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:9lGvwvseAWK1bMXDXZJ+bjw4oHH/XVPi8+C8vfXGJ3jX8VW95rJrczPbEfV9rCudVIJFnxAdaQZZWaJCsC8fCQ==,iv:GOvPs2I77yiKh381bcv0kCwT5xbk8Fao6ctPCW2aCHg=,tag:hSjWjaZ+nEJwZyv3sIuq7A==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBN2NvVXJ5TW1mQlhYWnRC
-            UlVLVjFtNk1Wdkk1NzdicXBQVGNtc3JZclNBCjl3UlVTZmFJWGJGam54VytVR0lZ
-            djB3ZjBCOFFNOUFMZGxCcnZiZ3VDMUUKLS0tIFhDSWR3ZXZPOHFzVGNhZGhqdUQv
-            WmF0NlNIeTJCeVV0ZGkxSFJIL3BqNlEKntEpxCdpy7Xd16ZKe6skQYL6hCqYOAIN
-            NtwtQ9oF/8i4i0T7B8kcORB71cu8dPu6w6LtaSC+1qBV5TBSSH5U6Q==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:dcgB6z9BDzaft4do6f1SqjCqmQopc5mM0ckfU/HfE09eOSIRjpoZ+qMzJW8RIwUZNKrjEAwrrAEtrN67GRxL0jOEsw/6xNrWJRFv6KTeE3ECy5qMO4jLCQTeMvZuvfZE7iqwvHCCnjhkqkedEAD4Hwefq0c0EZ0ZxZK3z1tVdHk=,iv:Ue+ZJq9dJl9qbjJ1l8Q24VmsK+KW/uSRFu04xG6/tM8=,tag:MvJ6Ww3h3b/C2PqA/62abA==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

platform/keycloak/kustomization.yaml

@@ -28,17 +28,9 @@ resources:
   - keycloak-admin-credentials.sops.yaml
   # --- OIDC client secrets (grafana + oauth2-proxy) the realm import injects via placeholders ---
   - keycloak-realm-clients.sops.yaml
-  # --- PER-ENVIRONMENT S3 credential for the keycloak-db CNPG Barman backup (R-IDP-05), split
-  #     2026-09-07. SOPS-encrypted, decrypted by the `platform` tier (which already has the
-  #     decryption stanza). BOTH land in ns keycloak on BOTH clusters; the keycloak-db Cluster
-  #     (platform-config) selects ONE by name via
-  #     `keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "",
-  #     prod "-prod"). Two objects, so an R2 token rotation cannot silently break dev's MinIO
-  #     WAL archiving — and archive_mode=on makes a broken archive a disk-full clock, not a
-  #     degraded backup (see postgres-cluster.yaml).
-  #       dev  -> keycloak-db-backup-credentials       (MinIO, s3://cnpg)
-  #       prod -> keycloak-db-backup-credentials-prod  (Cloudflare R2, s3://k3s-prod-backup/barman)
+  # --- Scoped MinIO S3 credential for the keycloak-db CNPG Barman backup (R-IDP-05) — SOPS-encrypted,
+  #     decrypted by the `platform` tier (which already has the decryption stanza). Lands in ns
+  #     keycloak; the keycloak-db Cluster (platform-config) reads it by name for WAL archiving. ---
   - keycloak-db-backup-credentials.sops.yaml
-  - keycloak-db-backup-credentials-prod.sops.yaml
   # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
   - grafana-dashboard.yaml

platform/keycloak/postgres-cluster.yaml

@@ -34,43 +34,29 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → object storage ---
+  # --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
   # Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
-  # PVC snapshot for a live DB). CNPG appends `serverName` (defaults to the Cluster name) to
-  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/keycloak-db/ — dev
-  # s3://cnpg/keycloak-db/ on the on-Proxmox MinIO, prod s3://k3s-prod-backup/barman/keycloak-db/ on
-  # Cloudflare R2. Both endpointURL and destinationPath are cluster-vars knobs (R-ENV-02),
-  # substituted by the `platform-config` Kustomization (this file sits under platform/keycloak/ but
-  # platform-config is the only tier that lists it as a resource — see its kustomization.yaml).
+  # PVC snapshot for a live DB). destinationPath is the BUCKET ROOT; CNPG appends serverName (defaults
+  # to the Cluster name), so objects land under s3://cnpg/keycloak-db/. Path-style HTTP MinIO endpoint.
   # Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
-  # (a Pod only reads same-ns Secrets).
-  #
-  # *** archive_mode IS ON, SO A FAILING ARCHIVE COMMAND IS A DISK-FULL CLOCK, NOT A DEGRADED
-  # BACKUP. *** Postgres will not recycle a WAL segment it has not successfully archived, so
-  # whenever these two values are wrong the pgdata PVC fills monotonically until the primary
-  # PANICs. This is not hypothetical: prod ran from bootstrap to 2026-09-07 with
-  # `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT` (barman-cloud-wal-archive exit 4 on every segment)
-  # and had burned ~579 MiB of a 5 GiB PVC on the way to taking SSO — and with it kubectl OIDC,
-  # Grafana, Headlamp, Gitea and forward-auth — down with it. Never leave a placeholder here, and
-  # treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one.
+  # (a Pod only reads same-ns Secrets), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000) — prod
+  # MUST point at its own off-cluster target. Substituted by the `platform-config` Kustomization.
   backup:
     retentionPolicy: "30d"          # keep 30 days of base backups + WAL (PITR window); prunes older
     barmanObjectStore:
+      # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
+      # NAME — and both environments have a `keycloak-db`. Sharing one MinIO with a shared path
+      # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
+      # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
       destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
-      # PER-ENVIRONMENT credential (2026-09-07): dev reads keycloak-db-backup-credentials (MinIO),
-      # prod reads keycloak-db-backup-credentials-prod (Cloudflare R2). Both Secrets are applied to
-      # both clusters by platform/keycloak/kustomization.yaml; ${BACKUP_CREDENTIALS_SUFFIX} (dev ""
-      # | prod "-prod") picks one. Before the split these were ONE shared Secret, so writing prod's
-      # R2 token into it would have left dev signing MinIO requests with an R2 key — a 403 on every
-      # archive command, which under archive_mode=on is the disk-full clock described above, not a
-      # missing backup.
       s3Credentials:
         accessKeyId:
-          name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: keycloak-db-backup-credentials
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: keycloak-db-backup-credentials
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

platform/monitoring/alerts.yaml

@@ -247,37 +247,8 @@ spec:
         # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
         # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
         # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
-        #
-        # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
-        # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
-        # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
-        # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
-        # to have a tunnel, via two independent guards OR'd together:
-        #
-        #   1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
-        #      CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
-        #      steering datapath has a target. Requires the kube-state-metrics node-label allowlist
-        #      enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
-        #      what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
-        #      manual re-enable to forget. The Node object survives a NotReady node, so this stays true
-        #      while the gateway node is DOWN: gateway-node-down is still alerted.
-        #   2. `tun0` was observed at all in the last hour — holds the alert armed through a
-        #      kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
-        #      of the KSM rollout.
-        #
-        # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
-        # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
-        # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
-        # dev (tunnel up), and clause 2 already true on dev.
         - alert: VPNTunnelDown
-          expr: |
-            absent(node_network_receive_bytes_total{device="tun0"})
-            and on()
-            (
-                (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
-              or
-                (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
-            )
+          expr: absent(node_network_receive_bytes_total{device="tun0"})
           for: 5m
           labels:
             severity: critical
@@ -388,56 +359,6 @@ spec:
             summary: "Keycloak database unavailable"
             description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
-        # --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
-        # WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
-        # had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
-        # signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
-        # KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
-        # Done=True passed). The only observable was a human failing to log in. The CronJob is
-        # the healer; these two rules make its SILENCE observable, so the class cannot sit
-        # undetected again. They are the runtime counterpart of the L2 audit test.
-        #
-        # UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
-        # these series had no samples to check against. `kube_job_status_failed` /
-        # `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
-        # import Job exports them); the reconciler's own series appear with its first run.
-        # Confirm with: kube_job_status_succeeded{namespace="keycloak",
-        # job_name=~"keycloak-client-secret-reconciler-.*"}
-        - alert: KeycloakClientSecretReconcilerFailing
-          # A reconciler run failed. Either it found drift it could not repair, or it could not
-          # reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
-          # deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
-          # cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
-          expr: |
-            max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
-          for: 15m
-          labels:
-            severity: warning
-          annotations:
-            summary: "Keycloak client-secret reconciler failing"
-            description: >-
-              A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
-              secret may differ from git (unresolved import placeholder, or an unrepairable
-              drift) and SSO logins for that client will fail. Read the Job log: it names the
-              client and never prints a secret.
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
-        - alert: KeycloakClientSecretReconcilerStale
-          # No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
-          # CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
-          # the healer being quietly absent, which is exactly the "looks applied and isn't"
-          # failure this whole change exists to end.
-          expr: |
-            absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
-          for: 2h
-          labels:
-            severity: warning
-          annotations:
-            summary: "Keycloak client-secret reconciler has not succeeded"
-            description: >-
-              No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
-              The CronJob may be suspended, unapplied, or unable to schedule — client-secret
-              drift would go uncorrected and unnoticed (docs/15).
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)

platform/monitoring/kube-prometheus-stack.yaml

@@ -221,17 +221,6 @@ spec:
         limits:
           memory: 256Mi
     kube-state-metrics:
-      # Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
-      # have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
-      # `kube_node_labels` does not exist and any join against it silently matches nothing.
-      # Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
-      # cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
-      # — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
-      # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
-      # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
-      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
-      metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
       resources:
         requests:
           cpu: 20m

platform/monitoring/pvc-capacity-alerts.yaml

@@ -1,148 +0,0 @@
----
-# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
-#
-# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
-# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
-# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
-# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
-# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
-# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
-# the SSO database with nothing watching it.
-#
-# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
-# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
-# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
-# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
-# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
-# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
-# working on both clusters.
-#
-# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
-# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
-# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
-# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
-# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
-# from platform-config/kustomization.yaml.
-#
-# ------------------------------------------------------------------------------------------------
-# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
-#
-# Two facts, both established live rather than assumed:
-#
-#  1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
-#     DIFFERENT labels for the same PVC:
-#        job="kubelet"    -> namespace=<the PVC's real namespace>, node=<node>
-#        job="apiserver"  -> namespace="default", exported_namespace=<the PVC's real namespace>
-#     Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
-#     only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
-#     rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
-#     `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
-#     pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
-#     say namespace="default") and this file WOULD double-fire on them — and would report the wrong
-#     namespace while doing it.
-#
-#  2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
-#     covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
-#     only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
-#     which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
-#
-# ------------------------------------------------------------------------------------------------
-# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
-#
-# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
-# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
-# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
-# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
-# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
-# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
-# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
-# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
-# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
-# it, unifi-rootdisk is the only match at every threshold.
-#
-# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
-# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
-# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
-# at 88% free.
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: platform-health-pvc-capacity
-  namespace: monitoring
-  labels:
-    release: kube-prometheus-stack
-    platform-health: "true"
-spec:
-  groups:
-    - name: platform-health.pvc-capacity
-      rules:
-        # Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
-        # (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
-        # scope differs.
-        - alert: PersistentVolumeFillingUp
-          expr: |
-            (
-              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
-                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
-            ) < 0.15
-            and kubelet_volume_stats_inodes{job="kubelet"} > 0
-          for: 15m
-          labels:
-            severity: warning
-          annotations:
-            summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
-            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
-        # Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
-        # volume is genuinely about to run out, which for a Postgres data volume means an imminent
-        # disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
-        # permanently-firing critical. There is no counterpart for this tier anywhere else in the
-        # repo, so it cannot duplicate one.
-        - alert: PersistentVolumeAlmostFull
-          expr: |
-            (
-              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
-                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
-            ) < 0.05
-            and kubelet_volume_stats_inodes{job="kubelet"} > 0
-          for: 15m
-          labels:
-            severity: critical
-          annotations:
-            summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
-            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
-        # TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
-        # for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
-        # ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
-        # surprise. This rule instead asks "is this volume on course to hit zero?".
-        #
-        # 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
-        # rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
-        # horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
-        # hour, so a single burst of writes cannot notify on its own.
-        #
-        # The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
-        # not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
-        # dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
-        # platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
-        # every OTHER PVC in the cluster, which is what was missing entirely.
-        #
-        # NOTE the monitoring namespace is excluded here too (partition rule above). That also
-        # side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
-        # retention high-water mark looks exactly like an unbounded leak until it plateaus.
-        - alert: PersistentVolumeFillingUpPredicted
-          expr: |
-            (
-              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
-                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
-            ) < 0.50
-            and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
-            and kubelet_volume_stats_inodes{job="kubelet"} > 0
-          for: 1h
-          labels:
-            severity: warning
-          annotations:
-            summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
-            description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"

platform/velero/README.md

@@ -9,17 +9,43 @@
 
 ## What this is (LIVE)
 
-Flux `HelmRelease` (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox **MinIO** S3 target:
+Flux `HelmRelease` (Velero chart 12.x / Velero 1.18.x) wired to whatever `${S3_BACKUP_ENDPOINT}` names.
+The target is a **cluster-vars knob, not a constant**:
 
-- **Endpoint:** `http://192.168.1.9:9000` (plain HTTP, **path-style**, region `us-east-1` placeholder)
-- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, `s3ForcePathStyle=true`.
-- **AWS plugin** (`velero/velero-plugin-for-aws`) loaded as an init container.
+- **Endpoint:** `${S3_BACKUP_ENDPOINT}` — dev has been running against the on-Proxmox **MinIO**
+  `http://192.168.1.9:9000`; as of **2026-09-05** the authored target is **Cloudflare R2**
+  (`https://<account-id>.r2.cloudflarestorage.com`). The switch-on procedure, the R2 gotchas and the
+  honest proven/unproven split all live in **[`infra/BACKUP-R2.md`](../../infra/BACKUP-R2.md)**.
+  Always **path-style**; region `${S3_BACKUP_REGION}` (MinIO ignores it, R2 requires exactly `auto`).
+- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, prefix
+  `${VELERO_BACKUP_PREFIX}` (dev/prod — both clusters share one R2 endpoint and are isolated by path
+  only), `s3ForcePathStyle=true`, **`checksumAlgorithm: ""`**.
+  That last one is load-bearing: `velero-plugin-for-aws` v1.14.2 defaults it to `CRC32` and R2 rejects
+  AWS checksums with HTTP 400 `InvalidArgument`. Do not remove it, and re-check it on every plugin bump.
+- **AWS plugin** (`velero/velero-plugin-for-aws:v1.14.2`) loaded as an init container.
 - **File System Backup (kopia):** `deployNodeAgent=true`, `uploaderType=kopia`, `snapshotsEnabled=false`
-  — storage-agnostic PVC backup, **no** CSI snapshot / data-mover.
+  — storage-agnostic PVC backup, **no** CSI snapshot / data-mover. This path uploads through kopia's
+  **own** S3 client, which does not read `checksumAlgorithm`: metadata compatibility and volume-data
+  compatibility with a given object store are two separate questions.
 - **Credentials:** `velero-cloud-credentials` — a SOPS Secret (`velero-cloud.sops.yaml`) holding the
-  `cloud` INI key, the **scoped** MinIO key limited to the `velero`+`cnpg` buckets. Kept in sync with
-  the staging source of truth `platform/backup/minio-credentials.sops.yaml` and the Proxmox host
-  ansible-vault.
+  `cloud` INI key, the **scoped** key limited to the `velero`+`cnpg` buckets. Four Secrets across three
+  namespaces hold this one key; `platform/backup/set-r2-credentials.sh` writes all four at once with
+  `sops set` (no plaintext ever on disk) and proves they agree.
+  `platform/backup/minio-credentials.sops.yaml` retains the **pre-migration MinIO** key — that is
+  deliberate, it is how the pre-migration backups stay readable.
+
+## Proving the target before trusting it
+
+```bash
+bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
+  [--s3-endpoint <candidate>] s3check
+```
+
+`s3check` runs a throwaway in-cluster pod per credential and proves credential load + authentication,
+bucket existence and listability under this environment's prefix, and a real
+**PUT → GET → byte-compare → DELETE → confirm-gone** round trip — plus whether the R2 checksum trap is
+present. It fails closed and loud; it never reports "unknown, carrying on". Pod egress is the only
+reading that counts (prod sits on pfSense's WAN, the MinIO inside its LAN).
 
 Reconciled by its **own non-gating** Flux Kustomization `clusters/base/backup.yaml`
 (`dependsOn: platform-config`, `wait:false`) — a backup controller must never gate the platform graph.
@@ -46,7 +72,13 @@ PITR to the `cnpg` bucket — see the DB manifests.)
 
 ## Post-deploy (PM)
 
-- `velero backup-location get` → `default` Available.
-- `velero backup create --from-schedule daily` → objects land under `s3://velero/`.
+- `velero backup-location get` → `default` Available. **Available only means the BSL could be read.**
+- `velero backup create --from-schedule daily` → Completed, then read `velero backup logs <name>` and
+  look for `PodVolumeBackup` failures specifically (see the kopia note above). Objects land under
+  `s3://velero/${VELERO_BACKUP_PREFIX}/`.
 - DR drill (R-STATE-03, Phase-4 gate): `velero restore create --from-backup <name>` into a scratch
   namespace and verify PVC data restores via the node-agent.
+
+**R-STATE-03 is PROVEN against MinIO only.** A restore from R2 has never been executed in this repo,
+and Velero-against-R2 is unproven here — `infra/BACKUP-R2.md` §5 is the re-proof sequence and §7 is the
+honest list of what is and is not established.

platform/velero/helmrelease.yaml

@@ -1,29 +1,18 @@
 ---
 # Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
 #
-# TARGET: the S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars) at
-# s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX}, in region ${S3_BACKUP_REGION}. Four knobs,
-# because the two environments' object stores differ in shape as well as address:
-#   dev   http://192.168.1.9:9000  bucket `velero`, NO prefix, region us-east-1  (on-Proxmox MinIO,
-#                                  plain HTTP, path-style; MinIO ignores the region)
-#   prod  https://<acct>.r2.cloudflarestorage.com  bucket `k3s-prod-backup`, prefix `velero`,
-#                                  region `auto`   (Cloudflare R2; prod cannot route to that MinIO,
-#                                  and R2 demands exactly `auto`)
-# Prod's bucket is SHARED with CNPG/barman (the operator created one bucket), so the `velero` prefix
-# is what keeps this controller's retention sweep away from the Postgres WAL stream. See the
-# "OBJECT-STORE TENANCY" block in clusters/base/cluster-vars.yaml.
+# TARGET: the MinIO S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars; dev http://192.168.1.9:9000 on
+# Proxmox — plain HTTP, path-style), bucket
+# `velero`. region us-east-1 is a placeholder MinIO ignores (S3 clients still require one).
 #
 # WHY THIS SHAPE:
 #  - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
 #  - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
 #    NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
 #    Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
-#  - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero). ONE
-#    Secret serves both environments, and it now carries the Cloudflare R2 API token (written by
-#    platform/backup/set-r2-credentials.sh, 2026-09-07) rather than the scoped MinIO key. That is
-#    safe only because the R2 token is bucket-scoped to k3s-prod-backup, which dev never names:
-#    dev's Velero would fail loudly against its MinIO endpoint with an R2 key, it could not
-#    silently write somewhere unexpected. Rotating either target means re-running that script.
+#  - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero) — the
+#    SCOPED MinIO key limited to the velero+cnpg buckets. Kept in sync with the staging source of
+#    truth platform/backup/minio-credentials.sops.yaml and the Proxmox host ansible-vault.
 #
 # VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
 # cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
@@ -83,45 +72,31 @@ spec:
       backupStorageLocation:
         - name: default
           provider: aws
-          # Bucket and prefix are BOTH knobs (cluster-vars), because the two environments' object
-          # stores are shaped differently. Dev's MinIO has a dedicated `velero` bucket and dev's
-          # backups sit at its ROOT, so dev renders bucket=velero with an EMPTY prefix — and the
-          # chart omits `objectStorage.prefix` entirely when it is empty (verified, chart 12.1.0),
-          # leaving dev's live BackupStorageLocation byte-unchanged. Prod's operator created ONE R2
-          # bucket shared with CNPG/barman, so prod renders bucket=k3s-prod-backup with prefix
-          # `velero` — a disjoint top-level prefix from barman's. Getting this wrong is not a
-          # cosmetic error: Velero ADOPTS every backup it finds under its location and TTL-DELETES
-          # it on retention, so a prefix that overlaps another tenant deletes that tenant's data.
-          bucket: ${VELERO_BACKUP_BUCKET}
-          prefix: "${VELERO_BACKUP_PREFIX}"
+          bucket: velero
+          # PER-ENVIRONMENT PREFIX (cluster-vars). Empty on dev = the bucket root it already uses;
+          # `prod` on prod. Without it both clusters share one BackupStorageLocation — and Velero
+          # SYNCS backups it finds there into itself as Backup CRs and GARBAGE-COLLECTS them when
+          # their TTL expires, so prod would adopt and then delete dev's backups. Both clusters also
+          # run schedules named `daily`/`daily-resources-only`, so even the object keys would collide.
+          prefix: ${VELERO_BACKUP_PREFIX}
           default: true
           config:
-            # Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores
-            # whatever it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1`
-            # was correct-by-accident and broke the moment the target stopped being MinIO.
+            # Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores whatever
+            # it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1` silently broke
+            # the moment the target stopped being MinIO.
             region: ${S3_BACKUP_REGION}
-            s3Url: ${S3_BACKUP_ENDPOINT}      # cluster-vars; dev MinIO (http) or prod R2 (https)
+            s3Url: ${S3_BACKUP_ENDPOINT}      # cluster-vars; R2 (https) or a MinIO/versitygw (http)
             s3ForcePathStyle: "true"          # MinIO is path-style; R2 accepts path-style too
             # *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
-            # velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums ON by
-            # default, so it sends `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32` on every
-            # PUT. R2 REJECTS those headers with HTTP 400 InvalidArgument, which would fail every
-            # backup AND every kopia blob upload. The empty string is the documented opt-out, and
-            # it must be PRESENT-but-empty rather than absent — the plugin branches on the key
-            # existing, and the chart does emit an empty value (verified, chart 12.1.0). Harmless
-            # against MinIO, which ignores it, so it is stated unconditionally rather than as a
-            # fourth knob. If backups start failing with InvalidArgument after a plugin bump, look
-            # here first.
+            # velero-plugin-for-aws v1.14+ bundles an AWS SDK that sends `x-amz-sdk-checksum-algorithm`
+            # / `x-amz-checksum-crc32` on every PUT. R2 REJECTS those headers (HTTP 400
+            # "InvalidArgument"), so every backup and every kopia blob upload fails. Setting this to the
+            # empty string disables the SDK's flexible-checksum behaviour. It is harmless against MinIO,
+            # which simply ignores it — so it is stated unconditionally rather than per-environment.
+            # If backups start failing with InvalidArgument after a plugin bump, look here first.
             checksumAlgorithm: ""
     credentials:
-      # PER-ENVIRONMENT credential (2026-09-07). dev renders velero-cloud-credentials (the scoped
-      # MinIO key); prod renders velero-cloud-credentials-prod (the Cloudflare R2 token). BOTH
-      # Secrets are applied to BOTH clusters by platform/velero/kustomization.yaml — the suffix
-      # only chooses which one this Deployment mounts. cluster-vars: BACKUP_CREDENTIALS_SUFFIX,
-      # dev "" | prod "-prod". A name that resolves to a MISSING Secret does not degrade the
-      # backup, it wedges the pod in ContainerCreating (the volume never mounts), which is why
-      # the base default is the empty suffix — the unsuffixed Secret always exists.
-      existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}   # SOPS Secret carrying the `cloud` INI key (ns velero)
+      existingSecret: velero-cloud-credentials   # SOPS Secret carrying the `cloud` INI key (ns velero)
     # --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
     #     by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
     #     capturing their Kubernetes resources. ---

platform/velero/kustomization.yaml

@@ -5,19 +5,10 @@ resources:
   - namespace.yaml
   - helmrepository.yaml
   - helmrelease.yaml
-  # --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) -----------------------------------
-  # BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via
-  # `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "",
-  # prod "-prod"). They are separate objects so rotating one environment's key can never touch the
-  # other's — the failure mode that made the split necessary (a single shared Secret carrying R2 keys
-  # would have pointed dev's Velero at MinIO with credentials MinIO has never issued).
-  #
-  # dev  -> velero-cloud-credentials       : the scoped MinIO key for http://192.168.1.9:9000.
-  #                                          Kept in sync with the staging source of truth
-  #                                          platform/backup/minio-credentials.sops.yaml + the
-  #                                          Proxmox host ansible-vault.
-  # prod -> velero-cloud-credentials-prod  : the Cloudflare R2 token for k3s-prod-backup.
-  #
-  # Both are decrypted by the `velero` Flux Kustomization's SOPS stanza (clusters/dev + clusters/prod).
+  # Scoped S3 credential for ${S3_BACKUP_ENDPOINT} (the `cloud` INI key the AWS plugin reads) —
+  # SOPS-encrypted, decrypted per-env by the `velero` Flux Kustomization (clusters/dev).
+  # ONE key lives in FOUR Secrets across three namespaces (this one + the three CNPG
+  # *-db-backup-credentials). platform/backup/set-r2-credentials.sh writes all four with `sops set`
+  # and proves they agree; do not hand-edit one of them. platform/backup/minio-credentials.sops.yaml
+  # retains the PRE-MIGRATION MinIO key on purpose. See infra/BACKUP-R2.md.
   - velero-cloud.sops.yaml
-  - velero-cloud-prod.sops.yaml

platform/velero/values.yaml

@@ -2,16 +2,20 @@
 # doc pointer, kept for parity with the other components' layout).
 #
 # LIVE wiring (see helmrelease.yaml):
-#   - provider aws + velero-plugin-for-aws (init container) → S3 ${S3_BACKUP_ENDPOINT}, path-style,
-#     at s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX} in region ${S3_BACKUP_REGION}. Four
-#     cluster-vars knobs: dev = the on-Proxmox MinIO (http://192.168.1.9:9000, bucket `velero`, no
-#     prefix, us-east-1); prod = Cloudflare R2 (bucket `k3s-prod-backup`, prefix `velero`, `auto`),
-#     sharing one bucket with CNPG/barman under a disjoint prefix.
-#   - config.checksumAlgorithm: "" — MANDATORY for R2, which rejects the AWS SDK's
-#     x-amz-checksum-* headers with HTTP 400 InvalidArgument. Harmless against MinIO.
+#   - provider aws + velero-plugin-for-aws v1.14.2 (init container) → the S3 endpoint
+#     ${S3_BACKUP_ENDPOINT} from cluster-vars. That is a KNOB, not MinIO: dev has been running against
+#     the on-Proxmox MinIO http://192.168.1.9:9000, and as of 2026-09-05 the authored target is
+#     Cloudflare R2 (https://<account-id>.r2.cloudflarestorage.com) — see infra/BACKUP-R2.md.
+#     Always path-style; bucket `velero`; region ${S3_BACKUP_REGION} (MinIO ignores it, R2 demands
+#     exactly `auto`); per-environment prefix ${VELERO_BACKUP_PREFIX}.
+#   - checksumAlgorithm: "" — REQUIRED for R2 and harmless for MinIO. The plugin defaults it to CRC32,
+#     which R2 rejects with HTTP 400 InvalidArgument. Do not drop it; see helmrelease.yaml.
 #   - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover).
+#     NOTE this path uses kopia's OWN S3 client, which does not read checksumAlgorithm — backup
+#     metadata and volume data are two separate S3-compatibility questions (infra/BACKUP-R2.md §7).
 #   - snapshotsEnabled:false → no VolumeSnapshotLocation.
 #   - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero).
 #   - two daily Schedules (${VELERO_BACKUP_SCHEDULE} / ttl ${VELERO_RETENTION}): the whole cluster
 #     via FSB, with monitoring+unifi backed up RESOURCES-ONLY (their volume data is reconstructable).
 # DR drill (tested restore) remains a Phase-4 acceptance gate (R-STATE-03) — the PM runs it post-deploy.
+# It is PROVEN against MinIO only; a restore from R2 has never been executed (infra/BACKUP-R2.md §5).

platform/velero/velero-cloud-prod.sops.yaml

@@ -1,28 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: velero-cloud-credentials-prod
-    namespace: velero
-type: Opaque
-stringData:
-    cloud: ENC[AES256_GCM,data:Kq+hfYMq/NZJJqbK2zqwW97XKNW0LbT2Un20yc0B6evPXqp+aC0vV0odwos4XwV0puPIsEhU7IeD+JidVjU1eKv6Qax0rnoj90FVWihxWyLMNAMC+b8PxBnPhWdk6ZlrMKs13roGZwYh4bbrstnJamMqpD4oZpthEl/NEQNWVAb5ogJHsmPNiQOtHhtBWkTubfgPIw==,iv:gn8/4/XpUZ8c6+nlGe6SAODUyrhDqskQ+uVRx2YR3Qc=,tag:aTk3cRJTA0WjdIi1B9S0hg==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwWmV5UGpmdnFKbEhjeHlZ
-            QUZCWm13ZlpVYU1URFErSmM5MEJ2WTRxZVFvCkZ5cEdJaXJ1NDNSaEF5d2lxdDQw
-            NGpzQm5ESGt6cUlIZVdHYTBjRGVwaVUKLS0tIHlpWkVacXdIZndBOXdlVk1LZzlN
-            NkZyRm1abk9CNlZRUERtbFM2RjVtS1UKWYtI7QZiTjPIHaLOOX9WvDK1cHwP2WfW
-            26WdNW90tTmt3cYdaFGnkpAVCMS3BRtdInirulfoGRgvffjWNBDVuQ==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:4v4L29KoIhQI94TU/s0u3HKR5C6mMc6mRWWsZzwIfqkl4fayRvLG67tbJ88vVkb6AXgqHoKXKlhbpn6hTJPBrL72f8+fii3CeLloVJyW2AuzHaHCSi92U04ywoSc45hClZrOLvUFf5KbtawPPKfP53oaNq+AVEZ1nqKd8ssUyBA=,iv:WVrdja00ptK/MmCC/Chu+Gq4ZYpBvpFRWW3lSo7Mon0=,tag:GzFfpkBlHu6Nc8ZdnCBuKQ==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

platform/vpn-egress/README.md

@@ -28,45 +28,6 @@ Three layers, two owners:
   `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
   `internet`-class and node traffic are unaffected.
 
-## Resilience — there is NO gateway failover (spec correction, escalated)
-
-**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
-provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
-does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
-
-Verified against the CRD actually deployed on this cluster
-(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
-Cilium **v1.20.1**) — the field's own description:
-
-> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
-> **In case multiple nodes are a match for the given set of labels, the first node in lexical
-> ordering based on their name will be selected.**"
-
-Consequences, stated plainly:
-
-- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
-  `[vpn_egress_gateway]` group is singleton by design.
-- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
-  and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
-  (fail-closed, per R-NET-06 — no leak, but no service).
-- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
-  `k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
-  sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
-- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
-  across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
-  blackholes that entry's share and re-shards the remainder (breaking live connections), and each
-  shard leaves from a *different* public IP. Upstream confirmation:
-  [cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
-  support failover between nodes") and
-  [cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
-  tolerate single node failure").
-
-**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
-(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
-here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
-`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
-and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
-
 ## Secrets — host-side now
 PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
 a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.

platform/vpn-egress/egress-steering.yaml

@@ -69,26 +69,6 @@ spec:
   # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
   # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
   # internet fails OPEN).
-  #
-  # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
-  # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
-  # deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
-  # `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
-  #
-  #     "EgressGateway is the gateway node responsible for SNATing traffic.
-  #      In case multiple nodes are a match for the given set of labels, the first node
-  #      in lexical ordering based on their name will be selected."
-  #
-  # So labelling a second node does not create an active/standby pair — it just hands the role to
-  # whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
-  # labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
-  # LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
-  # it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
-  # that shard and re-shards the rest (breaking live connections), and each shard exits from a
-  # DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
-  # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
-  # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
-  # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
   egressGateway:
     nodeSelector:
       matchLabels: