feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/PROD-PROVISIONING.md

@@ -1172,6 +1172,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
 | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
 | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
 | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
+| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
 
 - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
       overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1282,3 +1283,152 @@ single-server-etcd window, and avoids a later member-add dance.
       specifically.
 - [ ] Monitoring and alerts live on prod.
 - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
+      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
+- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
+      distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
+      prod's own Keycloak realm + per-service client secrets.
+
+## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
+
+### What happened
+
+Prod's `external-dns` **deleted every one of dev's public DNS records.**
+
+Both clusters ran `policy: sync` against the **same Cloudflare zone**, served the **same
+hostnames**, and — because neither overlay set `txtOwnerId` — both inherited the external-dns
+chart's default owner id **`default`**. ExternalDNS decides what it may delete by reading its own
+owner id back out of the TXT registry. Prod read dev's TXT entries, saw its own owner id, found no
+local resource claiming those names, and garbage-collected the records behind them. Dev recovered
+once prod was stopped. Prod's `platform` tier and everything downstream were then **suspended**, and
+remain suspended, so it cannot recur before this change is reviewed.
+
+`policy: sync` was not the bug and has not been changed. What was missing was **identity**.
+
+### The second bug, queued directly behind it
+
+`platform-config` would have requested a `*.home.keithjackson.dev` wildcard `Certificate` from
+**letsencrypt-prod** — for a cert **dev already holds** — and both clusters would then have raced
+each other's `_acme-challenge` TXT records on that one zone. Neither would reliably solve DNS-01.
+This never fired only because the tier was suspended first.
+
+### Root cause, stated precisely
+
+Not external-dns. **Every hostname in the repo was a hardcoded literal.** `BASE_DOMAIN` existed in
+`clusters/base/cluster-vars.yaml` but was referenced by **no reconciled manifest** — the repo-derived
+status report flagged it as a dead knob. R-ENV-02 promises dev and prod differ only by `cluster-vars`
+values; for DNS and TLS that promise was false, so `clusters/prod` was *authored* to serve dev's
+public identity. Two clusters that describe the same DNS name will fight over it no matter how the
+DNS controller is configured.
+
+### The architecture decision (operator)
+
+```
+dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
+prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
+apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev
+```
+
+### PHASE 1 — **prod only. Dev did not move.** (this change)
+
+`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
+
+| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
+|---|---|---|---|
+| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
+| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
+| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
+
+**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
+alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
+while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
+`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
+by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
+both overlays and diffing.
+
+**Two independent guards now, either sufficient alone:**
+
+1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
+   the direct fix for the incident. Dev **must** render exactly `default`: its live records are
+   already registered under that owner, and any other value would make dev's own external-dns stop
+   recognising them and re-create duplicates beside the orphans.
+2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
+   Defence in depth, added because the first guard is a single string in a single file.
+
+The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
+future environment that forgets the knob must not silently inherit dev's identity — that is exactly
+what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
+
+### What could NOT be tokenized, and why
+
+- **`OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` in `platform/headlamp/headlamp-oidc-credentials.sops.yaml`**
+  — SOPS-encrypted (`encrypted_regex: ^(data|stringData)$`), so the two hostnames are ciphertext.
+  Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
+  **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
+  would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
+  **Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
+  Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
+  secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
+  is per-realm.
+- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
+  **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
+  can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
+  `k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
+  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
+  `[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
+  validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
+  cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
+
+### A pre-existing prod problem this incidentally FIXED
+
+§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
+issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
+making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
+its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
+**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
+and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
+external-dns has actually published the record; until then use the break-glass cert kubeconfig.
+
+### Verify on prod's first reconcile
+
+- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
+      as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
+      ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
+      will create **nothing** — fail-closed, never destructive. Fallback: set prod's
+      `EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
+      sufficient on its own.
+- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
+      dev's `_acme-challenge` records.
+- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
+      entries still read owner `default` and that prod's read owner `prod`.
+- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
+      secret is realm-scoped; prod cannot reuse dev's).
+
+### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
+
+Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
+Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
+Required, roughly in order:
+
+1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
+   dev inventory group.
+2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
+3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
+   (`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
+   `qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
+   updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
+   **rolling k3s restart** on the dev servers.
+4. **kubectl SSO re-issue** for every user (issuer changed).
+5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
+   `clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
+   the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
+   Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
+6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
+   records are for names dev no longer serves; retire them deliberately rather than letting a
+   renamed owner orphan them.
+7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
+   actually **moves user traffic to prod**, and it must not land before prod serves those services.
+8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
+   `dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
+   `trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
+   — but they will fail against prod and must be parameterized before §9's "suite passes green
+   against the prod cluster" can be claimed.