feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

clusters/base/cluster-vars.yaml

@@ -26,36 +26,49 @@ data:
   # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
   # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
   # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
-  # subdomain, so the two clusters can no longer describe the same name.
+  # suffix, so the two clusters can no longer describe the same name.
   #
-  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
-  #   prod  -> prod.home.keithjackson.dev
+  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
   #
-  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
-  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
-  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
-  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
-  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
-  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
-  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
-  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
-  BASE_DOMAIN: "home.keithjackson.dev"
+  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # string dev holds.
+  #
+  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
+  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
+  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
+  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
+  # infra/PROD-PROVISIONING.md §10.
+  #
+  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
+  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
+  BASE_DOMAIN: "dev.home.keithjackson.dev"
 
   # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
   # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
   # so it cannot be created, updated or deleted regardless of TXT ownership.
   #
-  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
-  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
-  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
-  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
-  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
   #
-  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
-  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
-  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
-  # — it can never delete another environment's records — and the fallback is to set prod's value
-  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # it — the change is not defence in depth, it is a total outage of DNS publication.
+  #
+  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
+  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
+  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
+  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
+  # both environments live in the one zone either way.
   EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
 
   # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
@@ -64,14 +77,35 @@ data:
   # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
   # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
   #
-  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
-  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
-  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
   #
-  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
-  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
-  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
-  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
+  # is now free to be chosen on its merits. Two reasons to take `dev`:
+  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
+  #     clusters silently shared it, which IS the incident). An environment whose identity is the
+  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
+  #     hold it.
+  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
+  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
+  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
+  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
+  #     race between two reconcile loops.
+  #
+  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
+  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
+  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
+  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
+  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
+  # environment's id. A new environment that forgets this knob must not silently inherit another's
+  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
+  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
+  # instead of destroying a live environment's DNS.
   EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
 
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
@@ -79,7 +113,7 @@ data:
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                             # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                             # takes the pool's first free IP, and coredns-custom answers
-                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
+                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
                                             # hairpins through the Gateway (correct SNI -> wildcard cert).
                                             # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                             # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that