feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. *** Operator simplified the architecture, inverting c8c1d29: dev -> *.dev.home.keithjackson.dev (was the apex) prod -> *.home.keithjackson.dev (the apex; no prod. subdomain, no CNAME layer) WHY MERGING IS DANGEROUS, and this is the finding that matters most here: clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging these files EXECUTES the cutover on the live dev cluster, unattended. And because the same commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside the url itself. That url cannot be tokenized: the root Kustomization declares no substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token. Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.` cannot collide with dev on the apex, so that is the safer interim for prod. VALUES BASE_DOMAIN TXT_OWNER_ID DOMAIN_FILTER base dev.home.keithjackson.dev sentinel keithjackson.dev dev dev.home.keithjackson.dev dev keithjackson.dev prod home.keithjackson.dev prod keithjackson.dev Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod overrides to the apex. dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records were already registered under `default`, so renaming would have orphaned them). On a new suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly how both clusters came to share it and delete each other's records. Operationally it also means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them implicitly and vacating the apex becomes an explicit sequenceable step rather than a race between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's still-live apex records — the same incident with the roles swapped. DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING. VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution to the new render and demand byte-identity with the baseline. 145 changed line pairs, every one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev *.dev.home.keithjackson.dev, prod *.home.keithjackson.dev. THREE positive controls, all fire — including one that proves the two checks are not redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is caught only by the independent bare-apex scan. That is why both exist. Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes -> keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars outranks group_vars and would have FALSELY shown prod inheriting dev's value. Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions), render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/. RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6 one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7 retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense — the gate on everything after. Then P1 unsuspend prod. ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's 5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather than let it renew, rehearse on staging. The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while dev is live on the apex they are correct; they need per-environment parameterisation. Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 23:08:50 -05:00
1 parent c8c1d2935d
commit 9137bbd0c3
@@ -26,36 +26,49 @@ data:
  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
-  # subdomain, so the two clusters can no longer describe the same name.
+  # suffix, so the two clusters can no longer describe the same name.
  #
-  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
-  #   prod  -> prod.home.keithjackson.dev
+  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
  #
-  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
-  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
-  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
-  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
-  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
-  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
-  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
-  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
-  BASE_DOMAIN: "home.keithjackson.dev"
+  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # string dev holds.
+  #
+  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
+  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
+  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
+  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
+  # infra/PROD-PROVISIONING.md §10.
+  #
+  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
+  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
+  BASE_DOMAIN: "dev.home.keithjackson.dev"

  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
  # so it cannot be created, updated or deleted regardless of TXT ownership.
  #
-  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
-  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
-  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
-  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
-  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
  #
-  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
-  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
-  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
-  # — it can never delete another environment's records — and the fallback is to set prod's value
-  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # it — the change is not defence in depth, it is a total outage of DNS publication.
+  #
+  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
+  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
+  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
+  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
+  # both environments live in the one zone either way.
  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"

  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
@@ -64,14 +77,35 @@ data:
  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
  #
-  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
-  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
-  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
  #
-  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
-  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
-  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
-  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
+  # is now free to be chosen on its merits. Two reasons to take `dev`:
+  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
+  #     clusters silently shared it, which IS the incident). An environment whose identity is the
+  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
+  #     hold it.
+  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
+  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
+  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
+  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
+  #     race between two reconcile loops.
+  #
+  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
+  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
+  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
+  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
+  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
+  # environment's id. A new environment that forgets this knob must not silently inherit another's
+  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
+  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
+  # instead of destroying a live environment's DNS.
  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"

  ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
@@ -79,7 +113,7 @@ data:
                                            # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                            # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                            # takes the pool's first free IP, and coredns-custom answers
-                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
+                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
                                            # hairpins through the Gateway (correct SNI -> wildcard cert).
                                            # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                            # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
@@ -5,6 +5,20 @@
 # reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
 # It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
 # on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
+#
+# ##############################################################################################
+# ###  *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) ***                 ###
+# ###                                                                                        ###
+# ###  This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex   ###
+# ###  `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url       ###
+# ###  moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a   ###
+# ###  merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
+# ###  Dev's Gitea is both its GitOps source and the push target: if the new host is not up    ###
+# ###  with a Ready cert first, dev stops reconciling and pushing cannot repair it.            ###
+# ###                                                                                        ###
+# ###  Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator        ###
+# ###  present. Merging is step D3, not step one.                                             ###
+# ##############################################################################################
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -131,7 +145,24 @@ patches:
        # via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
        # secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
        # revert this url (docs/05).
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        #
+        # *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
+        # It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
+        # it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
+        # postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
+        # break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
+        # So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
+        #
+        # It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
+        # makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
+        # (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
+        # DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
+        # resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
+        # BY PUSHING — the broken thing is the path Flux uses to read pushes.
+        # ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
+        # Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
+        # the migration window, and only restore the in-cluster url after the new host is proven.
+        url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
        ref:
          branch: main
  # --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -148,33 +179,60 @@ patches:
      - op: replace
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
-      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
-      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
-      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
-      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
-      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
-      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
-      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
-      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
-      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      # ==========================================================================================
+      # DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
+      #
+      #   dev   -> dev.home.keithjackson.dev      (this knob; = the base default, restated)
+      #   prod  -> home.keithjackson.dev          (THE APEX — prod IS the apex, no `prod.` label,
+      #                                            and therefore no apex CNAME layer at all)
+      #
+      # *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
+      # now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
+      # an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
+      #
+      # WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
+      # lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
+      # source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
+      # DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
+      # not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
+      # BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
+      # is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
+      # in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
+      # FIRST, or park the source on the bootstrap Gitea for the window.
+      #
+      # ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
+      # ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
+      # server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
+      # split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
      - op: replace
        path: /data/BASE_DOMAIN
-        value: "home.keithjackson.dev"
-      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
-      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
-      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+        value: "dev.home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
+      # F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
+      # strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
+      # record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
+      # newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
      - op: replace
        path: /data/EXTERNALDNS_DOMAIN_FILTER
        value: "keithjackson.dev"
-      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
-      # registry under owner `default` (the external-dns chart default, which is what both clusters
-      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
-      # BY this string: any other value and it stops recognising every record it manages, then
-      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
-      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      # OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
+      # records were already in Cloudflare's TXT registry under owner `default`, so renaming would
+      # have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
+      # dev publishes a brand-new record set with no prior registry entries to recognise, so the id
+      # is chosen on its merits, and `default` has none — it is the value external-dns uses when you
+      # FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
+      # deliberate:
+      #   - dev's OLD apex records remain owned by `default`, which no environment now claims, so
+      #     nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
+      #     publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
+      #   - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
+      #     cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
+      # dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
      - op: replace
        path: /data/EXTERNALDNS_TXT_OWNER_ID
-        value: "default"
+        value: "dev"
      # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
      # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
      # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -183,7 +241,7 @@ patches:
      #
      # Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
      # (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
-      # *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
+      # *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
      - op: replace
        path: /data/GATEWAY_VIP
        value: "192.168.1.53"
@@ -129,14 +129,23 @@ patches:
  #
  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
-  # Two independent reasons, both consequences of the split-horizon design:
-  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
-  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
-  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
-  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
-  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
-  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
-  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
+  # which changes WHY the blocker holds but not THAT it holds:
+  #   1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
+  #      pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
+  #      inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
+  #      BEFORE dev is retired, or prod loses its source of truth.
+  #      NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
+  #      gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
+  #      the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
+  #      cutover below a one-line change once prod actually runs Gitea.
+  #   2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
+  #      easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
+  #      ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
+  #      prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
+  #      listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
+  #      to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
+  #      remains is purely reason 1 — the Gitea is not there YET.
  #
  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
@@ -202,28 +211,41 @@ patches:
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
      # ==========================================================================================
-      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
      # ==========================================================================================
-      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
-      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
-      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
-      # of which already happened:
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
+      # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
+      #
+      #   dev   -> dev.home.keithjackson.dev     (dev carries the label; clusters/dev)
+      #   prod  -> home.keithjackson.dev         (THE APEX — this knob)
+      #
+      # There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
+      # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
+      # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
+      #
+      # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
+      # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
+      # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
+      # consequences, one of which already happened:
      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
      #      have raced each other's `_acme-challenge` TXT records on that one zone.
-      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
-      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
      #
-      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
-      #   prod  -> prod.home.keithjackson.dev    (this knob)
-      #
-      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
-      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
-      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
-      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
-      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
+      # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
+      # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
+      # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
+      #   (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
+      #   (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
+      #       (verified against Cloudflare directly — a pfSense override is NOT evidence about public
+      #       DNS; that mistake was made on 2026-09-04), AND
+      #   (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
+      #       that same name set while dev's still exists counts against LE's duplicate-certificate
+      #       limit (5 per week per identical name set).
+      # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
+      # the 2026-09-04 incident with the roles reversed.
      #
      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
@@ -231,30 +253,33 @@ patches:
      # trusts a different Keycloak than the one prod actually runs.
      - op: replace
        path: /data/BASE_DOMAIN
-        value: "prod.home.keithjackson.dev"
-      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
-      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
-      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
-      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
-      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
-      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
-      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
-      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+        value: "home.keithjackson.dev"
      - op: replace
        path: /data/EXTERNALDNS_DOMAIN_FILTER
-        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
-        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
-        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
-        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
-        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
-        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
+        # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
+        # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
+        # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
+        # record DNS Name", so prod publishes NOTHING. An agent narrowed this to
+        # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
+        # Isolation from dev does NOT come from this filter — it comes from
+        # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
+        # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
        value: "keithjackson.dev"
      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
-      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
-      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
-      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      # its own and, under policy: sync, deleted the records behind them.
+      #
+      # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
+      # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
+      # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
+      # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
+      # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
+      # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
+      # simply fails to publish until those records are retired, which is the correct behaviour.
+      # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
+      # non-colliding sentinel. All three differ.
      - op: replace
        path: /data/EXTERNALDNS_TXT_OWNER_ID
        value: "prod"