feat(dns): prod takes the apex — home.keithjackson.dev

Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:

  1. dev's 18 A records released by its own external-dns (policy=sync; sources
     deleted after suspending dev's Flux so they could not be recreated).
  2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
     hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
     "All records are already up to date" — it would NEVER have cleared the older
     unprefixed format it no longer writes but still READS for compatibility. They
     carried `owner=default`; prod runs `owner=prod`, so prod could not have
     claimed these names while they existed. This was the real blocker, and it was
     invisible until measured.
  3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
     renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
     duplicate limit (5 per identical name set per week).

Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.

UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.

The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.

k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.

Co-Authored-By: Claude Opus 5 <[email protected]>

clusters/prod/kustomization.yaml

@@ -247,7 +247,24 @@ patches:
       # trusts a different Keycloak than the one prod actually runs.
       - op: replace
         path: /data/BASE_DOMAIN
-        value: "prod.home.keithjackson.dev"
+        # APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
+        # decommissioned and released the name. Prerequisites completed and VERIFIED before this
+        # flip, in this order:
+        #   1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
+        #   2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
+        #      external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
+        #      records are already up to date", so it would NEVER have cleared them. They carried
+        #      `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
+        #      names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
+        #      1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
+        #   3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
+        #      It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
+        #      limit of 5 per identical name set per week.
+        # NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
+        # at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
+        # SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
+        # lives on `*.keithjackson.dev` and is untouched by any of this.
+        value: "home.keithjackson.dev"
       # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
       # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
       # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -334,7 +334,13 @@ node_baseline_dns_search: []
 # prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
 # https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
 # in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
-k3s_base_domain: "prod.home.keithjackson.dev"
+# [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml. This value renders the
+# kube-apiserver's `oidc-issuer-url`, which is INSTALL-TIME config in /etc/rancher/k3s/config.yaml
+# — changing it here does NOT take effect until the k3s_server role re-renders that file and k3s
+# is restarted (serial: 1). Until that rolling restart completes the apiserver still trusts the
+# OLD issuer while Keycloak serves the NEW one, so kubectl OIDC logins fail in between. Cert-based
+# kubeconfigs are unaffected.
+k3s_base_domain: "home.keithjackson.dev"
 
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE