feat(dns): prod takes the apex — home.keithjackson.dev

Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:

  1. dev's 18 A records released by its own external-dns (policy=sync; sources
     deleted after suspending dev's Flux so they could not be recreated).
  2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
     hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
     "All records are already up to date" — it would NEVER have cleared the older
     unprefixed format it no longer writes but still READS for compatibility. They
     carried `owner=default`; prod runs `owner=prod`, so prod could not have
     claimed these names while they existed. This was the real blocker, and it was
     invisible until measured.
  3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
     renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
     duplicate limit (5 per identical name set per week).

Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.

UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.

The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.

k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -334,7 +334,13 @@ node_baseline_dns_search: []
 # prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
 # https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
 # in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
-k3s_base_domain: "prod.home.keithjackson.dev"
+# [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml. This value renders the
+# kube-apiserver's `oidc-issuer-url`, which is INSTALL-TIME config in /etc/rancher/k3s/config.yaml
+# — changing it here does NOT take effect until the k3s_server role re-renders that file and k3s
+# is restarted (serial: 1). Until that rolling restart completes the apiserver still trusts the
+# OLD issuer while Keycloak serves the NEW one, so kubectl OIDC logins fail in between. Cert-based
+# kubeconfigs are unaffected.
+k3s_base_domain: "home.keithjackson.dev"
 
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE