From 9161cab9cdf8389d16477b963068b59f59af2478 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 22:07:20 -0500 Subject: [PATCH] =?UTF-8?q?feat(dns):=20prod=20takes=20the=20apex=20?= =?UTF-8?q?=E2=80=94=20home.keithjackson.dev?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dev is decommissioned and has released the name. Prerequisites completed and verified BEFORE this flip: 1. dev's 18 A records released by its own external-dns (policy=sync; sources deleted after suspending dev's Flux so they could not be recreated). 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged "All records are already up to date" — it would NEVER have cleared the older unprefixed format it no longer writes but still READS for compatibility. They carried `owner=default`; prod runs `owner=prod`, so prod could not have claimed these names while they existed. This was the real blocker, and it was invisible until measured. 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops renewing. Issued ~18d ago, so prod's request is #1 this week against LE's duplicate limit (5 per identical name set per week). Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1 served stale TXT for minutes after deletion and would have shown a false failure. UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod `default` would let it adopt and sync-delete records it does not own, which is the 2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`: finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns then publishes NOTHING. The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter; prod's new specific records shadow it again. The legacy stack lives on `*.keithjackson.dev` and is untouched. k3s_base_domain moves with it to hold the [parity] contract, but that value renders the apiserver's install-time oidc-issuer-url: it does NOT take effect until the k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC fails in that window. Cert-based kubeconfigs are unaffected. Co-Authored-By: Claude Opus 5 --- clusters/prod/kustomization.yaml | 19 ++++++++++++++++++- .../inventory/group_vars/k3s_prod/vars.yml | 8 +++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 92ee79c..9081d3f 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -247,7 +247,24 @@ patches: # trusts a different Keycloak than the one prod actually runs. - op: replace path: /data/BASE_DOMAIN - value: "prod.home.keithjackson.dev" + # APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been + # decommissioned and released the name. Prerequisites completed and VERIFIED before this + # flip, in this order: + # 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted) + # 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand — + # external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All + # records are already up to date", so it would NEVER have cleared them. They carried + # `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these + # names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com); + # 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here. + # 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing. + # It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate + # limit of 5 per identical name set per week. + # NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing + # at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were + # SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack + # lives on `*.keithjackson.dev` and is untouched by any of this. + value: "home.keithjackson.dev" # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index bb4bbba..41883e1 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -334,7 +334,13 @@ node_baseline_dns_search: [] # prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is # https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client # in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against. -k3s_base_domain: "prod.home.keithjackson.dev" +# [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml. This value renders the +# kube-apiserver's `oidc-issuer-url`, which is INSTALL-TIME config in /etc/rancher/k3s/config.yaml +# — changing it here does NOT take effect until the k3s_server role re-renders that file and k3s +# is restarted (serial: 1). Until that rolling restart completes the apiserver still trusts the +# OLD issuer while Keycloak serves the NEW one, so kubectl OIDC logins fail in between. Cert-based +# kubeconfigs are unaffected. +k3s_base_domain: "home.keithjackson.dev" # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) ----- # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE