feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 23:08:50 -05:00
1 parent c8c1d2935d
commit 9137bbd0c3
8 files changed
+556 -234

No files matched your search

@@ -293,35 +293,47 @@ node_baseline_dns_servers:
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
#
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
#
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
# external-dns has actually published the record. Until then the issuer does not resolve
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
# path, and it always works. Ordering, not a design gap.
# ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
# name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
# That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
# points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
# node resolving its own issuer through these public upstreams therefore reaches dev until:
# (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
# (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
# Gateway VIP (192.168.2.20).
# Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
# /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
# admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
# the ordered steps are infra/PROD-PROVISIONING.md §10.
node_baseline_dns_search: []
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
# postBuild.substituteFrom never sees it. Full rationale in
# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
#
# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
k3s_base_domain: "prod.home.keithjackson.dev"
# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
# deleted every one of dev's public records.
#
# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
# prod -> home.keithjackson.dev (THE APEX; this variable — no `prod.` label, no CNAMEs)
# dev -> dev.home.keithjackson.dev (roles/k3s_server/defaults/main.yml)
# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
# against.
#
# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
k3s_base_domain: "home.keithjackson.dev"
# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
@@ -84,15 +84,26 @@ k3s_tls_sans: >-
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
#
# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
# what it already is and the Restart handler does NOT fire on a re-run.
# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
# from anyone remembering).
#
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
# from group membership rather than from anyone remembering.
k3s_base_domain: "home.keithjackson.dev"
# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
#
# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
# work: DO NOT run the k3s_server role against dev until step D6 of
# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
k3s_base_domain: "dev.home.keithjackson.dev"
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"