Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Revert "feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex"

Browse Source 
This reverts commit 9137bbd0c3.
This commit is contained in:
jakman85 committed 2026-09-06 20:27:41 -05:00
1 parent 6f61c9622d
commit f341a9b744
8 files changed
+229 -401
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/base/cluster-vars.yaml
+30 -64
View File
@@ -26,49 +26,36 @@ data:
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
# suffix, so the two clusters can no longer describe the same name.
# subdomain, so the two clusters can no longer describe the same name.
#
# dev -> dev.home.keithjackson.dev (a subdomain; = this base default)
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label)
# dev -> home.keithjackson.dev (UNCHANGED — see below)
# prod -> prod.home.keithjackson.dev
#
# OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
# had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
# whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
# DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
# default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
# string dev holds.
#
# DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
# now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
# URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
# push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
# Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
# cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
# infra/PROD-PROVISIONING.md §10.
#
# DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
# `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
BASE_DOMAIN: "dev.home.keithjackson.dev"
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
BASE_DOMAIN: "home.keithjackson.dev"
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
# so it cannot be created, updated or deleted regardless of TXT ownership.
#
# BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
# environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
#
# FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
# SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
# matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
# record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
# narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
# it — the change is not defence in depth, it is a total outage of DNS publication.
#
# WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
# (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
# (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
# That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
# both environments live in the one zone either way.
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
# — it can never delete another environment's records — and the fallback is to set prod's value
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
@@ -77,35 +64,14 @@ data:
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
#
# dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below)
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
#
# DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
# this migration dev HAD to render `default`, because its LIVE apex records were already registered
# in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
# THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
# *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
# is now free to be chosen on its merits. Two reasons to take `dev`:
# - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
# clusters silently shared it, which IS the incident). An environment whose identity is the
# forget-value can never be told apart from a misconfigured newcomer. No environment should
# hold it.
# - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
# owned by `default`, which no environment now claims, so nothing deletes them implicitly on
# some future reconcile — they are retired deliberately, and prod's publish is gated on that
# retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
# race between two reconcile loops.
#
# PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
# dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
# the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
# would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
# roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
#
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
# environment's id. A new environment that forgets this knob must not silently inherit another's
# identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
# can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
# instead of destroying a live environment's DNS.
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
@@ -113,7 +79,7 @@ data:
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
# takes the pool's first free IP, and coredns-custom answers
# *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
# hairpins through the Gateway (correct SNI -> wildcard cert).
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
clusters/dev/kustomization.yaml
+22 -80
View File
@@ -5,20 +5,6 @@
# reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
# It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
# on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
#
# ##############################################################################################
# ### *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) *** ###
# ### ###
# ### This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex ###
# ### `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url ###
# ### moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a ###
# ### merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
# ### Dev's Gitea is both its GitOps source and the push target: if the new host is not up ###
# ### with a Ready cert first, dev stops reconciling and pushing cannot repair it. ###
# ### ###
# ### Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator ###
# ### present. Merging is step D3, not step one. ###
# ##############################################################################################
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
@@ -145,24 +131,7 @@ patches:
# via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
# secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
# revert this url (docs/05).
#
# *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
# It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
# it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
# postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
# break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
# So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
#
# It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
# makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
# (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
# DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
# resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
# BY PUSHING — the broken thing is the path Flux uses to read pushes.
# ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
# Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
# the migration window, and only restore the in-cluster url after the new host is proven.
url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
ref:
branch: main
# --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -179,60 +148,33 @@ patches:
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
# ==========================================================================================
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
#
# dev -> dev.home.keithjackson.dev (this knob; = the base default, restated)
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label,
# and therefore no apex CNAME layer at all)
#
# *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
# now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
# an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
#
# WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
# lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
# source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
# DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
# not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
# BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
# is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
# in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
# FIRST, or park the source on the bootstrap Gitea for the window.
#
# ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
# ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
# server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
# split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
# not a variable change: infra/PROD-PROVISIONING.md §10.
- op: replace
path: /data/BASE_DOMAIN
value: "dev.home.keithjackson.dev"
# ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
# F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
# strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
# record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
# newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
value: "home.keithjackson.dev"
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
value: "keithjackson.dev"
# OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
# records were already in Cloudflare's TXT registry under owner `default`, so renaming would
# have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
# dev publishes a brand-new record set with no prior registry entries to recognise, so the id
# is chosen on its merits, and `default` has none — it is the value external-dns uses when you
# FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
# deliberate:
# - dev's OLD apex records remain owned by `default`, which no environment now claims, so
# nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
# publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
# - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
# cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
# dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
# registry under owner `default` (the external-dns chart default, which is what both clusters
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
# BY this string: any other value and it stops recognising every record it manages, then
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "dev"
value: "default"
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
# equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -241,7 +183,7 @@ patches:
#
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
# *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.1.53"
clusters/prod/kustomization.yaml
+48 -66
View File
@@ -129,23 +129,14 @@ patches:
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
# which changes WHY the blocker holds but not THAT it holds:
# 1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
# pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
# inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
# BEFORE dev is retired, or prod loses its source of truth.
# NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
# gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
# the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
# cutover below a one-line change once prod actually runs Gitea.
# 2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
# easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
# ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
# prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
# listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
# to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
# remains is purely reason 1 — the Gitea is not there YET.
# Two independent reasons, both consequences of the split-horizon design:
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
# gitea.home.keithjackson.dev to itself and never reach a repository.
#
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
@@ -211,41 +202,35 @@ patches:
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
# ==========================================================================================
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
# (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
#
# dev -> dev.home.keithjackson.dev (dev carries the label; clusters/dev)
# prod -> home.keithjackson.dev (THE APEX — this knob)
#
# There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
# name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
# Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
#
# BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
# HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
# EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
# consequences, one of which already happened:
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
# of which already happened:
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
# have raced each other's `_acme-challenge` TXT records on that one zone.
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
#
# *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
# Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
# therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
# tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
# (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
# (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
# (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public
# DNS; that mistake was made on 2026-09-04), AND
# (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
# that same name set while dev's still exists counts against LE's duplicate-certificate
# limit (5 per week per identical name set).
# Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
# the 2026-09-04 incident with the roles reversed.
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
# prod -> prod.home.keithjackson.dev (this knob)
#
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
#
# THE APEX INVERSION IS DELIBERATELY NOT ON THIS BRANCH. Moving prod to the bare apex (and dev
# to dev.home.) is a LIVE MIGRATION of the dev cluster, not a variable change — see
# infra/PROD-PROVISIONING.md §10a. It is kept as a single commit on branch
# `migration/dns-apex-inversion` (9137bbd) and is applied as step D3 of that runbook, with the
# operator present. Everything else on this branch is independent of it, which is the whole
# reason it was split out.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
@@ -253,33 +238,30 @@ patches:
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
value: "home.keithjackson.dev"
value: "prod.home.keithjackson.dev"
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
# MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
# like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
# F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
# zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
# record DNS Name", so prod publishes NOTHING. An agent narrowed this to
# prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
# Isolation from dev does NOT come from this filter — it comes from
# (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
# (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
value: "keithjackson.dev"
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
# its own and, under policy: sync, deleted the records behind them.
#
# PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
# NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
# the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
# prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
# never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
# invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
# simply fails to publish until those records are retired, which is the correct behaviour.
# dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
# non-colliding sentinel. All three differ.
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "prod"