Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Revert "feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex"

Browse Source 
This reverts commit 9137bbd0c3.
This commit is contained in:
jakman85 committed 2026-09-06 20:27:41 -05:00
1 parent 6f61c9622d
commit f341a9b744
8 files changed
+229 -401
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/prod/kustomization.yaml
+48 -66
View File
@@ -129,23 +129,14 @@ patches:
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
# which changes WHY the blocker holds but not THAT it holds:
# 1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
# pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
# inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
# BEFORE dev is retired, or prod loses its source of truth.
# NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
# gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
# the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
# cutover below a one-line change once prod actually runs Gitea.
# 2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
# easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
# ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
# prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
# listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
# to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
# remains is purely reason 1 — the Gitea is not there YET.
# Two independent reasons, both consequences of the split-horizon design:
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
# gitea.home.keithjackson.dev to itself and never reach a repository.
#
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
@@ -211,41 +202,35 @@ patches:
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
# ==========================================================================================
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
# (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
#
# dev -> dev.home.keithjackson.dev (dev carries the label; clusters/dev)
# prod -> home.keithjackson.dev (THE APEX — this knob)
#
# There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
# name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
# Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
#
# BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
# HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
# EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
# consequences, one of which already happened:
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
# of which already happened:
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
# have raced each other's `_acme-challenge` TXT records on that one zone.
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
#
# *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
# Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
# therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
# tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
# (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
# (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
# (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public
# DNS; that mistake was made on 2026-09-04), AND
# (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
# that same name set while dev's still exists counts against LE's duplicate-certificate
# limit (5 per week per identical name set).
# Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
# the 2026-09-04 incident with the roles reversed.
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
# prod -> prod.home.keithjackson.dev (this knob)
#
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
#
# THE APEX INVERSION IS DELIBERATELY NOT ON THIS BRANCH. Moving prod to the bare apex (and dev
# to dev.home.) is a LIVE MIGRATION of the dev cluster, not a variable change — see
# infra/PROD-PROVISIONING.md §10a. It is kept as a single commit on branch
# `migration/dns-apex-inversion` (9137bbd) and is applied as step D3 of that runbook, with the
# operator present. Everything else on this branch is independent of it, which is the whole
# reason it was split out.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
@@ -253,33 +238,30 @@ patches:
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
value: "home.keithjackson.dev"
value: "prod.home.keithjackson.dev"
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
# MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
# like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
# F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
# zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
# record DNS Name", so prod publishes NOTHING. An agent narrowed this to
# prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
# Isolation from dev does NOT come from this filter — it comes from
# (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
# (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
value: "keithjackson.dev"
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
# its own and, under policy: sync, deleted the records behind them.
#
# PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
# NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
# the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
# prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
# never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
# invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
# simply fails to publish until those records are retired, which is the correct behaviour.
# dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
# non-colliding sentinel. All three differ.
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "prod"