feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched

Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home
Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway
VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under
prod.home.keithjackson.dev, every TXT registry entry owner=prod.

End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway:
apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO).

DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup
baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD
placeholder, and resuming it would create a BackupStorageLocation that can never
reach a bucket: a tier reporting healthy while backups silently do not exist.

TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check
worthless. Recording them because a check you cannot trust is worse than none —
it trains you to skip the one alarm that matters.

1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the
   DNS baseline was discarded between steps and `verify`'s before/after
   comparison — the check that detects damage to the other cluster's records —
   could never run across invocations. Now repo-relative (bind-mounted, so it
   persists on the host) and gitignored as a run artifact.

2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED",
   listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation —
   the host ordered `*.keithjackson.dev` differently from the container, so a
   diff of two identical sets reported changes. Proved it was noise by
   re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips
   CRs and sorts under LC_ALL=C so ordering is reproducible everywhere.

ACTION REQUIRED (operator, pfSense): the local override is a wildcard on
*.home.keithjackson.dev and now also swallows the new prod.home.* subtree,
resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly
returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT
from behind pfSense. Narrow the override, or add one pointing prod.home.* at
192.168.2.20.

That override is also the trap that produced a false healthy reading during the
2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's
*.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label)
so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels)
fails with a certificate-name error. I hit this myself during verification and
mistook it for a broken prod cert. Never verify DNS or TLS here through a local
resolver.

Refs R-ENV-02, R-GITOPS-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

bootstrap/standup.sh

@@ -53,8 +53,12 @@ cd "${repo_root}"
 export KUBECONFIG="${KUBECONFIG_ARG}"
 [ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; }
 
-# Where DNS snapshots go. Deliberately OUTSIDE the repo: they are run artifacts, not authored state.
-SNAP_DIR="${STANDUP_SNAP_DIR:-${TMPDIR:-/tmp}/standup-dns}"
+# Where DNS snapshots go. This MUST be a path that survives between invocations: each `dev.sh run` is
+# a FRESH container, so a $TMPDIR default silently threw the baseline away and made `verify`'s
+# before/after comparison — the check that detects damage to the OTHER cluster's records — impossible
+# to run across steps. The repo is bind-mounted, so a repo-relative dir persists on the host.
+# Gitignored: these are run artifacts, not authored state.
+SNAP_DIR="${STANDUP_SNAP_DIR:-${repo_root}/.standup-dns}"
 mkdir -p "${SNAP_DIR}"
 
 say()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
@@ -124,7 +128,14 @@ dns_snapshot() { # <output-file>
     [ "${page}" -ge "${tp:-1}" ] && break
     page=$((page+1))
   done
-  grep -vE '^\s*$' "${out}" | sort -o "${out}"
+  # Strip CRs, and sort under a FIXED collation. Both matter, and both were learned the hard way:
+  #   - a stray \r makes every line differ from a baseline captured without one;
+  #   - locale-dependent sort ordering puts `*.keithjackson.dev` in a different position on the host
+  #     than in the container, so a diff reports pairs of IDENTICAL lines as changed. That produced a
+  #     "RECORDS OUTSIDE ... CHANGED" alarm on a zone where nothing had changed at all.
+  # A regression check that cries wolf is worse than none — it trains you to skip the one alarm that
+  # matters. LC_ALL=C makes the ordering reproducible everywhere.
+  tr -d '\r' < "${out}" | grep -vE '^\s*$' | LC_ALL=C sort -o "${out}"
   info "$(wc -l < "${out}") records -> ${out}"
 }
 
@@ -395,10 +406,40 @@ step_verify() {
   say "gateway / LoadBalancer VIPs"
   kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true
   say "published DNS for this cluster's own domain"
+  local owner; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
   dns_snapshot "${SNAP_DIR}/dns-verify.txt"
   info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)"
-  warn "confirm every TXT registry entry for ${base} reads this cluster's owner id ($(cvar EXTERNALDNS_TXT_OWNER_ID)),
-       and that NO record outside ${base} was touched. Compare against ${SNAP_DIR}/dns-before.txt."
+
+  # Every TXT registry entry this cluster publishes must carry OUR owner id. One reading another
+  # cluster's id means we have claimed its records and `policy: sync` will delete them next cycle.
+  local foreign
+  foreign="$(grep "${base}" "${SNAP_DIR}/dns-verify.txt" | grep 'heritage=external-dns' \
+             | grep -v "external-dns/owner=${owner}" || true)"
+  if [ -n "${foreign}" ]; then
+    warn "TXT entries under ${base} NOT owned by '${owner}' — this cluster has claimed records it does not own:"
+    printf '%s\n' "${foreign}" | sed 's/^/      /'
+  else
+    info "every TXT registry entry under ${base} reads owner=${owner}: OK"
+  fi
+
+  # THE REGRESSION CHECK. Additions under our own BASE_DOMAIN are expected. ANY change to a name
+  # outside it means we have touched the other cluster's zone — the 2026-09-04 incident.
+  if [ -f "${SNAP_DIR}/dns-before.txt" ]; then
+    say "regression check against the pre-standup baseline"
+    local outside
+    outside="$(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" \
+               | grep -E '^[<>]' | grep -v "${base}" || true)"
+    if [ -n "${outside}" ]; then
+      warn "RECORDS OUTSIDE ${base} CHANGED — investigate before doing anything else:"
+      printf '%s\n' "${outside}" | sed 's/^/      /'
+    else
+      info "no record outside ${base} was added, changed or removed: OK"
+      info "added under ${base}: $(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" | grep -c '^>' || true)"
+    fi
+  else
+    warn "no ${SNAP_DIR}/dns-before.txt baseline — cannot prove the other cluster's records are untouched.
+       Take one BEFORE the next standup so this check can run."
+  fi
 }
 
 # ============================================================================ dispatch