From acb5cabdad71a149e7fdca8220b601179e98fdaa Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 10:06:01 -0500 Subject: [PATCH] =?UTF-8?q?fix(standup):=20preflight=20must=20always=20rep?= =?UTF-8?q?ort=20=E2=80=94=20it=20was=20an=20env=20fork=20in=20disguise?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit preflight `die`d when EXTERNALDNS_TXT_OWNER_ID was empty, the sentinel, or `default`. Two problems, found by pointing it at dev: 1. `default` is DEV'S LEGITIMATE owner id — its live records are registered under it and cannot be renamed without orphaning them. Dying on it meant a script whose entire claim is "environment-agnostic, no dev/prod fork" (R-ENV-02) could never run against dev at all. That is a fork in disguise. 2. Dev's value is currently EMPTY, because dev runs dev-live@a4a3c2e — based on main@4809199, which predates the commit that added the knob. That is dev's long-standing status quo, not a fault. The die hid the ENTIRE remaining health report behind it, which is backwards for a diagnostic. preflight is now a pure diagnostic: it always finishes and reports, and the only thing it refuses over is the --expect-server target guard, because that one means you are pointed at the wrong cluster. Bad owner-id values are reported with a warning explaining why each is dangerous and for whom. THE REFUSAL MOVES, IT DOES NOT DISAPPEAR: step_externaldns — the MUTATING path, where acting on a bad value is what actually deletes another cluster's records — still dies on empty, sentinel and `default`. Verified both halves after the change: prod preflight still reports "distinct and non-default: OK", and the mutating guard is intact. Confirmed with it: dev is 13/13 Kustomizations Ready on dev-live@a4a3c2e, all HelmReleases Ready, nothing suspended — healthy throughout prod's standup. Refs R-ENV-02. Co-Authored-By: Claude Opus 5 --- bootstrap/standup.sh | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/bootstrap/standup.sh b/bootstrap/standup.sh index da111a6..4cd858f 100755 --- a/bootstrap/standup.sh +++ b/bootstrap/standup.sh @@ -172,10 +172,31 @@ step_preflight() { # by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the # parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the # single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records. - [ -n "${owner}" ] || die "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns would fall back to 'default', which is dev's owner." - [ "${owner}" != "unset-see-cluster-vars" ] || die "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply." - [ "${owner}" != "default" ] || die "EXTERNALDNS_TXT_OWNER_ID is 'default' — that is DEV's owner id. Refusing." - info "owner id is distinct and non-default: OK" + # preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the + # target guard, because that one means you are pointed at the wrong cluster. Everything else is + # reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting + # on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health + # report behind a condition that was, for dev, simply the long-standing status quo. + if [ -z "${owner}" ]; then + warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'. + Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse." + elif [ "${owner}" = "unset-see-cluster-vars" ]; then + warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply. + 'externaldns' will refuse until a real per-environment value is set." + elif [ "${owner}" = "default" ]; then + # NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records + # are registered under it and cannot be renamed without orphaning them. Dying on it would make + # this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in + # a script whose whole claim is that it has none. preflight only READS, so it warns; the + # MUTATING path (step_externaldns) still refuses outright. + warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value. + That is correct for DEV (its records are registered under it) and WRONG for any second cluster + sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the + 2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and + policy: sync deleted the records behind them." + else + info "owner id is distinct and non-default: OK" + fi case "${s3}" in ""|TBD-*) warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.