fix(standup): preflight must always report — it was an env fork in disguise

preflight `die`d when EXTERNALDNS_TXT_OWNER_ID was empty, the sentinel, or
`default`. Two problems, found by pointing it at dev:

1. `default` is DEV'S LEGITIMATE owner id — its live records are registered
   under it and cannot be renamed without orphaning them. Dying on it meant a
   script whose entire claim is "environment-agnostic, no dev/prod fork"
   (R-ENV-02) could never run against dev at all. That is a fork in disguise.

2. Dev's value is currently EMPTY, because dev runs dev-live@a4a3c2e — based on
   main@4809199, which predates the commit that added the knob. That is dev's
   long-standing status quo, not a fault. The die hid the ENTIRE remaining
   health report behind it, which is backwards for a diagnostic.

preflight is now a pure diagnostic: it always finishes and reports, and the only
thing it refuses over is the --expect-server target guard, because that one
means you are pointed at the wrong cluster. Bad owner-id values are reported
with a warning explaining why each is dangerous and for whom.

THE REFUSAL MOVES, IT DOES NOT DISAPPEAR: step_externaldns — the MUTATING path,
where acting on a bad value is what actually deletes another cluster's records —
still dies on empty, sentinel and `default`. Verified both halves after the
change: prod preflight still reports "distinct and non-default: OK", and the
mutating guard is intact.

Confirmed with it: dev is 13/13 Kustomizations Ready on dev-live@a4a3c2e, all
HelmReleases Ready, nothing suspended — healthy throughout prod's standup.

Refs R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-05 10:06:01 -05:00
1 parent 4f37fc375e
commit acb5cabdad
1 file changed
+25 -4
+25 -4
View File
@@ -172,10 +172,31 @@ step_preflight() {
# by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the
# parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the
# single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records.
[ -n "${owner}" ] || die "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns would fall back to 'default', which is dev's owner."
[ "${owner}" != "unset-see-cluster-vars" ] || die "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply."
[ "${owner}" != "default" ] || die "EXTERNALDNS_TXT_OWNER_ID is 'default' — that is DEV's owner id. Refusing."
info "owner id is distinct and non-default: OK"
# preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the
# target guard, because that one means you are pointed at the wrong cluster. Everything else is
# reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting
# on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health
# report behind a condition that was, for dev, simply the long-standing status quo.
if [ -z "${owner}" ]; then
warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'.
Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse."
elif [ "${owner}" = "unset-see-cluster-vars" ]; then
warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply.
'externaldns' will refuse until a real per-environment value is set."
elif [ "${owner}" = "default" ]; then
# NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records
# are registered under it and cannot be renamed without orphaning them. Dying on it would make
# this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in
# a script whose whole claim is that it has none. preflight only READS, so it warns; the
# MUTATING path (step_externaldns) still refuses outright.
warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value.
That is correct for DEV (its records are registered under it) and WRONG for any second cluster
sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the
2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and
policy: sync deleted the records behind them."
else
info "owner id is distinct and non-default: OK"
fi
case "${s3}" in
""|TBD-*) warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.