The last steps of a prod bring-up were living in shell history: resume the
tiers in the right order, clear a wedged HelmRelease, fix external-dns without
deleting the other cluster's DNS. That is the same failure the Kyverno CRD
pre-apply had, so it gets the same treatment.
bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six
idempotent steps: preflight, crds, unwedge, externaldns, resume, verify.
THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against
the live API server before any step runs; there is no override flag. Dev is
live and shares the keithjackson.dev Cloudflare zone with prod, and on
2026-09-04 prod deleted every one of dev's public records. Negative control
run: prod's kubeconfig with dev's server URL refuses and changes nothing.
preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base
sentinel, or `default` (dev's owner) — the exact condition that caused the
incident. It also surfaces SUSPENDED HelmReleases separately, because a
suspended release keeps its last Ready condition and a stale `Ready=True`
otherwise hides that Flux is not managing it at all (external-dns today).
externaldns is the dangerous step, and the ordering IS the mitigation. A Helm
upgrade resets replicas to 1 and applies new args in ONE operation, so there is
no window to check the result first; if --txt-owner-id were missing the pod
starts as owner `default` and syncs away dev's records. So the step PROVES THE
OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and
assert --txt-owner-id=<owner> appears, aborting if not. Rendering is
deterministic and touches nothing, which turns a race into a precondition. Then
snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if
wrong), and diff the zone.
resume walks the dependency order and STOPS at the first tier that does not go
Ready — a partial understood bring-up beats a forced green. velero is skipped
while S3_BACKUP_ENDPOINT is the TBD placeholder.
TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming:
1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22
— the policy-report types are in the wgpolicyk8s.io group. A preflight that
under-reports reads as "the pre-apply failed" when it succeeded. Fixed and
re-run: 22.
2. The documented node-health invocation used -e k3s_hosts=k3s_prod and
SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep
their own target var (node_health_hosts, memattr_survey_hosts,
vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped
elsewhere — a good design I did NOT change. The hazard is that the wrong flag
does not error, it falls through to the dev default. Caught only by
--list-hosts; the doc now leads with that warning and a dry-run command.
Verified after fixing: node-health and all six site.yml plays target prod.
.claude/settings.local.json — permission rules for the above. Every entry
hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible
groups), so the permission strings are themselves prod-scoped and cannot be
turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which
would have been blanket cluster access wearing a narrow-looking mask.
Refs R-ENV-02, R-GITOPS-01, R-SEC-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
