fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)

P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

bootstrap/flux/bootstrap.sh

@@ -69,16 +69,59 @@ for bin in flux kubectl; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
-# --- 1. install Flux controllers ---------------------------------------------
+# --- 1. install Flux controllers (rendered through ./components) --------------
 # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
 # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
 # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
-if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
-  echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
-  flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
+#
+# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
+# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
+# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
+# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
+#
+# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
+# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
+# documents for hand-installed clusters ("Customize Flux manifests"):
+#
+#     flux install --export  >  gotk-components.yaml     # exact manifests `flux install` would apply
+#     kustomize overlay      =  ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
+#     kubectl apply -k
+#
+# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
+# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
+# a temp dir — the repo tree is never written to.
+#
+# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
+# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
+echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+flux_install_args=(--namespace="${FLUX_NAMESPACE}")
+[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
+
+if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
+  echo "    WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
+  echo "             StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
+  flux install "${flux_install_args[@]}"
 else
-  echo "==> [1/3] flux install (default controllers only)"
-  flux install --namespace="${FLUX_NAMESPACE}"
+  components_dir="${script_dir}/components"
+  [ -f "${components_dir}/kustomization.yaml" ] || {
+    echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
+    echo "       (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
+    exit 1
+  }
+  render_dir="$(mktemp -d)"
+  trap 'rm -rf "${render_dir}"' EXIT
+  echo "    rendering flux install --export through ${components_dir}"
+  flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
+  cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
+  # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
+  # version bump) — better a hard stop than silently installing without the feature gate.
+  kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
+  grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
+    echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
+    echo "       Check ./components/kustomization.yaml against this flux CLI version." >&2
+    exit 1
+  }
+  kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
 # --- 2. verify the out-of-band age key Secret exists -------------------------