fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)

P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

clusters/base/platform.yaml

@@ -13,6 +13,33 @@ spec:
   timeout: 5m
   prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
   wait: true           # block `apps` until platform is healthy
+  # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
+  #
+  # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
+  # `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
+  # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
+  # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
+  # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
+  #
+  # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
+  # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
+  # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
+  # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
+  # writes a newly-added `cluster-vars` knob — so this one edge closes the race.
+  #
+  # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
+  # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
+  # velero) chains transitively through it. One edge therefore protects the whole graph.
+  #
+  # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
+  # ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
+  # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
+  # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
+  # entrypoint should stop the line, not let consumers render against a half-applied variable set.
+  # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
+  # health-gates its own children (see clusters/dev/sync.yaml).
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/prod/kustomization.yaml

@@ -5,10 +5,20 @@
 # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
 # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
 # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
+#
+# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
+# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
+# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
+# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
+# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
+# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
+# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
+# done here: prod is design-only and its root object is the prod bring-up owner's call.)
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ../base
+  - sync.yaml          # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
 patches:
   - target:
       kind: ConfigMap

clusters/prod/sync.yaml

@@ -0,0 +1,32 @@
+---
+# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
+#
+# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
+# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
+# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
+# its own sync.yaml -> ./clusters/prod"; this is it.
+#
+# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
+# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
+# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
+# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
+# not exist, and NOTHING downstream would ever reconcile.
+#
+# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
+# Ready on apply success without health-gating its children (platform does its own wait/health
+# gating). Do not "harden" this to true.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./clusters/prod