docs: INCIDENT — Flux post-build substitution fails OPEN, briefly broke Cilium

Merging PR #4 put cilium agents into Init:Error on 2/3 dev nodes for ~6 minutes. Self-healed;
cluster fully green. Recorded because the mechanism is systemic and far more dangerous on a
cold prod bootstrap.

Root cause: cluster-vars (which DEFINES a knob) is owned by the flux-system Kustomization,
while platform/ (which REFERENCES it) is owned by the platform Kustomization, with no
dependency ordering between defining a variable and using it. Flux post-build substitution is
NON-STRICT by default, so during the window where platform rendered the new manifest against
the old cluster-vars, ${K3S_API_HOST} became the EMPTY STRING. Cilium upgraded with
k8sServiceHost: "" and fell back to ClusterIP 10.43.0.1 — which only the Cilium agent can
translate, and that agent was the thing restarting. Its config init container then could not
reach the API server at all.

Proven by the HelmRelease config digest going e83bf695 -> eebd6c88 -> e83bf695: the config
changed and then changed back to BYTE-IDENTICAL. The pre-merge byte-identity verification was
correct — the steady state is identical; the damage was a transient intermediate render.

optional: false does NOT protect against this: it requires the ConfigMap to exist, not that
every referenced key is defined.

Fix recommended, NOT applied unilaterally: enable the StrictPostBuildSubstitutions feature gate
on kustomize-controller so an undefined variable FAILS the Kustomization instead of silently
emptying it — fail-closed, per this project's own doctrine. Requires first proving every ${VAR}
in every substituted path is defined or correctly escaped, or it would fail legitimate embedded
shell expansions. Interim discipline: add a knob in one commit, reference it in a later one.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01.

DECISIONS-LOG.md

@@ -1496,3 +1496,89 @@ thing that ever closes a requirement.
   chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
   the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
+
+---
+
+## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
+
+Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
+`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
+and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
+Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
+prod bootstrap than it was here.
+
+### What happened
+
+The HelmRelease config digest tells the whole story:
+
+```
+v8   05:49:39  digest e83bf695…      <- before
+v9   05:50:11  digest eebd6c88…      <- the bad render
+v10  05:55:23  digest e83bf695…      <- identical to v8 again
+```
+
+The config changed and then changed **back to byte-identical**. Nothing in git differed
+semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
+`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
+`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
+state is identical. The damage happened in a transient intermediate state.
+
+### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
+
+- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
+- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
+  Kustomization, via `postBuild.substituteFrom: cluster-vars`.
+- There is **no dependency ordering between defining a variable and using it.**
+
+So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
+which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
+undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
+upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
+which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
+`config` init container could not reach the API server:
+
+```
+level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
+  error="dial tcp 10.43.0.1:443: i/o timeout"
+level=fatal msg="Build config failed"
+```
+
+Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
+correct value, digest returned to `e83bf695…`, and all three agents came up.
+
+`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
+the ConfigMap to *exist*, not that every referenced key is defined.
+
+### Why this matters much more for prod
+
+On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
+race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
+there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
+never come up at all — with no healthy peer to recover from. This is a plausible
+bring-up-blocker for `clusters/prod`.
+
+### The fix — make substitution FAIL CLOSED
+
+`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
+substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
+the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
+("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
+this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
+failure*.
+
+**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
+that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
+embedded scripts. The safe sequence, before flipping it:
+1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
+   `cluster-vars` or correctly escaped as `$${VAR}`.
+2. Then enable the gate, and let CI + a dev reconcile prove it.
+
+Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
+reference it in a later one**, so the definition is always reconciled before its first use.
+
+### Operational note worth keeping
+
+A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
+`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
+set, and the agent that translates that ClusterIP is the one being replaced. Keep
+`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.