chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup)

Closes the transition opened in merge 1 (63f98aa). `prune: true` is restored on the
`platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster,
not merely expected:

  13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854
  Namespace/kube-system          label kustomize.toolkit.fluxcd.io/name = cilium
  HelmRelease/cilium             label ... = cilium
  HelmRepository/cilium          label ... = cilium
  platform  inventory: 0 entries matching kube-system__Namespace | kube-system_cilium_*
  cilium    inventory: exactly those three objects
  Cilium pods UNCHANGED from the pre-merge baseline —
    cilium-2f8h8 r=0 2026-09-01T05:55:27Z
    cilium-425m8 r=1 2026-08-30T13:25:10Z
    cilium-54cww r=0 2026-09-01T05:55:26Z
  HelmRelease still v11

So the relabel deterministically preceded any prune, exactly as the revision-gated
dependsOn edge predicted, and the ownership move cost zero pod restarts.

The in-file comment now records the lesson rather than the temporary state: before moving
ANY platform/* component in future, check whether it is the sole declarer of a shared
Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system
under ./platform, which would have turned a component-scoped prune into
`delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium.

L0 substitution gate: 103 assertions, 0 hazards.

Serves: R-PLAT-01, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

DECISIONS-LOG.md

@@ -1496,3 +1496,265 @@ thing that ever closes a requirement.
   chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
   the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
+
+---
+
+## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
+
+Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
+`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
+and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
+Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
+prod bootstrap than it was here.
+
+### What happened
+
+The HelmRelease config digest tells the whole story:
+
+```
+v8   05:49:39  digest e83bf695…      <- before
+v9   05:50:11  digest eebd6c88…      <- the bad render
+v10  05:55:23  digest e83bf695…      <- identical to v8 again
+```
+
+The config changed and then changed **back to byte-identical**. Nothing in git differed
+semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
+`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
+`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
+state is identical. The damage happened in a transient intermediate state.
+
+### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
+
+- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
+- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
+  Kustomization, via `postBuild.substituteFrom: cluster-vars`.
+- There is **no dependency ordering between defining a variable and using it.**
+
+So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
+which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
+undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
+upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
+which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
+`config` init container could not reach the API server:
+
+```
+level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
+  error="dial tcp 10.43.0.1:443: i/o timeout"
+level=fatal msg="Build config failed"
+```
+
+Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
+correct value, digest returned to `e83bf695…`, and all three agents came up.
+
+`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
+the ConfigMap to *exist*, not that every referenced key is defined.
+
+### Why this matters much more for prod
+
+On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
+race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
+there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
+never come up at all — with no healthy peer to recover from. This is a plausible
+bring-up-blocker for `clusters/prod`.
+
+### The fix — make substitution FAIL CLOSED
+
+`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
+substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
+the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
+("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
+this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
+failure*.
+
+**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
+that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
+embedded scripts. The safe sequence, before flipping it:
+1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
+   `cluster-vars` or correctly escaped as `$${VAR}`.
+2. Then enable the gate, and let CI + a dev reconcile prove it.
+
+Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
+reference it in a later one**, so the definition is always reconciled before its first use.
+
+### Operational note worth keeping
+
+A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
+`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
+set, and the agent that translates that ClusterIP is the one being replaced. Keep
+`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.
+
+---
+
+## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.**
+
+Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit),
+prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected
+here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything
+else is cited inference.
+
+### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere)
+
+| form | behaviour |
+|---|---|
+| `${DEFINED}` | substituted |
+| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) |
+| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** |
+| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. |
+| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. |
+| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** |
+| `$VAR` / `$(cmd)` | untouched, no escaping needed |
+| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** |
+
+Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict
+mode by design ("they intentionally handle the unset case"). **That is why strict mode would never
+have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required.
+
+**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal
+"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place.
+
+**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them,
+so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst,
+and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on
+stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke
+every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.)
+
+### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT
+
+It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly
+and keep last-good". **The actual ordering fix is one line**, verified against
+`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is
+**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the
+dependent cannot reconcile revision N until the dependency has APPLIED revision N.
+
+`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains
+transitively through it. So:
+
+```yaml
+# clusters/base/platform.yaml
+spec:
+  dependsOn:
+    - name: flux-system     # cluster-vars must be applied at THIS revision first
+```
+
+**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under
+`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with
+doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`.
+
+Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** —
+`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization
+applies its own resources, so it would always read the PREVIOUS value. That converts an occasional
+race into a permanent one-reconcile lag.
+
+### CORRECTION 2 (an agent's) — the safety machinery DID fire
+
+A research pass concluded that Helm considered the bad release healthy (because
+`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and
+therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm
+release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which
+is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was
+blast radius (2 of 3 agents down at once), not a missed detection.
+
+### Verified, and worth knowing
+
+- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current.
+- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips
+  this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade
+  regardless of today's decision.
+- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`.
+  Helm was never going to reject the bad render.
+- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the
+  environment exports the API ClusterIP and there is no kube-proxy to provision it.
+- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade
+  check must live outside Helm.
+- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's
+  silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces
+  (`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`).
+- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the
+  Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction
+  would have actively re-asserted the broken value.
+
+### RANKED PLAN
+
+| P | Action | State |
+|---|---|---|
+| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) |
+| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE |
+| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions |
+| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. |
+| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node |
+| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve |
+| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default |
+| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process |
+| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) |
+| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — |
+
+### Follow-ups surfaced, not yet actioned
+- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot.
+- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative
+  `${...}` that can silently drift from the real HelmRelease.
+- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
+  out while a legitimate Longhorn upgrade is still running.
+
+---
+
+## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
+
+All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
+
+### Landed and verified on the live cluster
+
+| Change | Verified |
+|---|---|
+| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
+| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
+| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
+| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
+| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
+
+### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
+
+Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
+`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
+
+```sh
+[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"   # LAST line of the hook
+```
+
+On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
+the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
+
+**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
+long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
+execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
+**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
+dead code is not a safe no-op — it is the first execution of never-tested code, and should be
+treated with the caution of new code, not of a one-character escape.
+
+Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
+on both paths, but only by accident of which statement happened to be last — made explicit there too
+so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
+Kustomizations Ready, both OIDC providers still registered.
+
+### STILL TO DO — strict substitution is committed but NOT ACTIVE
+
+`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
+`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
+ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
+controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
+standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
+~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
+
+**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
+With the gate on but the ordering edge absent, the same race would turn a transient empty render
+into a transient hard Kustomization failure.
+
+### Prod consequences recorded this session
+
+- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
+  soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
+  first" safety net that the last two days have leaned on heavily, twice.
+- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
+  currently points at Gitea running on the dev cluster.
+- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
+  with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
+  Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.

apps/vms/unifi/30-virtualmachine.yaml

@@ -92,10 +92,13 @@ spec:
             # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
             # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
             # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
+            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write
+            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace")
+            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply.
+            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
+            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
+            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
+            # wording in clusters/base/apps-vms.yaml overstated it.
             userData: |-
               #cloud-config
               hostname: unifi

apps/vms/unifi/50-service-l4.yaml

@@ -27,7 +27,7 @@
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
 # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
-# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
+# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
 # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
 #
 # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an

apps/wave-0/gitea.yaml

@@ -127,8 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
-                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
+                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

apps/wave-1/nextcloud.yaml

@@ -125,8 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
-                \ positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
+                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

bootstrap/flux/README.md

@@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
 | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
 
+## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
+
+Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
+and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
+
+```
+$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
+ "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
+# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
+```
+
+No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
+`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
+documents for hand-installed clusters ([Customize Flux
+manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
+
+```
+flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
+components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
+kubectl kustomize <tmp> | kubectl apply -f -
+```
+
+Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
+(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
+more feature gates) is another entry in `components/kustomization.yaml`.
+
+### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
+
+> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
+> controller Deployment is outside Flux's inventory. Someone must run step 1.
+
+Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
+
+```bash
+# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
+d=$(mktemp -d)
+flux install --export --namespace=flux-system \
+  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
+cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
+kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
+kubectl -n flux-system rollout status deploy/kustomize-controller
+```
+
+**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
+reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
+is lost. Confirm afterwards:
+
+```bash
+kubectl -n flux-system get deploy kustomize-controller \
+  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
+flux get kustomizations -A                                  # all 11 must return to Ready
+```
+
+**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
+simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):

bootstrap/flux/bootstrap.sh

@@ -69,16 +69,59 @@ for bin in flux kubectl; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
-# --- 1. install Flux controllers ---------------------------------------------
+# --- 1. install Flux controllers (rendered through ./components) --------------
 # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
 # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
 # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
-if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
+#
-  echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
+# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
-  flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
+# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
+# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
+# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
+#
+# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
+# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
+# documents for hand-installed clusters ("Customize Flux manifests"):
+#
+#     flux install --export  >  gotk-components.yaml     # exact manifests `flux install` would apply
+#     kustomize overlay      =  ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
+#     kubectl apply -k
+#
+# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
+# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
+# a temp dir — the repo tree is never written to.
+#
+# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
+# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
+echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+flux_install_args=(--namespace="${FLUX_NAMESPACE}")
+[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
+
+if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
+  echo "    WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
+  echo "             StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
+  flux install "${flux_install_args[@]}"
 else
-  echo "==> [1/3] flux install (default controllers only)"
+  components_dir="${script_dir}/components"
-  flux install --namespace="${FLUX_NAMESPACE}"
+  [ -f "${components_dir}/kustomization.yaml" ] || {
+    echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
+    echo "       (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
+    exit 1
+  }
+  render_dir="$(mktemp -d)"
+  trap 'rm -rf "${render_dir}"' EXIT
+  echo "    rendering flux install --export through ${components_dir}"
+  flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
+  cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
+  # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
+  # version bump) — better a hard stop than silently installing without the feature gate.
+  kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
+  grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
+    echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
+    echo "       Check ./components/kustomization.yaml against this flux CLI version." >&2
+    exit 1
+  }
+  kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
 # --- 2. verify the out-of-band age key Secret exists -------------------------

bootstrap/flux/components/kustomization.yaml

@@ -0,0 +1,64 @@
+---
+# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
+#
+# WHY THIS DIRECTORY EXISTS
+# =========================
+# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
+# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
+# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
+#
+#     kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+#     -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
+#
+# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
+# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
+#
+# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
+# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
+# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
+# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
+# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
+# repo, so the manifests can never drift from the pinned FLUX_VERSION
+# (infra/devcontainer/versions.env) the way a vendored copy would.
+#
+# Every future controller customization (vertical scaling, tolerations, extra feature gates,
+# --concurrent) belongs HERE, as another entry under `patches:`.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
+  - gotk-components.yaml
+patches:
+  # -------------------------------------------------------------------------------------------
+  # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
+  #
+  # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
+  # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
+  # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
+  # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
+  #
+  # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
+  # `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
+  # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
+  # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
+  #
+  # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
+  # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
+  #
+  # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
+  # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
+  #
+  # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
+  # list, so a strategic-merge patch would REPLACE every arg `flux install` generates
+  # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
+  # and silently drop any the CLI adds in a future version.
+  - target:
+      group: apps
+      version: v1
+      kind: Deployment
+      name: kustomize-controller
+      namespace: flux-system
+    patch: |-
+      - op: add
+        path: /spec/template/spec/containers/0/args/-
+        value: --feature-gates=StrictPostBuildSubstitutions=true

catalog/README.md

@@ -28,6 +28,16 @@ inherit every guarantee automatically.
   `auth.groups`; `native`/`none` are documented exceptions only.
 - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
   schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
+- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
+  `lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
+  `apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
+  envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
+  *shell* is replaced with the empty string before the container ever starts (this silently killed
+  gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
+  need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
+  written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
+  `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
+  cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
 
 ## Validate before commit — single command, no cluster (R-TEST-02)
 ```bash

catalog/gitea.yaml

@@ -109,7 +109,19 @@ lifecycle:
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
           }
-          if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
+          # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
+          # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
+          # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
+          # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
+          # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
+          # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
+          # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
+          # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
           else
             # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
@@ -150,6 +162,12 @@ lifecycle:
               gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
             fi
           fi
+          # A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
+          # it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
+          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
+          # the moment its OIDC guard was fixed and the block actually ran for the first time.
+          exit 0
 exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/nextcloud.yaml

@@ -111,7 +111,19 @@ lifecycle:
           # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
           OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
             || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
-          if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
+          # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
+          # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
+          # installing/enabling user_oidc nor registering the provider. SSO kept working only because
+          # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
+          # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
+          # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
+          # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
+          # `--clientsecret-env` and the bare uses below always worked and only the guard broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
             exit 0
           fi
@@ -191,7 +203,17 @@ lifecycle:
             [ "$missing" = 0 ] && break
             p=$((p + 1)); sleep 2
           done
-          [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          if [ "$p" -ge 15 ]; then
+            nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          fi
+          # A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
+          # the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
+          # SUCCESS path the test returned 1 and became the script's exit status. That was latent for
+          # as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
+          # exposed it immediately. Every failure path here logs loudly and writes a marker file and
+          # deliberately does NOT crash the pod — the break-glass local admin must stay usable.
+          exit 0
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
 storage:

catalog/render.py

@@ -29,6 +29,7 @@ conformance harness stays stdlib-only and only shells out to this script + diffs
 from __future__ import annotations
 
 import argparse
+import re
 import sys
 from pathlib import Path
 
@@ -936,6 +937,124 @@ def r_virtualmachine(s: dict) -> dict:
     }
 
 
+# =============================================================================
+# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
+# Flux's envsubst would silently eat before the workload ever runs.
+# =============================================================================
+# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
+# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
+# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
+# document — including any shell script carried verbatim in a `lifecycle` hook or an env
+# value — at reconcile time, long before a container sees it. The escape contract is stated
+# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
+#
+#   `${VAR}`   -> SUBSTITUTED (empty when unset)   => a shell brace must be written `$${VAR}`,
+#                                                     which envsubst emits back as `${VAR}`.
+#   `$VAR`     -> left alone                       => bare shell vars need NO escaping.
+#   `$(cmd)`   -> left alone                       => command substitution needs NO escaping.
+#   `${A[0]}`  -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
+#
+# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
+# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
+# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
+# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
+# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
+# exactly the form a defensive shell script uses. So the renderer checks it itself.
+CLUSTER_VARS_PATH = (
+    Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
+)
+
+# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
+# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
+# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
+# (`${A[0]}`) are caught too rather than skipped.
+_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
+
+_CLUSTER_VARS_CACHE: dict = {}
+
+
+def cluster_vars_keys(path: Path | None = None) -> set:
+    """The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
+
+    Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
+    renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
+    FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
+    """
+    key = str(path or CLUSTER_VARS_PATH)
+    if key not in _CLUSTER_VARS_CACHE:
+        try:
+            doc = yaml.safe_load(Path(key).read_text())
+            data = doc.get("data") if isinstance(doc, dict) else None
+            _CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
+        except Exception:  # pragma: no cover - absent/malformed => fail closed
+            _CLUSTER_VARS_CACHE[key] = set()
+    return _CLUSTER_VARS_CACHE[key]
+
+
+def substitution_violations(text: str, known: set | None = None) -> list:
+    """Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
+
+    Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
+    when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
+    a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
+    envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
+    """
+    keys = cluster_vars_keys() if known is None else known
+    out = []
+    for lineno, line in enumerate(text.splitlines(), start=1):
+        for m in _BRACED_EXPANSION_RE.finditer(line):
+            if len(m.group(1)) % 2 == 0:
+                continue  # even run of `$` => escaped; envsubst emits it back literally
+            expr = m.group(2)
+            if _PLAIN_IDENT_RE.match(expr):
+                if expr in keys:
+                    continue  # a declared cluster-vars knob, intentionally substituted
+                reason = (
+                    f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
+                    "it with the EMPTY STRING"
+                )
+            elif "[" in expr:
+                reason = (
+                    "indexed expansion — envsubst hard-fails with 'missing closing brace' "
+                    "and the whole Kustomization stops reconciling"
+                )
+            else:
+                reason = (
+                    "shell expansion (default-operator/parameter form) — envsubst eats it "
+                    "at reconcile time and STRICT mode exempts this form, so nothing else "
+                    "will catch it"
+                )
+            out.append((lineno, m.group(0), reason))
+    return out
+
+
+def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
+    """Refuse to emit a manifest carrying an unescaped braced expansion (see above).
+
+    Raises ValueError listing every offending token. The fix is almost always to escape the
+    shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
+    value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
+    and write it as a bare `${KNOB}`.
+    """
+    bad = substitution_violations(text)
+    if not bad:
+        return
+    lines = [
+        f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
+        "postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
+    ]
+    for lineno, token, reason in bad:
+        lines.append(f"  line {lineno}: {token}  -> {reason}")
+    lines.append(
+        "  FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
+        "`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
+        "must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
+        "Contract: clusters/base/apps-vms.yaml."
+    )
+    raise ValueError("\n".join(lines))
+
+
 # =============================================================================
 # Top-level render + emit
 # =============================================================================
@@ -1026,8 +1145,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
     return docs
 
 
-def emit(docs: list) -> str:
+def emit(docs: list, origin: str = "<render>") -> str:
-    """Deterministic multi-document YAML text (golden-diffable)."""
+    """Deterministic multi-document YAML text (golden-diffable).
+
+    Every emitted document is checked against the substitution contract before it is handed
+    back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
+    envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
+    ValueError rather than emitting such a manifest.
+    """
     parts = []
     for comment, resource in docs:
         body = yaml.dump(
@@ -1037,7 +1162,9 @@ def emit(docs: list) -> str:
             width=4096,
         )
         parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
-    return "".join(parts)
+    text = "".join(parts)
+    assert_substitution_safe(text, origin)
+    return text
 
 
 def load_sd(path: Path) -> dict:
@@ -1076,7 +1203,9 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
     src_dir = catalog_dir if catalog_dir is not None else path.parent
     rev = build_reverse_map(_load_catalog(src_dir))
     inbound = rev.get(sd["name"], [])
-    return emit(render(sd, image_automation=image_automation, inbound=inbound))
+    return emit(
+        render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
+    )
 
 
 # =============================================================================
@@ -1729,6 +1858,90 @@ def _selftest() -> int:
     check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
           "wave kustomization must list exactly the wave's resources")
 
+    # 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
+    #    shell script carries an unescaped `${...}`. Every rendered manifest lands under a
+    #    Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
+    #    the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
+    #    -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
+    known = {"BASE_DOMAIN", "GATEWAY_VIP"}
+    # allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
+    check(
+        substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
+        "a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
+    )
+    # allowed: escaped shell braces, and the forms envsubst leaves alone.
+    check(
+        substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
+        "an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
+    )
+    check(
+        substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
+        "bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
+    )
+    # `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
+    check(
+        len(substitution_violations("$$${X:-}", known)) == 1,
+        "an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
+    )
+    # flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
+    guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
+    check(
+        len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
+        "an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
+    )
+    # flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
+    check(
+        len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
+        "an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
+    )
+    idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
+    check(
+        len(idx) == 1 and "missing closing brace" in idx[0][2],
+        "an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
+    )
+    # the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
+    # bug RAISES rather than writing a manifest whose guard is dead on arrival.
+    bad_sd = {
+        "name": "subbad",
+        "source": {"image": "a:1"},
+        "lifecycle": {
+            "postStart": {
+                "exec": {
+                    "command": [
+                        "/bin/sh",
+                        "-c",
+                        'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                    ]
+                }
+            }
+        },
+    }
+    try:
+        emit(render(bad_sd), origin="<selftest>")
+        check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
+    except ValueError as exc:
+        check(
+            "SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
+            "the refusal must name the offending token AND state the `$${...}` fix",
+        )
+    # the ESCAPED form of the same hook renders normally (the fix must be emittable).
+    good_sd = dict(bad_sd, name="subgood")
+    good_sd["lifecycle"] = {
+        "postStart": {
+            "exec": {
+                "command": [
+                    "/bin/sh",
+                    "-c",
+                    'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                ]
+            }
+        }
+    }
+    check(
+        "$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
+        "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
+    )
+
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

ci/l0.sh

@@ -100,6 +100,43 @@ else
   fi
 fi
 
+# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
+# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
+# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
+# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
+# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
+# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
+# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
+# a legitimate manifest, and that no expansion silently empties today.
+#
+# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
+# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
+# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
+# attributes no requirement, so the requirement-scoped selection would filter this check
+# out of exactly the change that introduces the bug. The invariant is whole-repo (same
+# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
+# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
+# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
+SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
+log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
+if [ ! -f "$SUBST_CHECK" ]; then
+  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
+  summary "- subst-safety: **SKIPPED** (check missing)"
+  if [ "$rc" -eq 0 ]; then rc=3; fi
+else
+  subst_rc=0
+  bash "$SUBST_CHECK" || subst_rc=$?
+  case "$subst_rc" in
+    0) ok "Flux postBuild substitution safety passed"
+       summary "- subst-safety: **passed**" ;;
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
+    *) err "Flux postBuild substitution safety FAILED"
+       summary "- subst-safety: **FAILED**"; rc=1 ;;
+  esac
+fi
+
 case "$rc" in
   0) ok "L0 stage complete" ;;
   3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/kustomization.yaml

@@ -5,6 +5,13 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).

clusters/base/platform.yaml

@@ -11,8 +11,53 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
+  # on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
+  # live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
+  # `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
+  # any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
+  #
+  # LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
+  # Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
+  # made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
+  # Cilium. Before moving ANY platform/* component in future, check whether it is the sole
+  # declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
+  prune: true
   wait: true           # block `apps` until platform is healthy
+  # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
+  #
+  # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
+  # `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
+  # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
+  # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
+  # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
+  #
+  # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
+  # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
+  # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
+  # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
+  # writes a newly-added `cluster-vars` knob — so this one edge closes the race.
+  #
+  # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
+  # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
+  # velero) chains transitively through it. One edge therefore protects the whole graph.
+  #
+  # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
+  # ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
+  # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
+  # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
+  # entrypoint should stop the line, not let consumers render against a half-applied variable set.
+  # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
+  # health-gates its own children (see clusters/dev/sync.yaml).
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/prod/kustomization.yaml

@@ -5,11 +5,144 @@
 # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
 # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
 # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
+#
+# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
+# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
+# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
+# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
+# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
+# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
+# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
+# done here: prod is design-only and its root object is the prod bring-up owner's call.)
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ../base
+  - sync.yaml          # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
 patches:
+  # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
+  # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
+  # The `sops-age` Secret is created OUT OF BAND on the prod cluster
+  # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
+  # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: platform
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-0
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-1
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-2
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: oauth2-proxy
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: headlamp
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: crafty
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: velero
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
+  #
+  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  #
+  # Two independent reasons, both consequences of the split-horizon design:
+  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
+  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
+  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
+  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
+  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
+  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
+  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  #
+  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
+  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
+  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  #
+  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
+  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
+  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  - patch: |-
+      apiVersion: source.toolkit.fluxcd.io/v1
+      kind: GitRepository
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        ref:
+          branch: main
   - target:
       kind: ConfigMap
       name: cluster-vars

clusters/prod/sync.yaml

@@ -0,0 +1,32 @@
+---
+# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
+#
+# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
+# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
+# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
+# its own sync.yaml -> ./clusters/prod"; this is it.
+#
+# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
+# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
+# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
+# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
+# not exist, and NOTHING downstream would ever reconcile.
+#
+# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
+# Ready on apply success without health-gating its children (platform does its own wait/health
+# gating). Do not "harden" this to true.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./clusters/prod

conformance/run.sh

@@ -57,17 +57,15 @@ FORCE_NO_CLUSTER=0
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
 # exemption is explicit and reviewable, not silent.
-#   R-ENV-02   (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
+#
-#                                    coverage.md). The coarse path->requirement map keys
+# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
-#                                    clusters/* AND ci/* to it and the R-ENV-02 doc token
+# because they were unscaffolded while the coarse path->requirement map keys clusters/*
-#                                    appears in platform/kustomization.yaml's header, so ANY
+# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
-#                                    infra/overlay/CI edit (e.g. enabling a component, bumping
+# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
-#                                    SPEC_PIN) would otherwise false-block on it.
+# substitution mechanism that IS the "authored once, promoted" contract, so they are
-#   R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
+# covered rather than exempt and the completeness gate can hold them to it. That test
-#                                    path-map keys to it, so editing any ci/ file (SPEC_PIN
+# names its own limits in its header: it proves the STATIC property (no manifest renders
-#                                    adoption included) would false-block.
+# an undefined variable), not the live-cluster half of either requirement.
-# These are visible GAPs, not hidden — removing them from this list is the right move the
-# moment WS-TEST adds their verifying tests.
 #
 # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
 # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
@@ -83,7 +81,7 @@ FORCE_NO_CLUSTER=0
 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
 # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
-DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
+DOCUMENTED_GAPS=""
 
 usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
 
@@ -108,7 +106,11 @@ log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
-  local flags=(--allow-gap "${DOCUMENTED_GAPS}")
+  # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
+  # requirement is exempt, and handing the flag an empty string would be a silent no-op
+  # that reads like an exemption in the CLI log.
+  local flags=()
+  [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
   [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
   # A tagged subset run is also the merge gate: the touched requirements MUST be covered.
   [ -n "${REQS}" ] && flags+=(--require "${REQS}")

conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml

@@ -0,0 +1,60 @@
+# =============================================================================
+# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
+# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
+# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
+# the empty string while the Kustomization still reports Ready — which emptied Cilium's
+# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
+#
+# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
+# WHEN  each `spec.path` is resolved to the exact file set it renders (following
+#       kustomization.yaml `resources:` transitively, not globbing the repo),
+# THEN  every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
+#       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
+#       operator on an undefined name (the silent-empty class strict mode does NOT catch);
+#       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
+#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
+#       every overlay-patched knob has a base default; and the check is wired into the
+#       merge gate as a blocking, unconditionally-run stage.
+#
+# This is the prerequisite proof for enabling kustomize-controller's
+# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
+# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
+# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
+# not render, does not prove the feature gate is enabled on the live kustomize-controller,
+# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
+# only: what git says a Kustomization renders is fully determined, with no variable
+# silently emptied on the way in. The live-state half is L2, untested today.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: flux-postbuild-substitution
+  labels:
+    conformance.home.k3s/R-ENV-02: "true"
+    conformance.home.k3s/R-GITOPS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every variable a substituted Flux path renders is defined in cluster-vars or
+      escaped, so post-build substitution cannot silently empty a value and enabling
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
+spec:
+  steps:
+    - name: audit-every-substituted-flux-path
+      description: >-
+        Resolve each postBuild.substituteFrom Kustomization to the files it renders and
+        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/flux-postbuild-substitution/check.sh

@@ -0,0 +1,369 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — flux-postbuild-substitution
+# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
+#                   — spec docs/02-environments-and-provisioning.md)
+#         R-GITOPS-01 ("All desired state in git, reconciled by Flux."
+#                   — spec docs/05-gitops-and-delivery.md)
+#
+# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
+# --------------------------------------------------------
+# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
+# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
+# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
+# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
+# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
+# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
+# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
+# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
+# the repo asserted that. This check does.
+#
+# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
+# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
+#   ${DEFINED}        -> substituted with the cluster-vars value.
+#   ${UNDEFINED}      -> EMPTY STRING, silently. `--strict` / the cluster-wide
+#                        StrictPostBuildSubstitutions feature gate turns this into a
+#                        loud Kustomization failure ("variable not set (strict mode)").
+#   ${UNDEFINED:-d}   -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
+#                        NOT catch this (it exempts default-operator forms by design),
+#                        so a shell guard written `[ -z "${X:-}" ]` inside a rendered
+#                        manifest is pre-resolved to `[ -z "" ]` and its else-branch
+#                        becomes unreachable in the container. Silent, and permanent.
+#   $${VAR}, $${V:-d}, $${ARR[0]}  -> emitted as the LITERAL ${...} for a downstream
+#                        shell/cloud-init consumer. Escaping is total: it neutralises
+#                        even the indexed form.
+#   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
+#   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
+#                        variable name" / "missing closing brace") which fails the
+#                        WHOLE Kustomization — every resource in the path, not just
+#                        the offending one.
+# The escape contract this enforces is the one already documented in
+# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
+#
+# GIVEN the repository as authored,
+# WHEN  every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
+#       is resolved to the exact file set its `spec.path` renders (by following
+#       kustomization.yaml `resources:` transitively — a path, not a glob),
+# THEN  (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
+#       (2) no UNESCAPED expansion carries a default/length/substitution operator on an
+#           undefined name (the silent-empty class strict mode does not catch);
+#       (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
+#           that fails the entire Kustomization, so it is fatal wherever it appears;
+#       (4) every Kustomization that renders a cluster-vars token actually declares
+#           `substituteFrom` a ConfigMap named cluster-vars (else the literal token
+#           ships into the cluster);
+#       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
+#           the "add the knob with a base default BEFORE referencing it" rule that the
+#           incident's interim discipline depends on;
+#   AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
+#           ci/gate.sh sets the merge block when the l0 stage fails.
+# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
+# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
+# comment survives into the built apps/wave-1 output), so a braced expression in a
+# comment is a live hazard, not prose.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does NOT render the manifests. It scans the SOURCE files a path includes,
+#     which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
+#     drops some comments). A finding in a comment may not reach envsubst TODAY; it is
+#     still reported, because whether a given comment survives the build is a kustomize
+#     implementation detail that changes with the pinned version, and the failure it
+#     causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
+#     flux envsubst --strict` for every substituted path — needs kustomize + flux,
+#     which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
+#     l0), so it cannot run here without permanently SKIPPING. It was run by hand and
+#     recorded in DECISIONS-LOG; this check is the reproducible static half.
+#   - It does NOT prove the cluster's live state matches git, nor that the
+#     StrictPostBuildSubstitutions feature gate is actually enabled on
+#     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
+#     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
+#     is fully determined, with no variable silently emptied on the way in.
+#   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
+#     that an overlay's LAN block is self-consistent). Presence, not correctness.
+#   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
+#     several are deliberately reference-only and say so in cluster-vars.yaml
+#     (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
+#     failing on them would be a false block on documented intent.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green) if the thing under test is not present at all.
+if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
+  echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of Flux postBuild substitution safety. Pure standard library.
+
+No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
+documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
+`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
+definition of "what a Flux path actually renders" in the harness.
+"""
+from __future__ import annotations
+
+import os
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st                     # noqa: E402 — reuse the walk, don't fork it
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# ------------------------------------------------------------------ cluster-vars
+# Parse contract: `data:` block of a single ConfigMap, one `  KEY: "value"` per line
+# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
+# continuation-comment lines are indented further and never match).
+_DATA_KEY_RE = re.compile(r'^  ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
+
+
+def cluster_vars_keys(path: Path) -> set[str]:
+    keys: set[str] = set()
+    in_data = False
+    for raw in path.read_text().splitlines():
+        if re.match(r"^data:\s*$", raw):
+            in_data = True
+            continue
+        if not in_data:
+            continue
+        if st._TOP_KEY_RE.match(raw):   # a new top-level key ends the data block
+            break
+        m = _DATA_KEY_RE.match(raw)
+        if m:
+            keys.add(m.group(1))
+    return keys
+
+
+VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+KEYS = cluster_vars_keys(VARS_FILE)
+check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
+
+# ------------------------------------------- Flux Kustomizations with substituteFrom
+# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
+# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
+# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
+# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
+_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
+_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
+_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
+
+
+def substitute_sources(doc: str) -> list[tuple[str, str]]:
+    """[(kind, name)] declared under postBuild.substituteFrom in this doc."""
+    m = _SUBFROM_RE.search(doc)
+    if not m:
+        return []
+    indent = len(m.group(1))
+    out: list[tuple[str, str]] = []
+    kind = ""
+    for raw in doc[m.end():].splitlines():
+        if raw.strip() == "" or raw.lstrip().startswith("#"):
+            continue
+        if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
+            break                                    # dedented out of the list
+        km = _SUBITEM_KIND_RE.match(raw)
+        if km:
+            kind = km.group(1)
+            continue
+        nm = _SUBITEM_NAME_RE.match(raw)
+        if nm and kind:
+            out.append((kind, nm.group(1)))
+            kind = ""
+    return out
+
+
+kustomizations: list[dict] = []
+for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
+    text = yfile.read_text()
+    if st.FLUX_API not in text:
+        continue
+    for doc in st._DOC_SPLIT_RE.split(text):
+        if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
+            continue
+        pm = st._PATH_RE.search(doc)
+        if not pm:
+            continue
+        nm = st._NAME_RE.search(doc)
+        srcs = substitute_sources(doc)
+        if not srcs:
+            continue                                  # no substitution => nothing to audit
+        kustomizations.append({
+            "file": str(yfile.relative_to(ROOT)),
+            "name": nm.group(1) if nm else "?",
+            "path": pm.group(1),
+            "sources": srcs,
+        })
+
+check(len(kustomizations) > 0,
+      "no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
+      "the parser or the repo layout changed; this check would silently audit nothing")
+
+# ------------------------------------------------------------------ token scanning
+# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
+# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
+# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
+TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
+# these, so an undefined name yields the default/empty AT RECONCILE TIME.
+OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
+LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
+
+
+def classify(dollars: int, inner: str) -> tuple[str, str] | None:
+    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
+    if dollars >= 2:
+        return None                                   # escaped: emitted as a literal
+    if NAME_RE.match(inner):
+        if inner in KEYS:
+            return None                               # defined knob — the happy path
+        return ("UNDEFINED",
+                f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
+                f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
+                f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
+    m = OP_RE.match(inner) or LEN_RE.match(inner)
+    if m:
+        base = m.group(1)
+        if base in KEYS:
+            return None                               # defined knob with an operator
+        return ("SILENT-EMPTY",
+                f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
+                f"cluster-var, so the rendered manifest carries the default, not the "
+                f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
+                f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
+    return ("FATAL-PARSE",
+            "envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
+            "in the path). Escape it as $${...} or rewrite the prose without braces")
+
+
+def rendered_files(path_spec: str) -> tuple[list[str], bool]:
+    """Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
+    rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
+    if not (ROOT / rel).is_dir():
+        return [], False
+    applied, _ = st.reconciled_files(ROOT, rel)
+    return sorted(applied), True
+
+
+findings: list[str] = []
+referenced: dict[str, set[str]] = {}
+inventory: list[str] = []
+
+for k in kustomizations:
+    files, exists = rendered_files(k["path"])
+    check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
+                  f"which is not a directory in this repo")
+    has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
+    n_def = n_esc = n_bad = 0
+    for rel in files:
+        f = ROOT / rel
+        try:
+            text = f.read_text()
+        except (OSError, UnicodeDecodeError):
+            continue
+        for lineno, line in enumerate(text.splitlines(), 1):
+            for m in TOKEN_RE.finditer(line):
+                dollars, inner = len(m.group(1)), m.group(2)
+                verdict = classify(dollars, inner)
+                if verdict is None:
+                    if dollars >= 2:
+                        n_esc += 1
+                    else:
+                        n_def += 1
+                        referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
+                                              set()).add(k["name"])
+                        # (4) a DEFINED knob is only substituted if this Kustomization
+                        #     actually pulls cluster-vars in; otherwise the literal ships.
+                        check(has_cluster_vars,
+                              f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
+                              f"Kustomization '{k['name']}' ({k['file']}) does not "
+                              f"substituteFrom ConfigMap/cluster-vars — the literal token "
+                              f"would ship into the cluster")
+                    continue
+                n_bad += 1
+                tag, fix = verdict
+                findings.append(f"  [{tag}] {rel}:{lineno}  {m.group(0)}\n"
+                                f"        rendered by Flux Kustomization '{k['name']}' "
+                                f"({k['file']}, path {k['path']})\n"
+                                f"        {fix}\n"
+                                f"        | {line.strip()[:120]}")
+    inventory.append(f"  {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
+                     f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
+
+check(not findings,
+      f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
+
+# --------------------------------------------- (5) overlay knobs exist in the base
+# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
+# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
+# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
+# manifest references it — that ordering is what closes the define-then-use race.
+_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
+for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+    for key in set(_OP_PATH_RE.findall(overlay.read_text())):
+        check(key in KEYS,
+              f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
+              f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
+              f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
+
+# ------------------------------------------------------- (6) the check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+check("flux-postbuild-substitution/check.sh" in L0,
+      "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
+      "conformance selection can filter this check out of a gate run, so l0.sh must invoke "
+      "it directly and unconditionally or the gate is not actually enforcing it")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ------------------------------------------------------------- advisory: dead knobs
+for key in sorted(KEYS - set(referenced)):
+    ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
+                      f"(dead knob, or documentation-only — advisory, not a failure)")
+
+# ------------------------------------------------------------------------- report
+print("Flux postBuild substitution inventory (per Kustomization):")
+for line in inventory:
+    print(line)
+print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+if findings:
+    print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
+    for f in findings:
+        print(f, file=sys.stderr)
+
+print()
+if not FAILURES:
+    print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
+          f"(R-ENV-02, R-GITOPS-01)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

infra/PROD-PROVISIONING.md

@@ -311,6 +311,32 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
       own comment flags this as owed "when hardware is built."
 - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
       byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
+- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
+      the `dependsOn` fix. Only the first is now closed:
+      - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
+            *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
+            `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
+            forever on a dependency that does not exist and nothing downstream would reconcile.
+      - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
+            as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
+            exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
+            (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
+      - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
+            2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
+            will be decommissioned once prod is stable. Recorded consequence: once prod is live,
+            **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
+            The URL is patched in but **cannot work on a cold prod cluster**, for two independent
+            reasons found 2026-09-01:
+            1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
+               decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
+               loses its source of truth. **This is the decommission plan's first task, not its last.**
+            2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
+               `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
+               prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
+               `gitea.home.keithjackson.dev` to itself and never reach a repository.
+            Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
+            cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
+            Phase-6 cutover.
 - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
       the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
 

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -42,5 +42,22 @@ cgroup:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
 
+# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
+# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
+# 3-node cluster lets one bad render take down two of three agents simultaneously —
+# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config`
+# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium
+# agent can translate that ClusterIP, and the agent in question was the one being
+# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked.
+# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one
+# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP
+# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1
+# the DaemonSet will not proceed past an unready pod, bounding damage to a single node.
+# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade.
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
+
 operator:
   replicas: {{ cilium_operator_replicas }}

platform/cilium/helmrelease.yaml

@@ -89,6 +89,27 @@ spec:
       autoMount:
         enabled: false
       hostRoot: /run/cilium/cgroupv2
+    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
+    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
+    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
+    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
+    # Helm render take down two of three agents simultaneously. That is exactly what happened
+    # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
+    # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
+    # agent itself can translate — and the agent translating it was the one being replaced.
+    # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
+    # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
+    # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
+    # will not proceed past an unready pod, so a bad render costs one node, not the cluster.
+    # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
+    # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
+    # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
+    # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
+    # grows enough that serialised CNI upgrades become an operational problem.
+    updateStrategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 1
     # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
     # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
     # (Flux/prod may differ). Only the [parity] values above must match the bootstrap.

platform/cilium/values.yaml

@@ -42,6 +42,16 @@ cgroup:
   autoMount:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
+# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
+# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
+# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
+# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade
+# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops
+# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes).
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
 # NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease:
 #   operator.replicas
 #

platform/kustomization.yaml

@@ -12,7 +12,15 @@ kind: Kustomization
 resources:
   # ---- Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress ----
   - kyverno            # admission enforcer FIRST — policies present before other components reconcile
-  - cilium             # CNI (adopts the running Ansible-installed release; parity verified)
+  #      cilium (the CNI) is DELIBERATELY NOT here: it now reconciles in its OWN Flux Kustomization
+  #      (clusters/base/cilium.yaml, dependsOn flux-system, wait:true), and `platform` dependsOn IT.
+  #      Not the headlamp/oauth2-proxy rationale (a leaf that must not gate) but its mirror image:
+  #      the CNI is the one component that SHOULD gate everything, and inside this shared tier its
+  #      health was indistinguishable from eleven unrelated components, the only emergency valve was
+  #      freezing the whole tier, and the tier's 10m/5m interval/timeout are wrong for a 3-node
+  #      DaemonSet rollout. Gating is preserved via the explicit `platform.dependsOn: [cilium]` edge.
+  #      NOTE: platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system in
+  #      this tree — that is why the move required a prune-disabled transition. See DECISIONS-LOG.
   - longhorn           # replicated block storage / CSI
   - external-snapshotter  # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
   - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`

platform/longhorn/helmrelease.yaml

@@ -2,7 +2,7 @@
 # Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003).
 # Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication).
 #
-# Dev-appropriate values. Environment differences enter ONLY via ${...} substitution from
+# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from
 # cluster-vars (R-ENV-02); everything else is identical base config for dev and prod.
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease

platform/longhorn/storageclasses.yaml

@@ -20,7 +20,7 @@ parameters:
   # (replicaSoftAntiAffinity=false) guarantees these land on distinct nodes, so any single node
   # loss keeps the volume available.
   # HARDCODED, not ${LONGHORN_DEFAULT_REPLICA_COUNT}: StorageClass.parameters is map[string]string,
-  # but kustomize strips the quotes around a bare "${VAR}" placeholder, so Flux postBuild substitution
+  # but kustomize strips the quotes around a bare "$${VAR}" placeholder, so Flux postBuild substitution
   # yields an unquoted numeric (3) that server-side apply rejects (int, not string). A quoted numeric
   # literal like "3"/"30" IS preserved by kustomize. The replica knob still drives the GLOBAL default
   # via defaultSettings.defaultReplicaCount in helmrelease.yaml (a Helm int value, unaffected).