feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition)

Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.

New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.

*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.

TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
   moving resources between Kustomizations. Chosen over the per-resource
   `kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
   and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
   GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
   say name=platform. The new cilium Kustomization relabels all three to name=cilium on
   apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
   reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
   precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".

NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.

Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.

MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
  cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
  cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.

Serves: R-PLAT-01, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

DECISIONS-LOG.md

@@ -1496,3 +1496,265 @@ thing that ever closes a requirement.
   chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
   the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
+
+---
+
+## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
+
+Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
+`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
+and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
+Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
+prod bootstrap than it was here.
+
+### What happened
+
+The HelmRelease config digest tells the whole story:
+
+```
+v8   05:49:39  digest e83bf695…      <- before
+v9   05:50:11  digest eebd6c88…      <- the bad render
+v10  05:55:23  digest e83bf695…      <- identical to v8 again
+```
+
+The config changed and then changed **back to byte-identical**. Nothing in git differed
+semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
+`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
+`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
+state is identical. The damage happened in a transient intermediate state.
+
+### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
+
+- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
+- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
+  Kustomization, via `postBuild.substituteFrom: cluster-vars`.
+- There is **no dependency ordering between defining a variable and using it.**
+
+So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
+which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
+undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
+upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
+which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
+`config` init container could not reach the API server:
+
+```
+level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
+  error="dial tcp 10.43.0.1:443: i/o timeout"
+level=fatal msg="Build config failed"
+```
+
+Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
+correct value, digest returned to `e83bf695…`, and all three agents came up.
+
+`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
+the ConfigMap to *exist*, not that every referenced key is defined.
+
+### Why this matters much more for prod
+
+On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
+race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
+there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
+never come up at all — with no healthy peer to recover from. This is a plausible
+bring-up-blocker for `clusters/prod`.
+
+### The fix — make substitution FAIL CLOSED
+
+`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
+substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
+the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
+("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
+this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
+failure*.
+
+**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
+that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
+embedded scripts. The safe sequence, before flipping it:
+1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
+   `cluster-vars` or correctly escaped as `$${VAR}`.
+2. Then enable the gate, and let CI + a dev reconcile prove it.
+
+Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
+reference it in a later one**, so the definition is always reconciled before its first use.
+
+### Operational note worth keeping
+
+A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
+`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
+set, and the agent that translates that ClusterIP is the one being replaced. Keep
+`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.
+
+---
+
+## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.**
+
+Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit),
+prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected
+here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything
+else is cited inference.
+
+### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere)
+
+| form | behaviour |
+|---|---|
+| `${DEFINED}` | substituted |
+| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) |
+| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** |
+| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. |
+| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. |
+| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** |
+| `$VAR` / `$(cmd)` | untouched, no escaping needed |
+| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** |
+
+Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict
+mode by design ("they intentionally handle the unset case"). **That is why strict mode would never
+have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required.
+
+**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal
+"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place.
+
+**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them,
+so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst,
+and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on
+stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke
+every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.)
+
+### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT
+
+It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly
+and keep last-good". **The actual ordering fix is one line**, verified against
+`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is
+**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the
+dependent cannot reconcile revision N until the dependency has APPLIED revision N.
+
+`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains
+transitively through it. So:
+
+```yaml
+# clusters/base/platform.yaml
+spec:
+  dependsOn:
+    - name: flux-system     # cluster-vars must be applied at THIS revision first
+```
+
+**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under
+`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with
+doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`.
+
+Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** —
+`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization
+applies its own resources, so it would always read the PREVIOUS value. That converts an occasional
+race into a permanent one-reconcile lag.
+
+### CORRECTION 2 (an agent's) — the safety machinery DID fire
+
+A research pass concluded that Helm considered the bad release healthy (because
+`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and
+therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm
+release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which
+is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was
+blast radius (2 of 3 agents down at once), not a missed detection.
+
+### Verified, and worth knowing
+
+- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current.
+- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips
+  this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade
+  regardless of today's decision.
+- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`.
+  Helm was never going to reject the bad render.
+- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the
+  environment exports the API ClusterIP and there is no kube-proxy to provision it.
+- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade
+  check must live outside Helm.
+- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's
+  silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces
+  (`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`).
+- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the
+  Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction
+  would have actively re-asserted the broken value.
+
+### RANKED PLAN
+
+| P | Action | State |
+|---|---|---|
+| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) |
+| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE |
+| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions |
+| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. |
+| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node |
+| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve |
+| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default |
+| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process |
+| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) |
+| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — |
+
+### Follow-ups surfaced, not yet actioned
+- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot.
+- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative
+  `${...}` that can silently drift from the real HelmRelease.
+- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
+  out while a legitimate Longhorn upgrade is still running.
+
+---
+
+## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
+
+All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
+
+### Landed and verified on the live cluster
+
+| Change | Verified |
+|---|---|
+| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
+| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
+| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
+| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
+| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
+
+### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
+
+Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
+`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
+
+```sh
+[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"   # LAST line of the hook
+```
+
+On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
+the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
+
+**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
+long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
+execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
+**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
+dead code is not a safe no-op — it is the first execution of never-tested code, and should be
+treated with the caution of new code, not of a one-character escape.
+
+Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
+on both paths, but only by accident of which statement happened to be last — made explicit there too
+so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
+Kustomizations Ready, both OIDC providers still registered.
+
+### STILL TO DO — strict substitution is committed but NOT ACTIVE
+
+`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
+`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
+ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
+controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
+standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
+~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
+
+**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
+With the gate on but the ordering edge absent, the same race would turn a transient empty render
+into a transient hard Kustomization failure.
+
+### Prod consequences recorded this session
+
+- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
+  soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
+  first" safety net that the last two days have leaned on heavily, twice.
+- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
+  currently points at Gitea running on the dev cluster.
+- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
+  with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
+  Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.

apps/vms/unifi/30-virtualmachine.yaml

@@ -92,10 +92,13 @@ spec:
             # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
             # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
             # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
+            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write
+            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace")
+            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply.
+            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
+            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
+            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
+            # wording in clusters/base/apps-vms.yaml overstated it.
             userData: |-
               #cloud-config
               hostname: unifi

apps/vms/unifi/50-service-l4.yaml

@@ -27,7 +27,7 @@
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
 # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
-# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
+# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
 # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
 #
 # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an

apps/wave-0/gitea.yaml

@@ -127,8 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
-                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
+                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

apps/wave-1/nextcloud.yaml

@@ -125,8 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
-                \ positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
+                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

bootstrap/flux/README.md

@@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
 | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
 
+## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
+
+Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
+and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
+
+```
+$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
+ "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
+# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
+```
+
+No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
+`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
+documents for hand-installed clusters ([Customize Flux
+manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
+
+```
+flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
+components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
+kubectl kustomize <tmp> | kubectl apply -f -
+```
+
+Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
+(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
+more feature gates) is another entry in `components/kustomization.yaml`.
+
+### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
+
+> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
+> controller Deployment is outside Flux's inventory. Someone must run step 1.
+
+Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
+
+```bash
+# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
+d=$(mktemp -d)
+flux install --export --namespace=flux-system \
+  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
+cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
+kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
+kubectl -n flux-system rollout status deploy/kustomize-controller
+```
+
+**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
+reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
+is lost. Confirm afterwards:
+
+```bash
+kubectl -n flux-system get deploy kustomize-controller \
+  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
+flux get kustomizations -A                                  # all 11 must return to Ready
+```
+
+**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
+simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):

bootstrap/flux/bootstrap.sh

@@ -69,16 +69,59 @@ for bin in flux kubectl; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
-# --- 1. install Flux controllers ---------------------------------------------
+# --- 1. install Flux controllers (rendered through ./components) --------------
 # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
 # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
 # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
-if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
+#
-  echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
+# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
-  flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
+# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
+# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
+# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
+#
+# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
+# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
+# documents for hand-installed clusters ("Customize Flux manifests"):
+#
+#     flux install --export  >  gotk-components.yaml     # exact manifests `flux install` would apply
+#     kustomize overlay      =  ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
+#     kubectl apply -k
+#
+# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
+# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
+# a temp dir — the repo tree is never written to.
+#
+# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
+# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
+echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+flux_install_args=(--namespace="${FLUX_NAMESPACE}")
+[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
+
+if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
+  echo "    WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
+  echo "             StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
+  flux install "${flux_install_args[@]}"
 else
-  echo "==> [1/3] flux install (default controllers only)"
+  components_dir="${script_dir}/components"
-  flux install --namespace="${FLUX_NAMESPACE}"
+  [ -f "${components_dir}/kustomization.yaml" ] || {
+    echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
+    echo "       (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
+    exit 1
+  }
+  render_dir="$(mktemp -d)"
+  trap 'rm -rf "${render_dir}"' EXIT
+  echo "    rendering flux install --export through ${components_dir}"
+  flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
+  cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
+  # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
+  # version bump) — better a hard stop than silently installing without the feature gate.
+  kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
+  grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
+    echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
+    echo "       Check ./components/kustomization.yaml against this flux CLI version." >&2
+    exit 1
+  }
+  kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
 # --- 2. verify the out-of-band age key Secret exists -------------------------

bootstrap/flux/components/kustomization.yaml

@@ -0,0 +1,64 @@
+---
+# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
+#
+# WHY THIS DIRECTORY EXISTS
+# =========================
+# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
+# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
+# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
+#
+#     kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+#     -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
+#
+# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
+# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
+#
+# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
+# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
+# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
+# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
+# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
+# repo, so the manifests can never drift from the pinned FLUX_VERSION
+# (infra/devcontainer/versions.env) the way a vendored copy would.
+#
+# Every future controller customization (vertical scaling, tolerations, extra feature gates,
+# --concurrent) belongs HERE, as another entry under `patches:`.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
+  - gotk-components.yaml
+patches:
+  # -------------------------------------------------------------------------------------------
+  # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
+  #
+  # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
+  # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
+  # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
+  # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
+  #
+  # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
+  # `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
+  # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
+  # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
+  #
+  # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
+  # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
+  #
+  # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
+  # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
+  #
+  # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
+  # list, so a strategic-merge patch would REPLACE every arg `flux install` generates
+  # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
+  # and silently drop any the CLI adds in a future version.
+  - target:
+      group: apps
+      version: v1
+      kind: Deployment
+      name: kustomize-controller
+      namespace: flux-system
+    patch: |-
+      - op: add
+        path: /spec/template/spec/containers/0/args/-
+        value: --feature-gates=StrictPostBuildSubstitutions=true

catalog/README.md

@@ -28,6 +28,16 @@ inherit every guarantee automatically.
   `auth.groups`; `native`/`none` are documented exceptions only.
 - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
   schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
+- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
+  `lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
+  `apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
+  envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
+  *shell* is replaced with the empty string before the container ever starts (this silently killed
+  gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
+  need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
+  written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
+  `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
+  cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
 
 ## Validate before commit — single command, no cluster (R-TEST-02)
 ```bash

catalog/gitea.yaml

@@ -109,7 +109,19 @@ lifecycle:
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
           }
-          if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
+          # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
+          # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
+          # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
+          # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
+          # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
+          # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
+          # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
+          # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
           else
             # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
@@ -150,6 +162,12 @@ lifecycle:
               gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
             fi
           fi
+          # A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
+          # it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
+          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
+          # the moment its OIDC guard was fixed and the block actually ran for the first time.
+          exit 0
 exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/nextcloud.yaml

@@ -111,7 +111,19 @@ lifecycle:
           # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
           OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
             || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
-          if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
+          # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
+          # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
+          # installing/enabling user_oidc nor registering the provider. SSO kept working only because
+          # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
+          # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
+          # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
+          # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
+          # `--clientsecret-env` and the bare uses below always worked and only the guard broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
             exit 0
           fi
@@ -191,7 +203,17 @@ lifecycle:
             [ "$missing" = 0 ] && break
             p=$((p + 1)); sleep 2
           done
-          [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          if [ "$p" -ge 15 ]; then
+            nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          fi
+          # A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
+          # the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
+          # SUCCESS path the test returned 1 and became the script's exit status. That was latent for
+          # as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
+          # exposed it immediately. Every failure path here logs loudly and writes a marker file and
+          # deliberately does NOT crash the pod — the break-glass local admin must stay usable.
+          exit 0
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
 storage:

catalog/render.py

@@ -29,6 +29,7 @@ conformance harness stays stdlib-only and only shells out to this script + diffs
 from __future__ import annotations
 
 import argparse
+import re
 import sys
 from pathlib import Path
 
@@ -936,6 +937,124 @@ def r_virtualmachine(s: dict) -> dict:
     }
 
 
+# =============================================================================
+# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
+# Flux's envsubst would silently eat before the workload ever runs.
+# =============================================================================
+# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
+# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
+# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
+# document — including any shell script carried verbatim in a `lifecycle` hook or an env
+# value — at reconcile time, long before a container sees it. The escape contract is stated
+# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
+#
+#   `${VAR}`   -> SUBSTITUTED (empty when unset)   => a shell brace must be written `$${VAR}`,
+#                                                     which envsubst emits back as `${VAR}`.
+#   `$VAR`     -> left alone                       => bare shell vars need NO escaping.
+#   `$(cmd)`   -> left alone                       => command substitution needs NO escaping.
+#   `${A[0]}`  -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
+#
+# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
+# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
+# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
+# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
+# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
+# exactly the form a defensive shell script uses. So the renderer checks it itself.
+CLUSTER_VARS_PATH = (
+    Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
+)
+
+# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
+# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
+# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
+# (`${A[0]}`) are caught too rather than skipped.
+_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
+
+_CLUSTER_VARS_CACHE: dict = {}
+
+
+def cluster_vars_keys(path: Path | None = None) -> set:
+    """The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
+
+    Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
+    renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
+    FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
+    """
+    key = str(path or CLUSTER_VARS_PATH)
+    if key not in _CLUSTER_VARS_CACHE:
+        try:
+            doc = yaml.safe_load(Path(key).read_text())
+            data = doc.get("data") if isinstance(doc, dict) else None
+            _CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
+        except Exception:  # pragma: no cover - absent/malformed => fail closed
+            _CLUSTER_VARS_CACHE[key] = set()
+    return _CLUSTER_VARS_CACHE[key]
+
+
+def substitution_violations(text: str, known: set | None = None) -> list:
+    """Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
+
+    Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
+    when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
+    a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
+    envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
+    """
+    keys = cluster_vars_keys() if known is None else known
+    out = []
+    for lineno, line in enumerate(text.splitlines(), start=1):
+        for m in _BRACED_EXPANSION_RE.finditer(line):
+            if len(m.group(1)) % 2 == 0:
+                continue  # even run of `$` => escaped; envsubst emits it back literally
+            expr = m.group(2)
+            if _PLAIN_IDENT_RE.match(expr):
+                if expr in keys:
+                    continue  # a declared cluster-vars knob, intentionally substituted
+                reason = (
+                    f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
+                    "it with the EMPTY STRING"
+                )
+            elif "[" in expr:
+                reason = (
+                    "indexed expansion — envsubst hard-fails with 'missing closing brace' "
+                    "and the whole Kustomization stops reconciling"
+                )
+            else:
+                reason = (
+                    "shell expansion (default-operator/parameter form) — envsubst eats it "
+                    "at reconcile time and STRICT mode exempts this form, so nothing else "
+                    "will catch it"
+                )
+            out.append((lineno, m.group(0), reason))
+    return out
+
+
+def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
+    """Refuse to emit a manifest carrying an unescaped braced expansion (see above).
+
+    Raises ValueError listing every offending token. The fix is almost always to escape the
+    shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
+    value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
+    and write it as a bare `${KNOB}`.
+    """
+    bad = substitution_violations(text)
+    if not bad:
+        return
+    lines = [
+        f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
+        "postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
+    ]
+    for lineno, token, reason in bad:
+        lines.append(f"  line {lineno}: {token}  -> {reason}")
+    lines.append(
+        "  FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
+        "`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
+        "must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
+        "Contract: clusters/base/apps-vms.yaml."
+    )
+    raise ValueError("\n".join(lines))
+
+
 # =============================================================================
 # Top-level render + emit
 # =============================================================================
@@ -1026,8 +1145,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
     return docs
 
 
-def emit(docs: list) -> str:
+def emit(docs: list, origin: str = "<render>") -> str:
-    """Deterministic multi-document YAML text (golden-diffable)."""
+    """Deterministic multi-document YAML text (golden-diffable).
+
+    Every emitted document is checked against the substitution contract before it is handed
+    back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
+    envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
+    ValueError rather than emitting such a manifest.
+    """
     parts = []
     for comment, resource in docs:
         body = yaml.dump(
@@ -1037,7 +1162,9 @@ def emit(docs: list) -> str:
             width=4096,
         )
         parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
-    return "".join(parts)
+    text = "".join(parts)
+    assert_substitution_safe(text, origin)
+    return text
 
 
 def load_sd(path: Path) -> dict:
@@ -1076,7 +1203,9 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
     src_dir = catalog_dir if catalog_dir is not None else path.parent
     rev = build_reverse_map(_load_catalog(src_dir))
     inbound = rev.get(sd["name"], [])
-    return emit(render(sd, image_automation=image_automation, inbound=inbound))
+    return emit(
+        render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
+    )
 
 
 # =============================================================================
@@ -1729,6 +1858,90 @@ def _selftest() -> int:
     check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
           "wave kustomization must list exactly the wave's resources")
 
+    # 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
+    #    shell script carries an unescaped `${...}`. Every rendered manifest lands under a
+    #    Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
+    #    the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
+    #    -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
+    known = {"BASE_DOMAIN", "GATEWAY_VIP"}
+    # allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
+    check(
+        substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
+        "a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
+    )
+    # allowed: escaped shell braces, and the forms envsubst leaves alone.
+    check(
+        substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
+        "an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
+    )
+    check(
+        substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
+        "bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
+    )
+    # `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
+    check(
+        len(substitution_violations("$$${X:-}", known)) == 1,
+        "an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
+    )
+    # flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
+    guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
+    check(
+        len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
+        "an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
+    )
+    # flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
+    check(
+        len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
+        "an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
+    )
+    idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
+    check(
+        len(idx) == 1 and "missing closing brace" in idx[0][2],
+        "an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
+    )
+    # the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
+    # bug RAISES rather than writing a manifest whose guard is dead on arrival.
+    bad_sd = {
+        "name": "subbad",
+        "source": {"image": "a:1"},
+        "lifecycle": {
+            "postStart": {
+                "exec": {
+                    "command": [
+                        "/bin/sh",
+                        "-c",
+                        'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                    ]
+                }
+            }
+        },
+    }
+    try:
+        emit(render(bad_sd), origin="<selftest>")
+        check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
+    except ValueError as exc:
+        check(
+            "SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
+            "the refusal must name the offending token AND state the `$${...}` fix",
+        )
+    # the ESCAPED form of the same hook renders normally (the fix must be emittable).
+    good_sd = dict(bad_sd, name="subgood")
+    good_sd["lifecycle"] = {
+        "postStart": {
+            "exec": {
+                "command": [
+                    "/bin/sh",
+                    "-c",
+                    'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                ]
+            }
+        }
+    }
+    check(
+        "$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
+        "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
+    )
+
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

ci/l0.sh

@@ -100,6 +100,43 @@ else
   fi
 fi
 
+# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
+# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
+# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
+# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
+# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
+# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
+# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
+# a legitimate manifest, and that no expansion silently empties today.
+#
+# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
+# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
+# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
+# attributes no requirement, so the requirement-scoped selection would filter this check
+# out of exactly the change that introduces the bug. The invariant is whole-repo (same
+# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
+# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
+# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
+SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
+log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
+if [ ! -f "$SUBST_CHECK" ]; then
+  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
+  summary "- subst-safety: **SKIPPED** (check missing)"
+  if [ "$rc" -eq 0 ]; then rc=3; fi
+else
+  subst_rc=0
+  bash "$SUBST_CHECK" || subst_rc=$?
+  case "$subst_rc" in
+    0) ok "Flux postBuild substitution safety passed"
+       summary "- subst-safety: **passed**" ;;
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
+    *) err "Flux postBuild substitution safety FAILED"
+       summary "- subst-safety: **FAILED**"; rc=1 ;;
+  esac
+fi
+
 case "$rc" in
   0) ok "L0 stage complete" ;;
   3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/kustomization.yaml

@@ -5,6 +5,13 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).

clusters/base/platform.yaml

@@ -11,8 +11,62 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # ======================= TEMPORARY — prune:false TRANSITION STATE =======================
+  # RESTORE TO `prune: true` IN THE VERY NEXT MERGE. This is step 1 of the documented Flux
+  # procedure for MOVING RESOURCES BETWEEN KUSTOMIZATIONS (fluxcd.io/flux/faq: "set spec.prune to
+  # false, sync, move the manifests, then re-enable pruning"), used here to move ./platform/cilium
+  # out of this tier and into its own `cilium` Kustomization (clusters/base/cilium.yaml).
+  #
+  # WHY IT IS NOT OPTIONAL: `platform/cilium/namespace.yaml` is the ONLY declaration of
+  # Namespace/kube-system anywhere under ./platform, and the live object carries
+  # `kustomize.toolkit.fluxcd.io/name=platform`. The moment ./platform/cilium leaves this
+  # Kustomization's rendered path, Namespace/kube-system + the Cilium HelmRepository/HelmRelease
+  # become STALE in this Kustomization's inventory. With prune:true that is not "uninstall the
+  # CNI" — it is `kubectl delete namespace kube-system`, which takes the whole cluster.
+  #
+  # A SECOND, INDEPENDENT PROTECTION IS ALSO IN PLACE (belt and braces, deliberately):
+  # kustomize-controller's pruner passes `Inclusions: GetOwnerLabels(platform, flux-system)`, so it
+  # only deletes objects whose LIVE labels still say they belong to `platform`. The new `cilium`
+  # Kustomization relabels all three objects to `name=cilium` when it applies them, and the
+  # `dependsOn: [cilium]` edge below is REVISION-gated — `platform` cannot reconcile revision N
+  # until `cilium` has APPLIED revision N. So the relabel provably precedes the prune. Neither
+  # mechanism is trusted alone for an object of this blast radius.
+  prune: false         # <-- TEMPORARY; see block above. Normal value: true (R-GITOPS-03)
   wait: true           # block `apps` until platform is healthy
+  # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
+  #
+  # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
+  # `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
+  # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
+  # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
+  # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
+  #
+  # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
+  # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
+  # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
+  # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
+  # writes a newly-added `cluster-vars` knob — so this one edge closes the race.
+  #
+  # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
+  # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
+  # velero) chains transitively through it. One edge therefore protects the whole graph.
+  #
+  # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
+  # ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
+  # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
+  # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
+  # entrypoint should stop the line, not let consumers render against a half-applied variable set.
+  # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
+  # health-gates its own children (see clusters/dev/sync.yaml).
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/prod/kustomization.yaml

@@ -5,11 +5,144 @@
 # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
 # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
 # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
+#
+# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
+# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
+# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
+# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
+# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
+# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
+# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
+# done here: prod is design-only and its root object is the prod bring-up owner's call.)
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ../base
+  - sync.yaml          # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
 patches:
+  # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
+  # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
+  # The `sops-age` Secret is created OUT OF BAND on the prod cluster
+  # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
+  # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: platform
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-0
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-1
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-2
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: oauth2-proxy
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: headlamp
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: crafty
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: velero
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
+  #
+  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  #
+  # Two independent reasons, both consequences of the split-horizon design:
+  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
+  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
+  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
+  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
+  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
+  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
+  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  #
+  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
+  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
+  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  #
+  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
+  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
+  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  - patch: |-
+      apiVersion: source.toolkit.fluxcd.io/v1
+      kind: GitRepository
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        ref:
+          branch: main
   - target:
       kind: ConfigMap
       name: cluster-vars

clusters/prod/sync.yaml

@@ -0,0 +1,32 @@
+---
+# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
+#
+# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
+# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
+# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
+# its own sync.yaml -> ./clusters/prod"; this is it.
+#
+# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
+# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
+# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
+# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
+# not exist, and NOTHING downstream would ever reconcile.
+#
+# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
+# Ready on apply success without health-gating its children (platform does its own wait/health
+# gating). Do not "harden" this to true.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./clusters/prod

conformance/run.sh

@@ -57,17 +57,15 @@ FORCE_NO_CLUSTER=0
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
 # exemption is explicit and reviewable, not silent.
-#   R-ENV-02   (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
+#
-#                                    coverage.md). The coarse path->requirement map keys
+# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
-#                                    clusters/* AND ci/* to it and the R-ENV-02 doc token
+# because they were unscaffolded while the coarse path->requirement map keys clusters/*
-#                                    appears in platform/kustomization.yaml's header, so ANY
+# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
-#                                    infra/overlay/CI edit (e.g. enabling a component, bumping
+# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
-#                                    SPEC_PIN) would otherwise false-block on it.
+# substitution mechanism that IS the "authored once, promoted" contract, so they are
-#   R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
+# covered rather than exempt and the completeness gate can hold them to it. That test
-#                                    path-map keys to it, so editing any ci/ file (SPEC_PIN
+# names its own limits in its header: it proves the STATIC property (no manifest renders
-#                                    adoption included) would false-block.
+# an undefined variable), not the live-cluster half of either requirement.
-# These are visible GAPs, not hidden — removing them from this list is the right move the
-# moment WS-TEST adds their verifying tests.
 #
 # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
 # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
@@ -83,7 +81,7 @@ FORCE_NO_CLUSTER=0
 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
 # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
-DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
+DOCUMENTED_GAPS=""
 
 usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
 
@@ -108,7 +106,11 @@ log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
-  local flags=(--allow-gap "${DOCUMENTED_GAPS}")
+  # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
+  # requirement is exempt, and handing the flag an empty string would be a silent no-op
+  # that reads like an exemption in the CLI log.
+  local flags=()
+  [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
   [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
   # A tagged subset run is also the merge gate: the touched requirements MUST be covered.
   [ -n "${REQS}" ] && flags+=(--require "${REQS}")

conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml

@@ -0,0 +1,60 @@
+# =============================================================================
+# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
+# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
+# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
+# the empty string while the Kustomization still reports Ready — which emptied Cilium's
+# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
+#
+# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
+# WHEN  each `spec.path` is resolved to the exact file set it renders (following
+#       kustomization.yaml `resources:` transitively, not globbing the repo),
+# THEN  every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
+#       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
+#       operator on an undefined name (the silent-empty class strict mode does NOT catch);
+#       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
+#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
+#       every overlay-patched knob has a base default; and the check is wired into the
+#       merge gate as a blocking, unconditionally-run stage.
+#
+# This is the prerequisite proof for enabling kustomize-controller's
+# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
+# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
+# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
+# not render, does not prove the feature gate is enabled on the live kustomize-controller,
+# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
+# only: what git says a Kustomization renders is fully determined, with no variable
+# silently emptied on the way in. The live-state half is L2, untested today.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: flux-postbuild-substitution
+  labels:
+    conformance.home.k3s/R-ENV-02: "true"
+    conformance.home.k3s/R-GITOPS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every variable a substituted Flux path renders is defined in cluster-vars or
+      escaped, so post-build substitution cannot silently empty a value and enabling
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
+spec:
+  steps:
+    - name: audit-every-substituted-flux-path
+      description: >-
+        Resolve each postBuild.substituteFrom Kustomization to the files it renders and
+        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/flux-postbuild-substitution/check.sh

@@ -0,0 +1,369 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — flux-postbuild-substitution
+# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
+#                   — spec docs/02-environments-and-provisioning.md)
+#         R-GITOPS-01 ("All desired state in git, reconciled by Flux."
+#                   — spec docs/05-gitops-and-delivery.md)
+#
+# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
+# --------------------------------------------------------
+# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
+# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
+# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
+# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
+# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
+# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
+# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
+# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
+# the repo asserted that. This check does.
+#
+# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
+# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
+#   ${DEFINED}        -> substituted with the cluster-vars value.
+#   ${UNDEFINED}      -> EMPTY STRING, silently. `--strict` / the cluster-wide
+#                        StrictPostBuildSubstitutions feature gate turns this into a
+#                        loud Kustomization failure ("variable not set (strict mode)").
+#   ${UNDEFINED:-d}   -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
+#                        NOT catch this (it exempts default-operator forms by design),
+#                        so a shell guard written `[ -z "${X:-}" ]` inside a rendered
+#                        manifest is pre-resolved to `[ -z "" ]` and its else-branch
+#                        becomes unreachable in the container. Silent, and permanent.
+#   $${VAR}, $${V:-d}, $${ARR[0]}  -> emitted as the LITERAL ${...} for a downstream
+#                        shell/cloud-init consumer. Escaping is total: it neutralises
+#                        even the indexed form.
+#   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
+#   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
+#                        variable name" / "missing closing brace") which fails the
+#                        WHOLE Kustomization — every resource in the path, not just
+#                        the offending one.
+# The escape contract this enforces is the one already documented in
+# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
+#
+# GIVEN the repository as authored,
+# WHEN  every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
+#       is resolved to the exact file set its `spec.path` renders (by following
+#       kustomization.yaml `resources:` transitively — a path, not a glob),
+# THEN  (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
+#       (2) no UNESCAPED expansion carries a default/length/substitution operator on an
+#           undefined name (the silent-empty class strict mode does not catch);
+#       (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
+#           that fails the entire Kustomization, so it is fatal wherever it appears;
+#       (4) every Kustomization that renders a cluster-vars token actually declares
+#           `substituteFrom` a ConfigMap named cluster-vars (else the literal token
+#           ships into the cluster);
+#       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
+#           the "add the knob with a base default BEFORE referencing it" rule that the
+#           incident's interim discipline depends on;
+#   AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
+#           ci/gate.sh sets the merge block when the l0 stage fails.
+# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
+# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
+# comment survives into the built apps/wave-1 output), so a braced expression in a
+# comment is a live hazard, not prose.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does NOT render the manifests. It scans the SOURCE files a path includes,
+#     which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
+#     drops some comments). A finding in a comment may not reach envsubst TODAY; it is
+#     still reported, because whether a given comment survives the build is a kustomize
+#     implementation detail that changes with the pinned version, and the failure it
+#     causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
+#     flux envsubst --strict` for every substituted path — needs kustomize + flux,
+#     which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
+#     l0), so it cannot run here without permanently SKIPPING. It was run by hand and
+#     recorded in DECISIONS-LOG; this check is the reproducible static half.
+#   - It does NOT prove the cluster's live state matches git, nor that the
+#     StrictPostBuildSubstitutions feature gate is actually enabled on
+#     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
+#     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
+#     is fully determined, with no variable silently emptied on the way in.
+#   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
+#     that an overlay's LAN block is self-consistent). Presence, not correctness.
+#   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
+#     several are deliberately reference-only and say so in cluster-vars.yaml
+#     (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
+#     failing on them would be a false block on documented intent.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green) if the thing under test is not present at all.
+if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
+  echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of Flux postBuild substitution safety. Pure standard library.
+
+No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
+documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
+`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
+definition of "what a Flux path actually renders" in the harness.
+"""
+from __future__ import annotations
+
+import os
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st                     # noqa: E402 — reuse the walk, don't fork it
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# ------------------------------------------------------------------ cluster-vars
+# Parse contract: `data:` block of a single ConfigMap, one `  KEY: "value"` per line
+# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
+# continuation-comment lines are indented further and never match).
+_DATA_KEY_RE = re.compile(r'^  ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
+
+
+def cluster_vars_keys(path: Path) -> set[str]:
+    keys: set[str] = set()
+    in_data = False
+    for raw in path.read_text().splitlines():
+        if re.match(r"^data:\s*$", raw):
+            in_data = True
+            continue
+        if not in_data:
+            continue
+        if st._TOP_KEY_RE.match(raw):   # a new top-level key ends the data block
+            break
+        m = _DATA_KEY_RE.match(raw)
+        if m:
+            keys.add(m.group(1))
+    return keys
+
+
+VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+KEYS = cluster_vars_keys(VARS_FILE)
+check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
+
+# ------------------------------------------- Flux Kustomizations with substituteFrom
+# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
+# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
+# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
+# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
+_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
+_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
+_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
+
+
+def substitute_sources(doc: str) -> list[tuple[str, str]]:
+    """[(kind, name)] declared under postBuild.substituteFrom in this doc."""
+    m = _SUBFROM_RE.search(doc)
+    if not m:
+        return []
+    indent = len(m.group(1))
+    out: list[tuple[str, str]] = []
+    kind = ""
+    for raw in doc[m.end():].splitlines():
+        if raw.strip() == "" or raw.lstrip().startswith("#"):
+            continue
+        if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
+            break                                    # dedented out of the list
+        km = _SUBITEM_KIND_RE.match(raw)
+        if km:
+            kind = km.group(1)
+            continue
+        nm = _SUBITEM_NAME_RE.match(raw)
+        if nm and kind:
+            out.append((kind, nm.group(1)))
+            kind = ""
+    return out
+
+
+kustomizations: list[dict] = []
+for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
+    text = yfile.read_text()
+    if st.FLUX_API not in text:
+        continue
+    for doc in st._DOC_SPLIT_RE.split(text):
+        if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
+            continue
+        pm = st._PATH_RE.search(doc)
+        if not pm:
+            continue
+        nm = st._NAME_RE.search(doc)
+        srcs = substitute_sources(doc)
+        if not srcs:
+            continue                                  # no substitution => nothing to audit
+        kustomizations.append({
+            "file": str(yfile.relative_to(ROOT)),
+            "name": nm.group(1) if nm else "?",
+            "path": pm.group(1),
+            "sources": srcs,
+        })
+
+check(len(kustomizations) > 0,
+      "no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
+      "the parser or the repo layout changed; this check would silently audit nothing")
+
+# ------------------------------------------------------------------ token scanning
+# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
+# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
+# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
+TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
+# these, so an undefined name yields the default/empty AT RECONCILE TIME.
+OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
+LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
+
+
+def classify(dollars: int, inner: str) -> tuple[str, str] | None:
+    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
+    if dollars >= 2:
+        return None                                   # escaped: emitted as a literal
+    if NAME_RE.match(inner):
+        if inner in KEYS:
+            return None                               # defined knob — the happy path
+        return ("UNDEFINED",
+                f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
+                f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
+                f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
+    m = OP_RE.match(inner) or LEN_RE.match(inner)
+    if m:
+        base = m.group(1)
+        if base in KEYS:
+            return None                               # defined knob with an operator
+        return ("SILENT-EMPTY",
+                f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
+                f"cluster-var, so the rendered manifest carries the default, not the "
+                f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
+                f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
+    return ("FATAL-PARSE",
+            "envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
+            "in the path). Escape it as $${...} or rewrite the prose without braces")
+
+
+def rendered_files(path_spec: str) -> tuple[list[str], bool]:
+    """Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
+    rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
+    if not (ROOT / rel).is_dir():
+        return [], False
+    applied, _ = st.reconciled_files(ROOT, rel)
+    return sorted(applied), True
+
+
+findings: list[str] = []
+referenced: dict[str, set[str]] = {}
+inventory: list[str] = []
+
+for k in kustomizations:
+    files, exists = rendered_files(k["path"])
+    check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
+                  f"which is not a directory in this repo")
+    has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
+    n_def = n_esc = n_bad = 0
+    for rel in files:
+        f = ROOT / rel
+        try:
+            text = f.read_text()
+        except (OSError, UnicodeDecodeError):
+            continue
+        for lineno, line in enumerate(text.splitlines(), 1):
+            for m in TOKEN_RE.finditer(line):
+                dollars, inner = len(m.group(1)), m.group(2)
+                verdict = classify(dollars, inner)
+                if verdict is None:
+                    if dollars >= 2:
+                        n_esc += 1
+                    else:
+                        n_def += 1
+                        referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
+                                              set()).add(k["name"])
+                        # (4) a DEFINED knob is only substituted if this Kustomization
+                        #     actually pulls cluster-vars in; otherwise the literal ships.
+                        check(has_cluster_vars,
+                              f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
+                              f"Kustomization '{k['name']}' ({k['file']}) does not "
+                              f"substituteFrom ConfigMap/cluster-vars — the literal token "
+                              f"would ship into the cluster")
+                    continue
+                n_bad += 1
+                tag, fix = verdict
+                findings.append(f"  [{tag}] {rel}:{lineno}  {m.group(0)}\n"
+                                f"        rendered by Flux Kustomization '{k['name']}' "
+                                f"({k['file']}, path {k['path']})\n"
+                                f"        {fix}\n"
+                                f"        | {line.strip()[:120]}")
+    inventory.append(f"  {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
+                     f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
+
+check(not findings,
+      f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
+
+# --------------------------------------------- (5) overlay knobs exist in the base
+# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
+# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
+# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
+# manifest references it — that ordering is what closes the define-then-use race.
+_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
+for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+    for key in set(_OP_PATH_RE.findall(overlay.read_text())):
+        check(key in KEYS,
+              f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
+              f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
+              f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
+
+# ------------------------------------------------------- (6) the check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+check("flux-postbuild-substitution/check.sh" in L0,
+      "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
+      "conformance selection can filter this check out of a gate run, so l0.sh must invoke "
+      "it directly and unconditionally or the gate is not actually enforcing it")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ------------------------------------------------------------- advisory: dead knobs
+for key in sorted(KEYS - set(referenced)):
+    ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
+                      f"(dead knob, or documentation-only — advisory, not a failure)")
+
+# ------------------------------------------------------------------------- report
+print("Flux postBuild substitution inventory (per Kustomization):")
+for line in inventory:
+    print(line)
+print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+if findings:
+    print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
+    for f in findings:
+        print(f, file=sys.stderr)
+
+print()
+if not FAILURES:
+    print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
+          f"(R-ENV-02, R-GITOPS-01)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

infra/PROD-PROVISIONING.md

@@ -311,6 +311,32 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
       own comment flags this as owed "when hardware is built."
 - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
       byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
+- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
+      the `dependsOn` fix. Only the first is now closed:
+      - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
+            *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
+            `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
+            forever on a dependency that does not exist and nothing downstream would reconcile.
+      - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
+            as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
+            exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
+            (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
+      - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
+            2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
+            will be decommissioned once prod is stable. Recorded consequence: once prod is live,
+            **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
+            The URL is patched in but **cannot work on a cold prod cluster**, for two independent
+            reasons found 2026-09-01:
+            1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
+               decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
+               loses its source of truth. **This is the decommission plan's first task, not its last.**
+            2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
+               `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
+               prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
+               `gitea.home.keithjackson.dev` to itself and never reach a repository.
+            Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
+            cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
+            Phase-6 cutover.
 - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
       the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
 

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -42,5 +42,22 @@ cgroup:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
 
+# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
+# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
+# 3-node cluster lets one bad render take down two of three agents simultaneously —
+# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config`
+# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium
+# agent can translate that ClusterIP, and the agent in question was the one being
+# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked.
+# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one
+# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP
+# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1
+# the DaemonSet will not proceed past an unready pod, bounding damage to a single node.
+# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade.
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
+
 operator:
   replicas: {{ cilium_operator_replicas }}

platform/cilium/helmrelease.yaml

@@ -89,6 +89,27 @@ spec:
       autoMount:
         enabled: false
       hostRoot: /run/cilium/cgroupv2
+    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
+    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
+    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
+    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
+    # Helm render take down two of three agents simultaneously. That is exactly what happened
+    # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
+    # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
+    # agent itself can translate — and the agent translating it was the one being replaced.
+    # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
+    # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
+    # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
+    # will not proceed past an unready pod, so a bad render costs one node, not the cluster.
+    # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
+    # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
+    # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
+    # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
+    # grows enough that serialised CNI upgrades become an operational problem.
+    updateStrategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 1
     # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
     # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
     # (Flux/prod may differ). Only the [parity] values above must match the bootstrap.

platform/cilium/values.yaml

@@ -42,6 +42,16 @@ cgroup:
   autoMount:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
+# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
+# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
+# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
+# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade
+# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops
+# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes).
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
 # NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease:
 #   operator.replicas
 #

platform/kustomization.yaml

@@ -12,7 +12,15 @@ kind: Kustomization
 resources:
   # ---- Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress ----
   - kyverno            # admission enforcer FIRST — policies present before other components reconcile
-  - cilium             # CNI (adopts the running Ansible-installed release; parity verified)
+  #      cilium (the CNI) is DELIBERATELY NOT here: it now reconciles in its OWN Flux Kustomization
+  #      (clusters/base/cilium.yaml, dependsOn flux-system, wait:true), and `platform` dependsOn IT.
+  #      Not the headlamp/oauth2-proxy rationale (a leaf that must not gate) but its mirror image:
+  #      the CNI is the one component that SHOULD gate everything, and inside this shared tier its
+  #      health was indistinguishable from eleven unrelated components, the only emergency valve was
+  #      freezing the whole tier, and the tier's 10m/5m interval/timeout are wrong for a 3-node
+  #      DaemonSet rollout. Gating is preserved via the explicit `platform.dependsOn: [cilium]` edge.
+  #      NOTE: platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system in
+  #      this tree — that is why the move required a prune-disabled transition. See DECISIONS-LOG.
   - longhorn           # replicated block storage / CSI
   - external-snapshotter  # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
   - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`

platform/longhorn/helmrelease.yaml

@@ -2,7 +2,7 @@
 # Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003).
 # Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication).
 #
-# Dev-appropriate values. Environment differences enter ONLY via ${...} substitution from
+# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from
 # cluster-vars (R-ENV-02); everything else is identical base config for dev and prod.
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease

platform/longhorn/storageclasses.yaml

@@ -20,7 +20,7 @@ parameters:
   # (replicaSoftAntiAffinity=false) guarantees these land on distinct nodes, so any single node
   # loss keeps the volume available.
   # HARDCODED, not ${LONGHORN_DEFAULT_REPLICA_COUNT}: StorageClass.parameters is map[string]string,
-  # but kustomize strips the quotes around a bare "${VAR}" placeholder, so Flux postBuild substitution
+  # but kustomize strips the quotes around a bare "$${VAR}" placeholder, so Flux postBuild substitution
   # yields an unquoted numeric (3) that server-side apply rejects (int, not string). A quoted numeric
   # literal like "3"/"30" IS preserved by kustomize. The replica knob still drives the GLOBAL default
   # via defaultSettings.defaultReplicaCount in helmrelease.yaml (a Helm int value, unaffected).