Merge pull request 'docs: P0/P1/P2 landed + postStart crashloop incident' (#7) from docs/poststart-incident into main

DECISIONS-LOG.md

@@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection.
   `${...}` that can silently drift from the real HelmRelease.
 - `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
   out while a legitimate Longhorn upgrade is still running.
+
+---
+
+## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
+
+All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
+
+### Landed and verified on the live cluster
+
+| Change | Verified |
+|---|---|
+| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
+| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
+| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
+| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
+| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
+
+### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
+
+Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
+`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
+
+```sh
+[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"   # LAST line of the hook
+```
+
+On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
+the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
+
+**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
+long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
+execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
+**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
+dead code is not a safe no-op — it is the first execution of never-tested code, and should be
+treated with the caution of new code, not of a one-character escape.
+
+Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
+on both paths, but only by accident of which statement happened to be last — made explicit there too
+so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
+Kustomizations Ready, both OIDC providers still registered.
+
+### STILL TO DO — strict substitution is committed but NOT ACTIVE
+
+`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
+`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
+ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
+controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
+standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
+~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
+
+**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
+With the gate on but the ordering edge absent, the same race would turn a transient empty render
+into a transient hard Kustomization failure.
+
+### Prod consequences recorded this session
+
+- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
+  soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
+  first" safety net that the last two days have leaned on heavily, twice.
+- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
+  currently points at Gitea running on the dev cluster.
+- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
+  with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
+  Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.