diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 186e673..cf00eac 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection. `${...}` that can silently drift from the real HelmRelease. - `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time out while a legitimate Longhorn upgrade is still running. + +--- + +## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner + +All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6). + +### Landed and verified on the live cluster + +| Change | Verified | +|---|---| +| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged | +| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below | +| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted | +| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered | +| Prod overlay | SOPS decryption (same age key as dev) + source on `main` | + +### THE SECOND INCIDENT — nextcloud CrashLoopBackOff + +Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts, +`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause: + +```sh +[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook +``` + +On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became +the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.** + +**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as +long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block +execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly. +**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing +dead code is not a safe no-op — it is the first execution of never-tested code, and should be +treated with the caution of new code, not of a one-character escape. + +Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0 +on both paths, but only by accident of which statement happened to be last — made explicit there too +so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all +Kustomizations Ready, both OIDC providers still registered. + +### STILL TO DO — strict substitution is committed but NOT ACTIVE + +`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use +`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux +ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a +controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the +standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for +~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried. + +**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely. +With the gate on but the ordering edge absent, the same race would turn a transient empty render +into a transient hard Kustomization failure. + +### Prod consequences recorded this session + +- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no + soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev + first" safety net that the last two days have leaned on heavily, twice. +- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source + currently points at Gitea running on the dev cluster. +- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone + with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway. + Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.