Compare commits

..
Author SHA1 Message Date
jakman85 645994e3a3 docs: record P0/P1/P2 landing + the postStart crashloop incident
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Two bugs stacked and the outer one hid the inner. Fixing the dead OIDC guard made the
block execute for the first time in these apps' deployed lifetime, which immediately
exposed a latent defect: the hook's last statement was `[ cond ] && action`, so on the
SUCCESS path it returned 1 and became the hook's exit status — and a non-zero postStart
exit makes the kubelet kill the container.

Records the pattern for the next PM: fixing dead code is not a safe no-op. It is the first
execution of never-tested code and deserves the caution of new code, not of a
one-character escape.

Also records what is verified live (Cilium maxUnavailable applied with pod ages UNCHANGED,
proving nothing rolled), and that strict substitution is committed but NOT ACTIVE — it
needs bootstrap.sh step 1 because the controllers sit outside Flux's inventory.
2026-09-01 16:59:41 -05:00
gitea-admin ac9f20034c Merge pull request 'fix(sso): postStart hooks must exit 0 — nextcloud crashloop' (#6) from fix/poststart-exit-status into main 2026-09-01 21:55:38 +00:00
gitea-admin 63002d19c8 Merge pull request 'Close the post-build substitution race: dependsOn + strict mode + CNI blast radius' (#5) from fix/strict-postbuild-substitutions into main 2026-09-01 21:30:53 +00:00
+64
View File
@@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection.
`${...}` that can silently drift from the real HelmRelease.
- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
out while a legitimate Longhorn upgrade is still running.
---
## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
### Landed and verified on the live cluster
| Change | Verified |
|---|---|
| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
```sh
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook
```
On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
dead code is not a safe no-op — it is the first execution of never-tested code, and should be
treated with the caution of new code, not of a one-character escape.
Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
on both paths, but only by accident of which statement happened to be last — made explicit there too
so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
Kustomizations Ready, both OIDC providers still registered.
### STILL TO DO — strict substitution is committed but NOT ACTIVE
`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
With the gate on but the ordering edge absent, the same race would turn a transient empty render
into a transient hard Kustomization failure.
### Prod consequences recorded this session
- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
first" safety net that the last two days have leaned on heavily, twice.
- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
currently points at Gitea running on the dev cluster.
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.