docs: record P0/P1/P2 landing + the postStart crashloop incident
Two bugs stacked and the outer one hid the inner. Fixing the dead OIDC guard made the block execute for the first time in these apps' deployed lifetime, which immediately exposed a latent defect: the hook's last statement was `[ cond ] && action`, so on the SUCCESS path it returned 1 and became the hook's exit status — and a non-zero postStart exit makes the kubelet kill the container. Records the pattern for the next PM: fixing dead code is not a safe no-op. It is the first execution of never-tested code and deserves the caution of new code, not of a one-character escape. Also records what is verified live (Cilium maxUnavailable applied with pod ages UNCHANGED, proving nothing rolled), and that strict substitution is committed but NOT ACTIVE — it needs bootstrap.sh step 1 because the controllers sit outside Flux's inventory.
This commit is contained in:
1 parent
ac9f20034c
commit
645994e3a3
1 file changed
+64
@@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection.
|
||||
`${...}` that can silently drift from the real HelmRelease.
|
||||
- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
|
||||
out while a legitimate Longhorn upgrade is still running.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
|
||||
|
||||
All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
|
||||
|
||||
### Landed and verified on the live cluster
|
||||
|
||||
| Change | Verified |
|
||||
|---|---|
|
||||
| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
|
||||
| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
|
||||
| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
|
||||
| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
|
||||
| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
|
||||
|
||||
### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
|
||||
|
||||
Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
|
||||
`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
|
||||
|
||||
```sh
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook
|
||||
```
|
||||
|
||||
On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
|
||||
the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
|
||||
|
||||
**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
|
||||
long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
|
||||
execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
|
||||
**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
|
||||
dead code is not a safe no-op — it is the first execution of never-tested code, and should be
|
||||
treated with the caution of new code, not of a one-character escape.
|
||||
|
||||
Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
|
||||
on both paths, but only by accident of which statement happened to be last — made explicit there too
|
||||
so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
|
||||
Kustomizations Ready, both OIDC providers still registered.
|
||||
|
||||
### STILL TO DO — strict substitution is committed but NOT ACTIVE
|
||||
|
||||
`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
|
||||
`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
|
||||
ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
|
||||
controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
|
||||
standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
|
||||
~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
|
||||
|
||||
**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
|
||||
With the gate on but the ordering edge absent, the same race would turn a transient empty render
|
||||
into a transient hard Kustomization failure.
|
||||
|
||||
### Prod consequences recorded this session
|
||||
|
||||
- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
|
||||
soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
|
||||
first" safety net that the last two days have leaned on heavily, twice.
|
||||
- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
|
||||
currently points at Gitea running on the dev cluster.
|
||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||
Reference in new issue
Block a user