Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
645994e3a3 | ||
|
|
ac9f20034c | ||
|
|
63002d19c8 |
@@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection.
|
||||
`${...}` that can silently drift from the real HelmRelease.
|
||||
- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
|
||||
out while a legitimate Longhorn upgrade is still running.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
|
||||
|
||||
All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
|
||||
|
||||
### Landed and verified on the live cluster
|
||||
|
||||
| Change | Verified |
|
||||
|---|---|
|
||||
| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
|
||||
| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
|
||||
| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
|
||||
| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
|
||||
| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
|
||||
|
||||
### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
|
||||
|
||||
Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
|
||||
`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
|
||||
|
||||
```sh
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook
|
||||
```
|
||||
|
||||
On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
|
||||
the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
|
||||
|
||||
**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
|
||||
long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
|
||||
execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
|
||||
**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
|
||||
dead code is not a safe no-op — it is the first execution of never-tested code, and should be
|
||||
treated with the caution of new code, not of a one-character escape.
|
||||
|
||||
Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
|
||||
on both paths, but only by accident of which statement happened to be last — made explicit there too
|
||||
so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
|
||||
Kustomizations Ready, both OIDC providers still registered.
|
||||
|
||||
### STILL TO DO — strict substitution is committed but NOT ACTIVE
|
||||
|
||||
`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
|
||||
`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
|
||||
ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
|
||||
controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
|
||||
standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
|
||||
~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
|
||||
|
||||
**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
|
||||
With the gate on but the ordering edge absent, the same race would turn a transient empty render
|
||||
into a transient hard Kustomization failure.
|
||||
|
||||
### Prod consequences recorded this session
|
||||
|
||||
- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
|
||||
soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
|
||||
first" safety net that the last two days have leaned on heavily, twice.
|
||||
- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
|
||||
currently points at Gitea running on the dev cluster.
|
||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||
Reference in new issue
Block a user