refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
1bfb0db6f1
commit
36a1a4e1f9
34 files changed
+793
-93
No files matched your search
+249
@@ -0,0 +1,249 @@
|
||||
# Wave-based app-tier delivery (PROTOTYPE — for PM review)
|
||||
|
||||
**Problem.** Today one Flux Kustomization `apps` (clusters/base/apps.yaml, `dependsOn:
|
||||
platform-config`, `wait: true`, `prune: true`) reconciles all of `apps/` — ~10 services + CNPG
|
||||
databases + SOPS secrets. Because `wait: true` health-gates the **entire** set, a single
|
||||
crashlooping workload blocks **all** apps and every new revision (this has bitten twice).
|
||||
|
||||
**Solution.** Split the tier into dependency-ordered **waves**. Each wave is its own Flux
|
||||
Kustomization `apps-wave-<N>` that `dependsOn` wave N-1 (wave 0 `dependsOn` `platform-config`),
|
||||
with `wait: true` + `prune: true` and `path: ./apps/wave-<N>`. `wait: true` now health-gates
|
||||
**only that wave**, so a crashloop isolates to its own wave and everything downstream, while
|
||||
services **within** a wave still reconcile in parallel. Wave assignment is computed by
|
||||
`catalog/render.py` from the declared dependency graph, so future onboarding auto-places a
|
||||
service in the correct wave.
|
||||
|
||||
---
|
||||
|
||||
## 1. Wave assignment algorithm
|
||||
|
||||
`wave(node)` = **longest dependency-path depth** over the catalog `dependencies:` edges
|
||||
(service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node
|
||||
is wave ≥ k+1. Implemented in `catalog/render.py :: compute_waves()`.
|
||||
|
||||
**Co-located CNPG database rule.** A service `<svc>` that declares a dependency `to: <svc>-db`
|
||||
(e.g. `nextcloud` → `nextcloud-db`, `semaphore` → `semaphore-db`) has a hard DB prerequisite.
|
||||
The `<svc>-db` node carries **no** ServiceDefinition, so it is a **leaf** (wave 0); because the
|
||||
consumer declares a dependency on it, the consumer lands in a strictly higher wave. **The
|
||||
"DB strictly below its consumer" guarantee therefore falls straight out of the longest-path
|
||||
rule — no special case.** The DB manifest (`<svc>-db.yaml`) and its CNPG `initdb` owner secret
|
||||
(`<svc>-db-app.sops.yaml`) are placed in the DB's (lower) wave; the app manifest and its
|
||||
`<svc>-bootstrap-secret.sops.yaml` are placed in the consumer's wave.
|
||||
|
||||
**Non-catalog units.** A few app-tier files carry no ServiceDefinition (`compute_waves` can't
|
||||
see them). They are anchored explicitly in `render.py :: EXTRA_UNITS`. Today the only one is
|
||||
**gitea-actions** (the in-cluster `act_runner` + its DinD sidecar), anchored to `gitea` with
|
||||
`above: true` — the runner registers against and clones from Gitea, so Gitea must be healthy
|
||||
first → it lands one wave **above** gitea.
|
||||
|
||||
Cycle detection: `compute_waves` raises on any dependency cycle.
|
||||
|
||||
## 2. Computed WAVE TABLE
|
||||
|
||||
Verify with `python3 catalog/render.py --print-waves` (nodes) and `--emit-waves` (file plan).
|
||||
|
||||
| Wave | File | Kind | Reason (dependency path) |
|
||||
|---|---|---|---|
|
||||
| **0** | `pihole.yaml` | app | no dependencies (leaf) |
|
||||
| 0 | `gitea.yaml` (+ `gitea-bootstrap-secret.sops.yaml`) | app + secret | no dependencies (leaf) |
|
||||
| 0 | `jackett.yaml` | app | no dependencies (leaf) |
|
||||
| 0 | `plex.yaml` | app | no dependencies (leaf) |
|
||||
| 0 | `qbittorrent.yaml` | app | no dependencies (leaf) |
|
||||
| 0 | `nextcloud-db.yaml` (+ `nextcloud-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `nextcloud` |
|
||||
| 0 | `semaphore-db.yaml` (+ `semaphore-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `semaphore` |
|
||||
| **1** | `sonarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
|
||||
| 1 | `radarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
|
||||
| 1 | `nextcloud.yaml` (+ `nextcloud-bootstrap-secret.sops.yaml`) | app + secret | → `nextcloud-db` (w0) |
|
||||
| 1 | `semaphore.yaml` (+ `semaphore-bootstrap-secret.sops.yaml`) | app + secret | → `semaphore-db` (w0) |
|
||||
| 1 | `gitea-actions.yaml` (+ `gitea-actions-runner-secret.sops.yaml`) | CI runner + secret | EXTRA_UNITS: anchor `gitea` (w0), `above` → one wave up |
|
||||
| **2** | `seerr.yaml` | app | → `sonarr` (w1), `radarr` (w1) |
|
||||
|
||||
Catalog services **not yet onboarded** (no committed manifest) are computed but **not placed**:
|
||||
`keycloak` (w1, → `keycloak-db` w0) and `vm-demo` (w0). They join a wave automatically when
|
||||
onboarded.
|
||||
|
||||
Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = **130 resources**, byte-for-byte the
|
||||
same set the monolithic `apps` Kustomization built before the split (validated with
|
||||
`kubectl kustomize`).
|
||||
|
||||
## 3. Flux Kustomization topology
|
||||
|
||||
```
|
||||
platform (controllers/CRDs)
|
||||
│ healthy
|
||||
▼
|
||||
platform-config (cluster CRs)
|
||||
│ healthy
|
||||
▼
|
||||
apps-wave-0 → ./apps/wave-0 dependsOn platform-config wait:true prune:true (leaves + CNPG DBs)
|
||||
│ healthy
|
||||
▼
|
||||
apps-wave-1 → ./apps/wave-1 dependsOn apps-wave-0 wait:true prune:true (arr, DB-backed apps, gitea-actions)
|
||||
│ healthy
|
||||
▼
|
||||
apps-wave-2 → ./apps/wave-2 dependsOn apps-wave-1 wait:true prune:true (seerr)
|
||||
```
|
||||
|
||||
Every wave Kustomization is identical to the retired `apps` one except `name`, `path`, and
|
||||
`dependsOn`: same `interval/retryInterval/timeout`, `prune: true`, `wait: true`,
|
||||
`postBuild.substituteFrom cluster-vars`, and the **same** SOPS decryption (added per-environment
|
||||
in `clusters/dev/kustomization.yaml` — one inlined patch per `apps-wave-<N>`, referencing the
|
||||
out-of-band `sops-age` Secret by name only, exactly as the single `apps` Kustomization did).
|
||||
|
||||
**Failure isolation gained:** a crashloop in `seerr` (wave 2) no longer blocks `apps-wave-0`
|
||||
or `apps-wave-1`, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its
|
||||
downstream), never wave 0. Before, any one crashloop stalled the whole tier.
|
||||
|
||||
## 4. What `render.py` now does
|
||||
|
||||
- `compute_waves(sds)` — longest-path wave per node (services + DB leaf nodes); cycle-safe.
|
||||
- `plan_apps_waves(sds, present_files)` — maps the committed app file set → `{wave: [files]}`
|
||||
by naming convention (`<svc>`, `<svc>-bootstrap-secret`, `<svc>-db`, `<svc>-db-app`) plus
|
||||
`EXTRA_UNITS`; only files actually present are placed (not-yet-onboarded SDs are skipped).
|
||||
- `--emit-waves` — (re)generates every `apps/wave-<N>/kustomization.yaml` and
|
||||
`clusters/base/apps-wave-<N>.yaml`. Idempotent; **never rewrites a service manifest** (only
|
||||
placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched.
|
||||
- `--print-waves` — prints the node→wave table.
|
||||
- `-o <dir> --wave-layout` — future onboarding writes `<dir>/wave-<N>/<svc>.yaml` directly.
|
||||
- Self-test extended (`--selftest`): longest-path depth, CNPG-DB-below-consumer, cycle
|
||||
detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait).
|
||||
|
||||
**Onboarding a new service now:** add `catalog/<svc>.yaml`, render it
|
||||
(`render.py catalog/<svc>.yaml -o apps --wave-layout [--no-image-automation]`), then
|
||||
`render.py --emit-waves`. If it lands in an existing wave, no clusters/base edit is needed. If
|
||||
it creates a **new deepest** wave, add one line to `clusters/base/kustomization.yaml` and one
|
||||
decryption patch to `clusters/dev/kustomization.yaml` (both flagged in-file).
|
||||
|
||||
---
|
||||
|
||||
## 5. CUTOVER RUNBOOK — Flux resource adoption without data loss
|
||||
|
||||
> **Read this before merge.** Moving a manifest from the `apps` Kustomization to an
|
||||
> `apps-wave-<N>` Kustomization is a Flux **ownership transfer** (adoption). Done wrong, one
|
||||
> Kustomization can **prune** (delete) a resource another is adopting — and for the CNPG
|
||||
> databases that is **data loss**.
|
||||
|
||||
### 5.1 Why the naive "just merge" is dangerous
|
||||
|
||||
Flux marks every resource it applies with `kustomize.toolkit.fluxcd.io/name` +
|
||||
`…/namespace`, and with `prune: true` it **garbage-collects** any owned resource that leaves
|
||||
its build. Two hazards on a plain merge of this PR:
|
||||
|
||||
1. **The live `apps` Kustomization has `prune: true`.** This PR removes `apps.yaml` from
|
||||
`clusters/base`. The root Kustomization (`flux-system`, `./clusters/dev`, `prune: true`)
|
||||
will then **delete the `apps` Kustomization object**. Deleting a Flux Kustomization that
|
||||
has `prune: true` runs its finalizer, which **garbage-collects every resource it owns** —
|
||||
all ~130 app resources, **including the `nextcloud-db` / `semaphore-db` CNPG Clusters**.
|
||||
Deleting a CNPG `Cluster` cascade-deletes its PVCs → **the databases are destroyed**, even
|
||||
though the new waves were about to adopt them.
|
||||
2. **Ordering race.** Even setting that aside, if the old Kustomization's prune and the new
|
||||
waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart
|
||||
for stateless apps, and unacceptable for the stateful DBs.
|
||||
|
||||
The fix is to make the old Kustomization **release** its resources (orphan-alive, no GC) so the
|
||||
waves **adopt** them by identity — never a delete/recreate window.
|
||||
|
||||
### 5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime)
|
||||
|
||||
Split the change into two PRs so the release happens before the split, entirely through git.
|
||||
|
||||
**Pre-flight (no changes):**
|
||||
- `flux get kustomizations` → `apps` is `Ready`. Cluster healthy.
|
||||
- Record the stateful state you must not lose:
|
||||
`kubectl get cluster -A`, `kubectl get pvc -n nextcloud-db`, `kubectl get pvc -n semaphore-db`
|
||||
(note names + `AGE`). Take a Longhorn/Velero backup of the `nextcloud-db` and `semaphore-db`
|
||||
namespaces (defense in depth).
|
||||
|
||||
**PR 1 — neutralize GC on the old `apps` Kustomization (one-line change):**
|
||||
- Edit `clusters/base/apps.yaml`: `prune: true` → `prune: false`. Merge.
|
||||
- Wait until `flux get kustomization apps` reconciles with the new spec
|
||||
(`kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}'` → `false`).
|
||||
- Effect: when `apps` is later deleted, its finalizer will **not** garbage-collect its
|
||||
resources — they survive as orphaned-but-alive objects.
|
||||
|
||||
**PR 2 — this wave-split PR.** On merge, the root Kustomization reconciles `./clusters/dev`:
|
||||
- `apps` is gone from the build → root prunes the **`apps` Kustomization object**. Because
|
||||
`apps` now has `prune: false` (from PR 1), **no app resources are deleted**; they are
|
||||
orphaned, still running.
|
||||
- `apps-wave-0/1/2` are created. Each does a server-side apply of the **same, byte-identical**
|
||||
manifests, **adopting** the live objects (matched by GVK + namespace + name) and relabeling
|
||||
`kustomize.toolkit.fluxcd.io/name` to its wave. Because content is unchanged, SSA is a no-op
|
||||
on `spec` — **no pod restart, no PVC churn, no DB re-init**. Same field manager
|
||||
(`kustomize-controller`) → no ownership conflict.
|
||||
- Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted
|
||||
and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and
|
||||
corrupt the Secret.
|
||||
|
||||
**Verify:**
|
||||
```
|
||||
flux get kustomizations # apps-wave-0/1/2 Ready; apps absent
|
||||
kubectl get cluster -A # DBs present, ORIGINAL age (not recreated)
|
||||
kubectl get pvc -n nextcloud-db -n semaphore-db # same PVCs, same age
|
||||
kubectl get deploy -A -o wide # apps not restarted (AGE/restarts unchanged)
|
||||
kubectl get <any-app-deploy> -n <ns> \
|
||||
-o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}' # == apps-wave-<N>
|
||||
```
|
||||
|
||||
### 5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1)
|
||||
|
||||
If you would rather merge this PR alone, perform PR 1's effect out-of-band **first**, then
|
||||
merge:
|
||||
```
|
||||
flux -n flux-system suspend kustomization apps # stop it racing the waves
|
||||
kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}'
|
||||
```
|
||||
Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of
|
||||
which Kustomization ends up owning them during the window):
|
||||
```
|
||||
kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune=disabled
|
||||
kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune=disabled
|
||||
```
|
||||
Then merge this PR and verify as in 5.2. **Cleanup once the waves own the DBs** (restore normal
|
||||
prune semantics):
|
||||
```
|
||||
kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune-
|
||||
kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune-
|
||||
```
|
||||
|
||||
### 5.4 Rollback
|
||||
|
||||
Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the
|
||||
resources are still alive (orphaned). `flux suspend` the waves, revert the merge (re-creating
|
||||
`apps`, which re-adopts by identity), and investigate. No data is lost because the cutover
|
||||
never issues a delete.
|
||||
|
||||
### 5.5 Why adoption is safe here (summary of the invariant)
|
||||
|
||||
- The old owner **releases without GC** (`prune: false` before it is deleted) → no delete.
|
||||
- The new owners **adopt by identity** via SSA on byte-identical manifests → no recreate.
|
||||
- There is **no window** in which a resource is owned by nothing *and* pruned: it is
|
||||
continuously alive, transitioning label ownership from `apps` → `apps-wave-<N>`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks / open items for the PM
|
||||
|
||||
1. **The GC hazard above is the whole ball game.** A plain merge of this PR onto a live cluster
|
||||
whose `apps` Kustomization still has `prune: true` will destroy the CNPG databases. §5.2 /
|
||||
§5.3 must be followed. (In dev this has never been applied — prototype only.)
|
||||
2. **`prune` blast radius within a wave is unchanged.** Splitting reduces cross-wave coupling,
|
||||
but a wave is still `wait: true` — a crashloop in `gitea-actions` (wave 1) will block wave 2
|
||||
(`seerr`) until it recovers or its `timeout` elapses. That is the intended linear-wave
|
||||
trade-off; if the PM wants `gitea-actions` (flaky privileged DinD) to not gate `seerr`,
|
||||
consider giving CI plumbing its own non-gating Kustomization branch (future work, not in
|
||||
this PR).
|
||||
3. **New-deepest-wave onboarding needs two one-line manual edits** (`clusters/base/kustomization.yaml`
|
||||
+ a `clusters/dev` decryption patch). `render.py --emit-waves` generates the CR but does not
|
||||
edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing
|
||||
in an existing wave is fully automatic.
|
||||
4. **Image automation.** `platform/image-automation/image-update-automation.yaml` scans
|
||||
`path: ./apps` (recursive) for `$imagepolicy` setters — `apps/wave-<N>/` is under `./apps`,
|
||||
so per-service image markers are still covered. No change needed; noted so it isn't a
|
||||
surprise.
|
||||
5. **Decryption drift.** Each wave must carry the SOPS decryption patch (done in this PR for
|
||||
waves 0–2). If a future wave is added and the `clusters/dev` patch is forgotten, that wave
|
||||
will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and
|
||||
§4 call this out; a lint check that "every apps-wave-<N> has a decryption patch" would be a
|
||||
good hardening follow-up.
|
||||
6. **`EXTRA_UNITS` is a small hand-maintained table** (only `gitea-actions` today). Any future
|
||||
non-catalog app-tier unit must be added there or it will not be placed into a wave.
|
||||
Reference in new issue
Block a user