# Wave-based app-tier delivery (PROTOTYPE — for PM review) **Problem.** Today one Flux Kustomization `apps` (clusters/base/apps.yaml, `dependsOn: platform-config`, `wait: true`, `prune: true`) reconciles all of `apps/` — ~10 services + CNPG databases + SOPS secrets. Because `wait: true` health-gates the **entire** set, a single crashlooping workload blocks **all** apps and every new revision (this has bitten twice). **Solution.** Split the tier into dependency-ordered **waves**. Each wave is its own Flux Kustomization `apps-wave-` that `dependsOn` wave N-1 (wave 0 `dependsOn` `platform-config`), with `wait: true` + `prune: true` and `path: ./apps/wave-`. `wait: true` now health-gates **only that wave**, so a crashloop isolates to its own wave and everything downstream, while services **within** a wave still reconcile in parallel. Wave assignment is computed by `catalog/render.py` from the declared dependency graph, so future onboarding auto-places a service in the correct wave. --- ## 1. Wave assignment algorithm `wave(node)` = **longest dependency-path depth** over the catalog `dependencies:` edges (service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node is wave ≥ k+1. Implemented in `catalog/render.py :: compute_waves()`. **Co-located CNPG database rule.** A service `` that declares a dependency `to: -db` (e.g. `nextcloud` → `nextcloud-db`, `semaphore` → `semaphore-db`) has a hard DB prerequisite. The `-db` node carries **no** ServiceDefinition, so it is a **leaf** (wave 0); because the consumer declares a dependency on it, the consumer lands in a strictly higher wave. **The "DB strictly below its consumer" guarantee therefore falls straight out of the longest-path rule — no special case.** The DB manifest (`-db.yaml`) and its CNPG `initdb` owner secret (`-db-app.sops.yaml`) are placed in the DB's (lower) wave; the app manifest and its `-bootstrap-secret.sops.yaml` are placed in the consumer's wave. **Non-catalog units.** A few app-tier files carry no ServiceDefinition (`compute_waves` can't see them). They are anchored explicitly in `render.py :: EXTRA_UNITS`. Today the only one is **gitea-actions** (the in-cluster `act_runner` + its DinD sidecar), anchored to `gitea` with `above: true` — the runner registers against and clones from Gitea, so Gitea must be healthy first → it lands one wave **above** gitea. Cycle detection: `compute_waves` raises on any dependency cycle. ## 2. Computed WAVE TABLE Verify with `python3 catalog/render.py --print-waves` (nodes) and `--emit-waves` (file plan). | Wave | File | Kind | Reason (dependency path) | |---|---|---|---| | **0** | `pihole.yaml` | app | no dependencies (leaf) | | 0 | `gitea.yaml` (+ `gitea-bootstrap-secret.sops.yaml`) | app + secret | no dependencies (leaf) | | 0 | `jackett.yaml` | app | no dependencies (leaf) | | 0 | `plex.yaml` | app | no dependencies (leaf) | | 0 | `qbittorrent.yaml` | app | no dependencies (leaf) | | 0 | `nextcloud-db.yaml` (+ `nextcloud-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `nextcloud` | | 0 | `semaphore-db.yaml` (+ `semaphore-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `semaphore` | | **1** | `sonarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) | | 1 | `radarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) | | 1 | `nextcloud.yaml` (+ `nextcloud-bootstrap-secret.sops.yaml`) | app + secret | → `nextcloud-db` (w0) | | 1 | `semaphore.yaml` (+ `semaphore-bootstrap-secret.sops.yaml`) | app + secret | → `semaphore-db` (w0) | | 1 | `gitea-actions.yaml` (+ `gitea-actions-runner-secret.sops.yaml`) | CI runner + secret | EXTRA_UNITS: anchor `gitea` (w0), `above` → one wave up | | **2** | `seerr.yaml` | app | → `sonarr` (w1), `radarr` (w1) | Catalog services **not yet onboarded** (no committed manifest) are computed but **not placed**: `keycloak` (w1, → `keycloak-db` w0) and `vm-demo` (w0). They join a wave automatically when onboarded. Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = **130 resources**, byte-for-byte the same set the monolithic `apps` Kustomization built before the split (validated with `kubectl kustomize`). ## 3. Flux Kustomization topology ``` platform (controllers/CRDs) │ healthy ▼ platform-config (cluster CRs) │ healthy ▼ apps-wave-0 → ./apps/wave-0 dependsOn platform-config wait:true prune:true (leaves + CNPG DBs) │ healthy ▼ apps-wave-1 → ./apps/wave-1 dependsOn apps-wave-0 wait:true prune:true (arr, DB-backed apps, gitea-actions) │ healthy ▼ apps-wave-2 → ./apps/wave-2 dependsOn apps-wave-1 wait:true prune:true (seerr) ``` Every wave Kustomization is identical to the retired `apps` one except `name`, `path`, and `dependsOn`: same `interval/retryInterval/timeout`, `prune: true`, `wait: true`, `postBuild.substituteFrom cluster-vars`, and the **same** SOPS decryption (added per-environment in `clusters/dev/kustomization.yaml` — one inlined patch per `apps-wave-`, referencing the out-of-band `sops-age` Secret by name only, exactly as the single `apps` Kustomization did). **Failure isolation gained:** a crashloop in `seerr` (wave 2) no longer blocks `apps-wave-0` or `apps-wave-1`, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its downstream), never wave 0. Before, any one crashloop stalled the whole tier. ## 4. What `render.py` now does - `compute_waves(sds)` — longest-path wave per node (services + DB leaf nodes); cycle-safe. - `plan_apps_waves(sds, present_files)` — maps the committed app file set → `{wave: [files]}` by naming convention (``, `-bootstrap-secret`, `-db`, `-db-app`) plus `EXTRA_UNITS`; only files actually present are placed (not-yet-onboarded SDs are skipped). - `--emit-waves` — (re)generates every `apps/wave-/kustomization.yaml` and `clusters/base/apps-wave-.yaml`. Idempotent; **never rewrites a service manifest** (only placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched. - `--print-waves` — prints the node→wave table. - `-o --wave-layout` — future onboarding writes `/wave-/.yaml` directly. - Self-test extended (`--selftest`): longest-path depth, CNPG-DB-below-consumer, cycle detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait). **Onboarding a new service now:** add `catalog/.yaml`, render it (`render.py catalog/.yaml -o apps --wave-layout [--no-image-automation]`), then `render.py --emit-waves`. If it lands in an existing wave, no clusters/base edit is needed. If it creates a **new deepest** wave, add one line to `clusters/base/kustomization.yaml` and one decryption patch to `clusters/dev/kustomization.yaml` (both flagged in-file). --- ## 5. CUTOVER RUNBOOK — Flux resource adoption without data loss > **Read this before merge.** Moving a manifest from the `apps` Kustomization to an > `apps-wave-` Kustomization is a Flux **ownership transfer** (adoption). Done wrong, one > Kustomization can **prune** (delete) a resource another is adopting — and for the CNPG > databases that is **data loss**. ### 5.1 Why the naive "just merge" is dangerous Flux marks every resource it applies with `kustomize.toolkit.fluxcd.io/name` + `…/namespace`, and with `prune: true` it **garbage-collects** any owned resource that leaves its build. Two hazards on a plain merge of this PR: 1. **The live `apps` Kustomization has `prune: true`.** This PR removes `apps.yaml` from `clusters/base`. The root Kustomization (`flux-system`, `./clusters/dev`, `prune: true`) will then **delete the `apps` Kustomization object**. Deleting a Flux Kustomization that has `prune: true` runs its finalizer, which **garbage-collects every resource it owns** — all ~130 app resources, **including the `nextcloud-db` / `semaphore-db` CNPG Clusters**. Deleting a CNPG `Cluster` cascade-deletes its PVCs → **the databases are destroyed**, even though the new waves were about to adopt them. 2. **Ordering race.** Even setting that aside, if the old Kustomization's prune and the new waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart for stateless apps, and unacceptable for the stateful DBs. The fix is to make the old Kustomization **release** its resources (orphan-alive, no GC) so the waves **adopt** them by identity — never a delete/recreate window. ### 5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime) Split the change into two PRs so the release happens before the split, entirely through git. **Pre-flight (no changes):** - `flux get kustomizations` → `apps` is `Ready`. Cluster healthy. - Record the stateful state you must not lose: `kubectl get cluster -A`, `kubectl get pvc -n nextcloud-db`, `kubectl get pvc -n semaphore-db` (note names + `AGE`). Take a Longhorn/Velero backup of the `nextcloud-db` and `semaphore-db` namespaces (defense in depth). **PR 1 — neutralize GC on the old `apps` Kustomization (one-line change):** - Edit `clusters/base/apps.yaml`: `prune: true` → `prune: false`. Merge. - Wait until `flux get kustomization apps` reconciles with the new spec (`kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}'` → `false`). - Effect: when `apps` is later deleted, its finalizer will **not** garbage-collect its resources — they survive as orphaned-but-alive objects. **PR 2 — this wave-split PR.** On merge, the root Kustomization reconciles `./clusters/dev`: - `apps` is gone from the build → root prunes the **`apps` Kustomization object**. Because `apps` now has `prune: false` (from PR 1), **no app resources are deleted**; they are orphaned, still running. - `apps-wave-0/1/2` are created. Each does a server-side apply of the **same, byte-identical** manifests, **adopting** the live objects (matched by GVK + namespace + name) and relabeling `kustomize.toolkit.fluxcd.io/name` to its wave. Because content is unchanged, SSA is a no-op on `spec` — **no pod restart, no PVC churn, no DB re-init**. Same field manager (`kustomize-controller`) → no ownership conflict. - Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and corrupt the Secret. **Verify:** ``` flux get kustomizations # apps-wave-0/1/2 Ready; apps absent kubectl get cluster -A # DBs present, ORIGINAL age (not recreated) kubectl get pvc -n nextcloud-db -n semaphore-db # same PVCs, same age kubectl get deploy -A -o wide # apps not restarted (AGE/restarts unchanged) kubectl get -n \ -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}' # == apps-wave- ``` ### 5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1) If you would rather merge this PR alone, perform PR 1's effect out-of-band **first**, then merge: ``` flux -n flux-system suspend kustomization apps # stop it racing the waves kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}' ``` Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of which Kustomization ends up owning them during the window): ``` kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune=disabled kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune=disabled ``` Then merge this PR and verify as in 5.2. **Cleanup once the waves own the DBs** (restore normal prune semantics): ``` kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune- kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune- ``` ### 5.4 Rollback Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the resources are still alive (orphaned). `flux suspend` the waves, revert the merge (re-creating `apps`, which re-adopts by identity), and investigate. No data is lost because the cutover never issues a delete. ### 5.5 Why adoption is safe here (summary of the invariant) - The old owner **releases without GC** (`prune: false` before it is deleted) → no delete. - The new owners **adopt by identity** via SSA on byte-identical manifests → no recreate. - There is **no window** in which a resource is owned by nothing *and* pruned: it is continuously alive, transitioning label ownership from `apps` → `apps-wave-`. --- ## 6. Risks / open items for the PM 1. **The GC hazard above is the whole ball game.** A plain merge of this PR onto a live cluster whose `apps` Kustomization still has `prune: true` will destroy the CNPG databases. §5.2 / §5.3 must be followed. (In dev this has never been applied — prototype only.) 2. **`prune` blast radius within a wave is unchanged.** Splitting reduces cross-wave coupling, but a wave is still `wait: true` — a crashloop in `gitea-actions` (wave 1) will block wave 2 (`seerr`) until it recovers or its `timeout` elapses. That is the intended linear-wave trade-off; if the PM wants `gitea-actions` (flaky privileged DinD) to not gate `seerr`, consider giving CI plumbing its own non-gating Kustomization branch (future work, not in this PR). 3. **New-deepest-wave onboarding needs two one-line manual edits** (`clusters/base/kustomization.yaml` + a `clusters/dev` decryption patch). `render.py --emit-waves` generates the CR but does not edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing in an existing wave is fully automatic. 4. **Image automation.** `platform/image-automation/image-update-automation.yaml` scans `path: ./apps` (recursive) for `$imagepolicy` setters — `apps/wave-/` is under `./apps`, so per-service image markers are still covered. No change needed; noted so it isn't a surprise. 5. **Decryption drift.** Each wave must carry the SOPS decryption patch (done in this PR for waves 0–2). If a future wave is added and the `clusters/dev` patch is forgotten, that wave will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and §4 call this out; a lint check that "every apps-wave- has a decryption patch" would be a good hardening follow-up. 6. **`EXTRA_UNITS` is a small hand-maintained table** (only `gitea-actions` today). Any future non-catalog app-tier unit must be added there or it will not be placed into a wave.