home-network-k3s-impl/DESIGN-waves.md

Wave-based app-tier delivery (PROTOTYPE — for PM review)

Problem. Today one Flux Kustomization apps (clusters/base/apps.yaml, dependsOn: platform-config, wait: true, prune: true) reconciles all of apps/ — ~10 services + CNPG databases + SOPS secrets. Because wait: true health-gates the entire set, a single crashlooping workload blocks all apps and every new revision (this has bitten twice).

Solution. Split the tier into dependency-ordered waves. Each wave is its own Flux Kustomization apps-wave-<N> that dependsOn wave N-1 (wave 0 dependsOn platform-config), with wait: true + prune: true and path: ./apps/wave-<N>. wait: true now health-gates only that wave, so a crashloop isolates to its own wave and everything downstream, while services within a wave still reconcile in parallel. Wave assignment is computed by catalog/render.py from the declared dependency graph, so future onboarding auto-places a service in the correct wave.

---

1. Wave assignment algorithm

wave(node) = longest dependency-path depth over the catalog dependencies: edges (service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node is wave ≥ k+1. Implemented in catalog/render.py :: compute_waves().

Co-located CNPG database rule. A service <svc> that declares a dependency to: <svc>-db (e.g. nextcloud → nextcloud-db, semaphore → semaphore-db) has a hard DB prerequisite. The <svc>-db node carries no ServiceDefinition, so it is a leaf (wave 0); because the consumer declares a dependency on it, the consumer lands in a strictly higher wave. The "DB strictly below its consumer" guarantee therefore falls straight out of the longest-path rule — no special case. The DB manifest (<svc>-db.yaml) and its CNPG initdb owner secret (<svc>-db-app.sops.yaml) are placed in the DB's (lower) wave; the app manifest and its <svc>-bootstrap-secret.sops.yaml are placed in the consumer's wave.

Non-catalog units. A few app-tier files carry no ServiceDefinition (compute_waves can't see them). They are anchored explicitly in render.py :: EXTRA_UNITS. Today the only one is gitea-actions (the in-cluster act_runner + its DinD sidecar), anchored to gitea with above: true — the runner registers against and clones from Gitea, so Gitea must be healthy first → it lands one wave above gitea.

Cycle detection: compute_waves raises on any dependency cycle.

2. Computed WAVE TABLE

Verify with python3 catalog/render.py --print-waves (nodes) and --emit-waves (file plan).

Wave	File	Kind	Reason (dependency path)
0	pihole.yaml	app	no dependencies (leaf)
0	gitea.yaml (+ gitea-bootstrap-secret.sops.yaml)	app + secret	no dependencies (leaf)
0	jackett.yaml	app	no dependencies (leaf)
0	plex.yaml	app	no dependencies (leaf)
0	qbittorrent.yaml	app	no dependencies (leaf)
0	nextcloud-db.yaml (+ nextcloud-db-app.sops.yaml)	CNPG DB + secret	co-located DB leaf; strictly below nextcloud
0	semaphore-db.yaml (+ semaphore-db-app.sops.yaml)	CNPG DB + secret	co-located DB leaf; strictly below semaphore
1	sonarr.yaml	app	→ qbittorrent (w0), jackett (w0)
1	radarr.yaml	app	→ qbittorrent (w0), jackett (w0)
1	nextcloud.yaml (+ nextcloud-bootstrap-secret.sops.yaml)	app + secret	→ nextcloud-db (w0)
1	semaphore.yaml (+ semaphore-bootstrap-secret.sops.yaml)	app + secret	→ semaphore-db (w0)
1	gitea-actions.yaml (+ gitea-actions-runner-secret.sops.yaml)	CI runner + secret	EXTRA_UNITS: anchor gitea (w0), above → one wave up
2	seerr.yaml	app	→ sonarr (w1), radarr (w1)

Catalog services not yet onboarded (no committed manifest) are computed but not placed: keycloak (w1, → keycloak-db w0) and vm-demo (w0). They join a wave automatically when onboarded.

Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = 130 resources, byte-for-byte the same set the monolithic apps Kustomization built before the split (validated with kubectl kustomize).

3. Flux Kustomization topology

platform (controllers/CRDs)
   │ healthy
   ▼
platform-config (cluster CRs)
   │ healthy
   ▼
apps-wave-0  → ./apps/wave-0   dependsOn platform-config   wait:true prune:true   (leaves + CNPG DBs)
   │ healthy
   ▼
apps-wave-1  → ./apps/wave-1   dependsOn apps-wave-0        wait:true prune:true   (arr, DB-backed apps, gitea-actions)
   │ healthy
   ▼
apps-wave-2  → ./apps/wave-2   dependsOn apps-wave-1        wait:true prune:true   (seerr)

Every wave Kustomization is identical to the retired apps one except name, path, and dependsOn: same interval/retryInterval/timeout, prune: true, wait: true, postBuild.substituteFrom cluster-vars, and the same SOPS decryption (added per-environment in clusters/dev/kustomization.yaml — one inlined patch per apps-wave-<N>, referencing the out-of-band sops-age Secret by name only, exactly as the single apps Kustomization did).

Failure isolation gained: a crashloop in seerr (wave 2) no longer blocks apps-wave-0 or apps-wave-1, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its downstream), never wave 0. Before, any one crashloop stalled the whole tier.

4. What render.py now does

• compute_waves(sds) — longest-path wave per node (services + DB leaf nodes); cycle-safe.
• plan_apps_waves(sds, present_files) — maps the committed app file set → {wave: [files]} by naming convention (<svc>, <svc>-bootstrap-secret, <svc>-db, <svc>-db-app) plus EXTRA_UNITS; only files actually present are placed (not-yet-onboarded SDs are skipped).
• --emit-waves — (re)generates every apps/wave-<N>/kustomization.yaml and clusters/base/apps-wave-<N>.yaml. Idempotent; never rewrites a service manifest (only placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched.
• --print-waves — prints the node→wave table.
• -o <dir> --wave-layout — future onboarding writes <dir>/wave-<N>/<svc>.yaml directly.
• Self-test extended (--selftest): longest-path depth, CNPG-DB-below-consumer, cycle detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait).

Onboarding a new service now: add catalog/<svc>.yaml, render it (render.py catalog/<svc>.yaml -o apps --wave-layout [--no-image-automation]), then render.py --emit-waves. If it lands in an existing wave, no clusters/base edit is needed. If it creates a new deepest wave, add one line to clusters/base/kustomization.yaml and one decryption patch to clusters/dev/kustomization.yaml (both flagged in-file).

---

5. CUTOVER RUNBOOK — Flux resource adoption without data loss

Read this before merge. Moving a manifest from the apps Kustomization to an apps-wave-<N> Kustomization is a Flux ownership transfer (adoption). Done wrong, one Kustomization can prune (delete) a resource another is adopting — and for the CNPG databases that is data loss.

5.1 Why the naive "just merge" is dangerous

Flux marks every resource it applies with kustomize.toolkit.fluxcd.io/name + …/namespace, and with prune: true it garbage-collects any owned resource that leaves its build. Two hazards on a plain merge of this PR:

1. The live apps Kustomization has prune: true. This PR removes apps.yaml from clusters/base. The root Kustomization (flux-system, ./clusters/dev, prune: true) will then delete the apps Kustomization object. Deleting a Flux Kustomization that has prune: true runs its finalizer, which garbage-collects every resource it owns — all ~130 app resources, including the nextcloud-db / semaphore-db CNPG Clusters. Deleting a CNPG Cluster cascade-deletes its PVCs → the databases are destroyed, even though the new waves were about to adopt them.
2. Ordering race. Even setting that aside, if the old Kustomization's prune and the new waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart for stateless apps, and unacceptable for the stateful DBs.

The fix is to make the old Kustomization release its resources (orphan-alive, no GC) so the waves adopt them by identity — never a delete/recreate window.

5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime)

Split the change into two PRs so the release happens before the split, entirely through git.

Pre-flight (no changes):

• flux get kustomizations → apps is Ready. Cluster healthy.
• Record the stateful state you must not lose: kubectl get cluster -A, kubectl get pvc -n nextcloud-db, kubectl get pvc -n semaphore-db (note names + AGE). Take a Longhorn/Velero backup of the nextcloud-db and semaphore-db namespaces (defense in depth).

PR 1 — neutralize GC on the old apps Kustomization (one-line change):

• Edit clusters/base/apps.yaml: prune: true → prune: false. Merge.
• Wait until flux get kustomization apps reconciles with the new spec (kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}' → false).
• Effect: when apps is later deleted, its finalizer will not garbage-collect its resources — they survive as orphaned-but-alive objects.

PR 2 — this wave-split PR. On merge, the root Kustomization reconciles ./clusters/dev:

• apps is gone from the build → root prunes the apps Kustomization object. Because apps now has prune: false (from PR 1), no app resources are deleted; they are orphaned, still running.
• apps-wave-0/1/2 are created. Each does a server-side apply of the same, byte-identical manifests, adopting the live objects (matched by GVK + namespace + name) and relabeling kustomize.toolkit.fluxcd.io/name to its wave. Because content is unchanged, SSA is a no-op on spec — no pod restart, no PVC churn, no DB re-init. Same field manager (kustomize-controller) → no ownership conflict.
• Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and corrupt the Secret.

Verify:

flux get kustomizations                                   # apps-wave-0/1/2 Ready; apps absent
kubectl get cluster -A                                    # DBs present, ORIGINAL age (not recreated)
kubectl get pvc -n nextcloud-db -n semaphore-db           # same PVCs, same age
kubectl get deploy -A -o wide                             # apps not restarted (AGE/restarts unchanged)
kubectl get <any-app-deploy> -n <ns> \
  -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}'   # == apps-wave-<N>

5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1)

If you would rather merge this PR alone, perform PR 1's effect out-of-band first, then merge:

flux -n flux-system suspend kustomization apps                                   # stop it racing the waves
kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}'

Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of which Kustomization ends up owning them during the window):

kubectl -n nextcloud-db  annotate cluster/nextcloud-db  kustomize.toolkit.fluxcd.io/prune=disabled
kubectl -n semaphore-db  annotate cluster/semaphore-db  kustomize.toolkit.fluxcd.io/prune=disabled

Then merge this PR and verify as in 5.2. Cleanup once the waves own the DBs (restore normal prune semantics):

kubectl -n nextcloud-db  annotate cluster/nextcloud-db  kustomize.toolkit.fluxcd.io/prune-
kubectl -n semaphore-db  annotate cluster/semaphore-db  kustomize.toolkit.fluxcd.io/prune-

5.4 Rollback

Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the resources are still alive (orphaned). flux suspend the waves, revert the merge (re-creating apps, which re-adopts by identity), and investigate. No data is lost because the cutover never issues a delete.

5.5 Why adoption is safe here (summary of the invariant)

• The old owner releases without GC (prune: false before it is deleted) → no delete.
• The new owners adopt by identity via SSA on byte-identical manifests → no recreate.
• There is no window in which a resource is owned by nothing and pruned: it is continuously alive, transitioning label ownership from apps → apps-wave-<N>.

---

6. Risks / open items for the PM

1. The GC hazard above is the whole ball game. A plain merge of this PR onto a live cluster whose apps Kustomization still has prune: true will destroy the CNPG databases. §5.2 / §5.3 must be followed. (In dev this has never been applied — prototype only.)
2. prune blast radius within a wave is unchanged. Splitting reduces cross-wave coupling, but a wave is still wait: true — a crashloop in gitea-actions (wave 1) will block wave 2 (seerr) until it recovers or its timeout elapses. That is the intended linear-wave trade-off; if the PM wants gitea-actions (flaky privileged DinD) to not gate seerr, consider giving CI plumbing its own non-gating Kustomization branch (future work, not in this PR).
3. New-deepest-wave onboarding needs two one-line manual edits (clusters/base/kustomization.yaml
   • a clusters/dev decryption patch). render.py --emit-waves generates the CR but does not edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing in an existing wave is fully automatic.
4. Image automation. platform/image-automation/image-update-automation.yaml scans path: ./apps (recursive) for $imagepolicy setters — apps/wave-<N>/ is under ./apps, so per-service image markers are still covered. No change needed; noted so it isn't a surprise.
5. Decryption drift. Each wave must carry the SOPS decryption patch (done in this PR for waves 0–2). If a future wave is added and the clusters/dev patch is forgotten, that wave will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and §4 call this out; a lint check that "every apps-wave- has a decryption patch" would be a good hardening follow-up.
6. EXTRA_UNITS is a small hand-maintained table (only gitea-actions today). Any future non-catalog app-tier unit must be added there or it will not be placed into a wave.