home-network-k3s-impl/DESIGN-waves.md

# Wave-based app-tier delivery (PROTOTYPE — for PM review)

**Problem.** Today one Flux Kustomization `apps` (clusters/base/apps.yaml, `dependsOn:
platform-config`, `wait: true`, `prune: true`) reconciles all of `apps/` — ~10 services + CNPG
databases + SOPS secrets. Because `wait: true` health-gates the **entire** set, a single
crashlooping workload blocks **all** apps and every new revision (this has bitten twice).

**Solution.** Split the tier into dependency-ordered **waves**. Each wave is its own Flux
Kustomization `apps-wave-<N>` that `dependsOn` wave N-1 (wave 0 `dependsOn` `platform-config`),
with `wait: true` + `prune: true` and `path: ./apps/wave-<N>`. `wait: true` now health-gates
**only that wave**, so a crashloop isolates to its own wave and everything downstream, while
services **within** a wave still reconcile in parallel. Wave assignment is computed by
`catalog/render.py` from the declared dependency graph, so future onboarding auto-places a
service in the correct wave.

---

## 1. Wave assignment algorithm

`wave(node)` = **longest dependency-path depth** over the catalog `dependencies:` edges
(service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node
is wave ≥ k+1. Implemented in `catalog/render.py :: compute_waves()`.

**Co-located CNPG database rule.** A service `<svc>` that declares a dependency `to: <svc>-db`
(e.g. `nextcloud` → `nextcloud-db`, `semaphore` → `semaphore-db`) has a hard DB prerequisite.
The `<svc>-db` node carries **no** ServiceDefinition, so it is a **leaf** (wave 0); because the
consumer declares a dependency on it, the consumer lands in a strictly higher wave. **The
"DB strictly below its consumer" guarantee therefore falls straight out of the longest-path
rule — no special case.** The DB manifest (`<svc>-db.yaml`) and its CNPG `initdb` owner secret
(`<svc>-db-app.sops.yaml`) are placed in the DB's (lower) wave; the app manifest and its
`<svc>-bootstrap-secret.sops.yaml` are placed in the consumer's wave.

**Non-catalog units.** A few app-tier files carry no ServiceDefinition (`compute_waves` can't
see them). They are anchored explicitly in `render.py :: EXTRA_UNITS`. Today the only one is
**gitea-actions** (the in-cluster `act_runner` + its DinD sidecar), anchored to `gitea` with
`above: true` — the runner registers against and clones from Gitea, so Gitea must be healthy
first → it lands one wave **above** gitea.

Cycle detection: `compute_waves` raises on any dependency cycle.

## 2. Computed WAVE TABLE

Verify with `python3 catalog/render.py --print-waves` (nodes) and `--emit-waves` (file plan).

| Wave | File | Kind | Reason (dependency path) |
|---|---|---|---|
| **0** | `pihole.yaml` | app | no dependencies (leaf) |
| 0 | `gitea.yaml` (+ `gitea-bootstrap-secret.sops.yaml`) | app + secret | no dependencies (leaf) |
| 0 | `jackett.yaml` | app | no dependencies (leaf) |
| 0 | `plex.yaml` | app | no dependencies (leaf) |
| 0 | `qbittorrent.yaml` | app | no dependencies (leaf) |
| 0 | `nextcloud-db.yaml` (+ `nextcloud-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `nextcloud` |
| 0 | `semaphore-db.yaml` (+ `semaphore-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `semaphore` |
| **1** | `sonarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
| 1 | `radarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
| 1 | `nextcloud.yaml` (+ `nextcloud-bootstrap-secret.sops.yaml`) | app + secret | → `nextcloud-db` (w0) |
| 1 | `semaphore.yaml` (+ `semaphore-bootstrap-secret.sops.yaml`) | app + secret | → `semaphore-db` (w0) |
| 1 | `gitea-actions.yaml` (+ `gitea-actions-runner-secret.sops.yaml`) | CI runner + secret | EXTRA_UNITS: anchor `gitea` (w0), `above` → one wave up |
| **2** | `seerr.yaml` | app | → `sonarr` (w1), `radarr` (w1) |

Catalog services **not yet onboarded** (no committed manifest) are computed but **not placed**:
`keycloak` (w1, → `keycloak-db` w0) and `vm-demo` (w0). They join a wave automatically when
onboarded.

Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = **130 resources**, byte-for-byte the
same set the monolithic `apps` Kustomization built before the split (validated with
`kubectl kustomize`).

## 3. Flux Kustomization topology

```
platform (controllers/CRDs)
   │ healthy
   ▼
platform-config (cluster CRs)
   │ healthy
   ▼
apps-wave-0  → ./apps/wave-0   dependsOn platform-config   wait:true prune:true   (leaves + CNPG DBs)
   │ healthy
   ▼
apps-wave-1  → ./apps/wave-1   dependsOn apps-wave-0        wait:true prune:true   (arr, DB-backed apps, gitea-actions)
   │ healthy
   ▼
apps-wave-2  → ./apps/wave-2   dependsOn apps-wave-1        wait:true prune:true   (seerr)
```

Every wave Kustomization is identical to the retired `apps` one except `name`, `path`, and
`dependsOn`: same `interval/retryInterval/timeout`, `prune: true`, `wait: true`,
`postBuild.substituteFrom cluster-vars`, and the **same** SOPS decryption (added per-environment
in `clusters/dev/kustomization.yaml` — one inlined patch per `apps-wave-<N>`, referencing the
out-of-band `sops-age` Secret by name only, exactly as the single `apps` Kustomization did).

**Failure isolation gained:** a crashloop in `seerr` (wave 2) no longer blocks `apps-wave-0`
or `apps-wave-1`, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its
downstream), never wave 0. Before, any one crashloop stalled the whole tier.

## 4. What `render.py` now does

- `compute_waves(sds)` — longest-path wave per node (services + DB leaf nodes); cycle-safe.
- `plan_apps_waves(sds, present_files)` — maps the committed app file set → `{wave: [files]}`
  by naming convention (`<svc>`, `<svc>-bootstrap-secret`, `<svc>-db`, `<svc>-db-app`) plus
  `EXTRA_UNITS`; only files actually present are placed (not-yet-onboarded SDs are skipped).
- `--emit-waves` — (re)generates every `apps/wave-<N>/kustomization.yaml` and
  `clusters/base/apps-wave-<N>.yaml`. Idempotent; **never rewrites a service manifest** (only
  placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched.
- `--print-waves` — prints the node→wave table.
- `-o <dir> --wave-layout` — future onboarding writes `<dir>/wave-<N>/<svc>.yaml` directly.
- Self-test extended (`--selftest`): longest-path depth, CNPG-DB-below-consumer, cycle
  detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait).

**Onboarding a new service now:** add `catalog/<svc>.yaml`, render it
(`render.py catalog/<svc>.yaml -o apps --wave-layout [--no-image-automation]`), then
`render.py --emit-waves`. If it lands in an existing wave, no clusters/base edit is needed. If
it creates a **new deepest** wave, add one line to `clusters/base/kustomization.yaml` and one
decryption patch to `clusters/dev/kustomization.yaml` (both flagged in-file).

---

## 5. CUTOVER RUNBOOK — Flux resource adoption without data loss

> **Read this before merge.** Moving a manifest from the `apps` Kustomization to an
> `apps-wave-<N>` Kustomization is a Flux **ownership transfer** (adoption). Done wrong, one
> Kustomization can **prune** (delete) a resource another is adopting — and for the CNPG
> databases that is **data loss**.

### 5.1 Why the naive "just merge" is dangerous

Flux marks every resource it applies with `kustomize.toolkit.fluxcd.io/name` +
`…/namespace`, and with `prune: true` it **garbage-collects** any owned resource that leaves
its build. Two hazards on a plain merge of this PR:

1. **The live `apps` Kustomization has `prune: true`.** This PR removes `apps.yaml` from
   `clusters/base`. The root Kustomization (`flux-system`, `./clusters/dev`, `prune: true`)
   will then **delete the `apps` Kustomization object**. Deleting a Flux Kustomization that
   has `prune: true` runs its finalizer, which **garbage-collects every resource it owns** —
   all ~130 app resources, **including the `nextcloud-db` / `semaphore-db` CNPG Clusters**.
   Deleting a CNPG `Cluster` cascade-deletes its PVCs → **the databases are destroyed**, even
   though the new waves were about to adopt them.
2. **Ordering race.** Even setting that aside, if the old Kustomization's prune and the new
   waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart
   for stateless apps, and unacceptable for the stateful DBs.

The fix is to make the old Kustomization **release** its resources (orphan-alive, no GC) so the
waves **adopt** them by identity — never a delete/recreate window.

### 5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime)

Split the change into two PRs so the release happens before the split, entirely through git.

**Pre-flight (no changes):**
- `flux get kustomizations` → `apps` is `Ready`. Cluster healthy.
- Record the stateful state you must not lose:
  `kubectl get cluster -A`, `kubectl get pvc -n nextcloud-db`, `kubectl get pvc -n semaphore-db`
  (note names + `AGE`). Take a Longhorn/Velero backup of the `nextcloud-db` and `semaphore-db`
  namespaces (defense in depth).

**PR 1 — neutralize GC on the old `apps` Kustomization (one-line change):**
- Edit `clusters/base/apps.yaml`: `prune: true` → `prune: false`. Merge.
- Wait until `flux get kustomization apps` reconciles with the new spec
  (`kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}'` → `false`).
- Effect: when `apps` is later deleted, its finalizer will **not** garbage-collect its
  resources — they survive as orphaned-but-alive objects.

**PR 2 — this wave-split PR.** On merge, the root Kustomization reconciles `./clusters/dev`:
- `apps` is gone from the build → root prunes the **`apps` Kustomization object**. Because
  `apps` now has `prune: false` (from PR 1), **no app resources are deleted**; they are
  orphaned, still running.
- `apps-wave-0/1/2` are created. Each does a server-side apply of the **same, byte-identical**
  manifests, **adopting** the live objects (matched by GVK + namespace + name) and relabeling
  `kustomize.toolkit.fluxcd.io/name` to its wave. Because content is unchanged, SSA is a no-op
  on `spec` — **no pod restart, no PVC churn, no DB re-init**. Same field manager
  (`kustomize-controller`) → no ownership conflict.
- Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted
  and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and
  corrupt the Secret.

**Verify:**
```
flux get kustomizations                                   # apps-wave-0/1/2 Ready; apps absent
kubectl get cluster -A                                    # DBs present, ORIGINAL age (not recreated)
kubectl get pvc -n nextcloud-db -n semaphore-db           # same PVCs, same age
kubectl get deploy -A -o wide                             # apps not restarted (AGE/restarts unchanged)
kubectl get <any-app-deploy> -n <ns> \
  -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}'   # == apps-wave-<N>
```

### 5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1)

If you would rather merge this PR alone, perform PR 1's effect out-of-band **first**, then
merge:
```
flux -n flux-system suspend kustomization apps                                   # stop it racing the waves
kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}'
```
Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of
which Kustomization ends up owning them during the window):
```
kubectl -n nextcloud-db  annotate cluster/nextcloud-db  kustomize.toolkit.fluxcd.io/prune=disabled
kubectl -n semaphore-db  annotate cluster/semaphore-db  kustomize.toolkit.fluxcd.io/prune=disabled
```
Then merge this PR and verify as in 5.2. **Cleanup once the waves own the DBs** (restore normal
prune semantics):
```
kubectl -n nextcloud-db  annotate cluster/nextcloud-db  kustomize.toolkit.fluxcd.io/prune-
kubectl -n semaphore-db  annotate cluster/semaphore-db  kustomize.toolkit.fluxcd.io/prune-
```

### 5.4 Rollback

Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the
resources are still alive (orphaned). `flux suspend` the waves, revert the merge (re-creating
`apps`, which re-adopts by identity), and investigate. No data is lost because the cutover
never issues a delete.

### 5.5 Why adoption is safe here (summary of the invariant)

- The old owner **releases without GC** (`prune: false` before it is deleted) → no delete.
- The new owners **adopt by identity** via SSA on byte-identical manifests → no recreate.
- There is **no window** in which a resource is owned by nothing *and* pruned: it is
  continuously alive, transitioning label ownership from `apps` → `apps-wave-<N>`.

---

## 6. Risks / open items for the PM

1. **The GC hazard above is the whole ball game.** A plain merge of this PR onto a live cluster
   whose `apps` Kustomization still has `prune: true` will destroy the CNPG databases. §5.2 /
   §5.3 must be followed. (In dev this has never been applied — prototype only.)
2. **`prune` blast radius within a wave is unchanged.** Splitting reduces cross-wave coupling,
   but a wave is still `wait: true` — a crashloop in `gitea-actions` (wave 1) will block wave 2
   (`seerr`) until it recovers or its `timeout` elapses. That is the intended linear-wave
   trade-off; if the PM wants `gitea-actions` (flaky privileged DinD) to not gate `seerr`,
   consider giving CI plumbing its own non-gating Kustomization branch (future work, not in
   this PR).
3. **New-deepest-wave onboarding needs two one-line manual edits** (`clusters/base/kustomization.yaml`
   + a `clusters/dev` decryption patch). `render.py --emit-waves` generates the CR but does not
   edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing
   in an existing wave is fully automatic.
4. **Image automation.** `platform/image-automation/image-update-automation.yaml` scans
   `path: ./apps` (recursive) for `$imagepolicy` setters — `apps/wave-<N>/` is under `./apps`,
   so per-service image markers are still covered. No change needed; noted so it isn't a
   surprise.
5. **Decryption drift.** Each wave must carry the SOPS decryption patch (done in this PR for
   waves 0–2). If a future wave is added and the `clusters/dev` patch is forgotten, that wave
   will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and
   §4 call this out; a lint check that "every apps-wave-<N> has a decryption patch" would be a
   good hardening follow-up.
6. **`EXTRA_UNITS` is a small hand-maintained table** (only `gitea-actions` today). Any future
   non-catalog app-tier unit must be added there or it will not be placed into a wave.