From 36a1a4e1f9475f1085e8c9cd0ff9021b20e290d4 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 26 Aug 2026 12:19:00 -0500 Subject: [PATCH] refactor(apps): split monolithic apps Kustomization into dependency waves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the single `apps` Flux Kustomization with dependency-ordered waves (apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream instead of stalling the entire app tier. Parallel within a wave, ordered across. - catalog/render.py: compute waves by longest dependency-path depth over the ServiceDefinition `dependencies:` edges (a `-db` is a leaf, so its consumer lands strictly higher — the "DB below consumer" rule falls out of longest-path, no special case). `--emit-waves` generates the per-wave kustomizations + Flux CRs; `--wave-layout` places renders into the right dir. wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db wave 1: sonarr radarr nextcloud semaphore gitea-actions wave 2: seerr - apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact). - clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each). - clusters/dev: one SOPS decryption patch per wave (load-bearing). - DESIGN-waves.md: wave table, topology, cutover runbook. Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that object here does not GC its resources; the wave Kustomizations SSA-adopt the byte-identical manifests in place (no restart, no PVC churn). Co-Authored-By: Claude Opus 4.8 --- DESIGN-waves.md | 249 ++++++++++++ apps/README.md | 15 +- apps/kustomization.yaml | 41 -- .../gitea-bootstrap-secret.sops.yaml | 0 apps/{ => wave-0}/gitea.yaml | 0 apps/{ => wave-0}/jackett.yaml | 0 apps/wave-0/kustomization.yaml | 18 + apps/{ => wave-0}/nextcloud-db-app.sops.yaml | 0 apps/{ => wave-0}/nextcloud-db.yaml | 0 apps/{ => wave-0}/pihole.yaml | 0 apps/{ => wave-0}/plex.yaml | 0 apps/{ => wave-0}/qbittorrent.yaml | 0 apps/{ => wave-0}/semaphore-db-app.sops.yaml | 0 apps/{ => wave-0}/semaphore-db.yaml | 0 .../gitea-actions-runner-secret.sops.yaml | 0 apps/{ => wave-1}/gitea-actions.yaml | 0 apps/wave-1/kustomization.yaml | 16 + .../nextcloud-bootstrap-secret.sops.yaml | 0 apps/{ => wave-1}/nextcloud.yaml | 0 apps/{ => wave-1}/radarr.yaml | 0 .../semaphore-bootstrap-secret.sops.yaml | 0 apps/{ => wave-1}/semaphore.yaml | 0 apps/{ => wave-1}/sonarr.yaml | 0 apps/wave-2/kustomization.yaml | 9 + apps/{ => wave-2}/seerr.yaml | 0 .../flux/patches/apps-sops-decryption.yaml | 19 +- catalog/render.py | 355 +++++++++++++++++- clusters/README.md | 18 +- clusters/base/apps-wave-0.yaml | 26 ++ clusters/base/apps-wave-1.yaml | 26 ++ clusters/base/apps-wave-2.yaml | 26 ++ clusters/base/apps.yaml | 31 -- clusters/base/kustomization.yaml | 9 +- clusters/dev/kustomization.yaml | 28 +- 34 files changed, 793 insertions(+), 93 deletions(-) create mode 100644 DESIGN-waves.md delete mode 100644 apps/kustomization.yaml rename apps/{ => wave-0}/gitea-bootstrap-secret.sops.yaml (100%) rename apps/{ => wave-0}/gitea.yaml (100%) rename apps/{ => wave-0}/jackett.yaml (100%) create mode 100644 apps/wave-0/kustomization.yaml rename apps/{ => wave-0}/nextcloud-db-app.sops.yaml (100%) rename apps/{ => wave-0}/nextcloud-db.yaml (100%) rename apps/{ => wave-0}/pihole.yaml (100%) rename apps/{ => wave-0}/plex.yaml (100%) rename apps/{ => wave-0}/qbittorrent.yaml (100%) rename apps/{ => wave-0}/semaphore-db-app.sops.yaml (100%) rename apps/{ => wave-0}/semaphore-db.yaml (100%) rename apps/{ => wave-1}/gitea-actions-runner-secret.sops.yaml (100%) rename apps/{ => wave-1}/gitea-actions.yaml (100%) create mode 100644 apps/wave-1/kustomization.yaml rename apps/{ => wave-1}/nextcloud-bootstrap-secret.sops.yaml (100%) rename apps/{ => wave-1}/nextcloud.yaml (100%) rename apps/{ => wave-1}/radarr.yaml (100%) rename apps/{ => wave-1}/semaphore-bootstrap-secret.sops.yaml (100%) rename apps/{ => wave-1}/semaphore.yaml (100%) rename apps/{ => wave-1}/sonarr.yaml (100%) create mode 100644 apps/wave-2/kustomization.yaml rename apps/{ => wave-2}/seerr.yaml (100%) create mode 100644 clusters/base/apps-wave-0.yaml create mode 100644 clusters/base/apps-wave-1.yaml create mode 100644 clusters/base/apps-wave-2.yaml delete mode 100644 clusters/base/apps.yaml diff --git a/DESIGN-waves.md b/DESIGN-waves.md new file mode 100644 index 0000000..837f6e6 --- /dev/null +++ b/DESIGN-waves.md @@ -0,0 +1,249 @@ +# Wave-based app-tier delivery (PROTOTYPE — for PM review) + +**Problem.** Today one Flux Kustomization `apps` (clusters/base/apps.yaml, `dependsOn: +platform-config`, `wait: true`, `prune: true`) reconciles all of `apps/` — ~10 services + CNPG +databases + SOPS secrets. Because `wait: true` health-gates the **entire** set, a single +crashlooping workload blocks **all** apps and every new revision (this has bitten twice). + +**Solution.** Split the tier into dependency-ordered **waves**. Each wave is its own Flux +Kustomization `apps-wave-` that `dependsOn` wave N-1 (wave 0 `dependsOn` `platform-config`), +with `wait: true` + `prune: true` and `path: ./apps/wave-`. `wait: true` now health-gates +**only that wave**, so a crashloop isolates to its own wave and everything downstream, while +services **within** a wave still reconcile in parallel. Wave assignment is computed by +`catalog/render.py` from the declared dependency graph, so future onboarding auto-places a +service in the correct wave. + +--- + +## 1. Wave assignment algorithm + +`wave(node)` = **longest dependency-path depth** over the catalog `dependencies:` edges +(service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node +is wave ≥ k+1. Implemented in `catalog/render.py :: compute_waves()`. + +**Co-located CNPG database rule.** A service `` that declares a dependency `to: -db` +(e.g. `nextcloud` → `nextcloud-db`, `semaphore` → `semaphore-db`) has a hard DB prerequisite. +The `-db` node carries **no** ServiceDefinition, so it is a **leaf** (wave 0); because the +consumer declares a dependency on it, the consumer lands in a strictly higher wave. **The +"DB strictly below its consumer" guarantee therefore falls straight out of the longest-path +rule — no special case.** The DB manifest (`-db.yaml`) and its CNPG `initdb` owner secret +(`-db-app.sops.yaml`) are placed in the DB's (lower) wave; the app manifest and its +`-bootstrap-secret.sops.yaml` are placed in the consumer's wave. + +**Non-catalog units.** A few app-tier files carry no ServiceDefinition (`compute_waves` can't +see them). They are anchored explicitly in `render.py :: EXTRA_UNITS`. Today the only one is +**gitea-actions** (the in-cluster `act_runner` + its DinD sidecar), anchored to `gitea` with +`above: true` — the runner registers against and clones from Gitea, so Gitea must be healthy +first → it lands one wave **above** gitea. + +Cycle detection: `compute_waves` raises on any dependency cycle. + +## 2. Computed WAVE TABLE + +Verify with `python3 catalog/render.py --print-waves` (nodes) and `--emit-waves` (file plan). + +| Wave | File | Kind | Reason (dependency path) | +|---|---|---|---| +| **0** | `pihole.yaml` | app | no dependencies (leaf) | +| 0 | `gitea.yaml` (+ `gitea-bootstrap-secret.sops.yaml`) | app + secret | no dependencies (leaf) | +| 0 | `jackett.yaml` | app | no dependencies (leaf) | +| 0 | `plex.yaml` | app | no dependencies (leaf) | +| 0 | `qbittorrent.yaml` | app | no dependencies (leaf) | +| 0 | `nextcloud-db.yaml` (+ `nextcloud-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `nextcloud` | +| 0 | `semaphore-db.yaml` (+ `semaphore-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `semaphore` | +| **1** | `sonarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) | +| 1 | `radarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) | +| 1 | `nextcloud.yaml` (+ `nextcloud-bootstrap-secret.sops.yaml`) | app + secret | → `nextcloud-db` (w0) | +| 1 | `semaphore.yaml` (+ `semaphore-bootstrap-secret.sops.yaml`) | app + secret | → `semaphore-db` (w0) | +| 1 | `gitea-actions.yaml` (+ `gitea-actions-runner-secret.sops.yaml`) | CI runner + secret | EXTRA_UNITS: anchor `gitea` (w0), `above` → one wave up | +| **2** | `seerr.yaml` | app | → `sonarr` (w1), `radarr` (w1) | + +Catalog services **not yet onboarded** (no committed manifest) are computed but **not placed**: +`keycloak` (w1, → `keycloak-db` w0) and `vm-demo` (w0). They join a wave automatically when +onboarded. + +Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = **130 resources**, byte-for-byte the +same set the monolithic `apps` Kustomization built before the split (validated with +`kubectl kustomize`). + +## 3. Flux Kustomization topology + +``` +platform (controllers/CRDs) + │ healthy + ▼ +platform-config (cluster CRs) + │ healthy + ▼ +apps-wave-0 → ./apps/wave-0 dependsOn platform-config wait:true prune:true (leaves + CNPG DBs) + │ healthy + ▼ +apps-wave-1 → ./apps/wave-1 dependsOn apps-wave-0 wait:true prune:true (arr, DB-backed apps, gitea-actions) + │ healthy + ▼ +apps-wave-2 → ./apps/wave-2 dependsOn apps-wave-1 wait:true prune:true (seerr) +``` + +Every wave Kustomization is identical to the retired `apps` one except `name`, `path`, and +`dependsOn`: same `interval/retryInterval/timeout`, `prune: true`, `wait: true`, +`postBuild.substituteFrom cluster-vars`, and the **same** SOPS decryption (added per-environment +in `clusters/dev/kustomization.yaml` — one inlined patch per `apps-wave-`, referencing the +out-of-band `sops-age` Secret by name only, exactly as the single `apps` Kustomization did). + +**Failure isolation gained:** a crashloop in `seerr` (wave 2) no longer blocks `apps-wave-0` +or `apps-wave-1`, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its +downstream), never wave 0. Before, any one crashloop stalled the whole tier. + +## 4. What `render.py` now does + +- `compute_waves(sds)` — longest-path wave per node (services + DB leaf nodes); cycle-safe. +- `plan_apps_waves(sds, present_files)` — maps the committed app file set → `{wave: [files]}` + by naming convention (``, `-bootstrap-secret`, `-db`, `-db-app`) plus + `EXTRA_UNITS`; only files actually present are placed (not-yet-onboarded SDs are skipped). +- `--emit-waves` — (re)generates every `apps/wave-/kustomization.yaml` and + `clusters/base/apps-wave-.yaml`. Idempotent; **never rewrites a service manifest** (only + placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched. +- `--print-waves` — prints the node→wave table. +- `-o --wave-layout` — future onboarding writes `/wave-/.yaml` directly. +- Self-test extended (`--selftest`): longest-path depth, CNPG-DB-below-consumer, cycle + detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait). + +**Onboarding a new service now:** add `catalog/.yaml`, render it +(`render.py catalog/.yaml -o apps --wave-layout [--no-image-automation]`), then +`render.py --emit-waves`. If it lands in an existing wave, no clusters/base edit is needed. If +it creates a **new deepest** wave, add one line to `clusters/base/kustomization.yaml` and one +decryption patch to `clusters/dev/kustomization.yaml` (both flagged in-file). + +--- + +## 5. CUTOVER RUNBOOK — Flux resource adoption without data loss + +> **Read this before merge.** Moving a manifest from the `apps` Kustomization to an +> `apps-wave-` Kustomization is a Flux **ownership transfer** (adoption). Done wrong, one +> Kustomization can **prune** (delete) a resource another is adopting — and for the CNPG +> databases that is **data loss**. + +### 5.1 Why the naive "just merge" is dangerous + +Flux marks every resource it applies with `kustomize.toolkit.fluxcd.io/name` + +`…/namespace`, and with `prune: true` it **garbage-collects** any owned resource that leaves +its build. Two hazards on a plain merge of this PR: + +1. **The live `apps` Kustomization has `prune: true`.** This PR removes `apps.yaml` from + `clusters/base`. The root Kustomization (`flux-system`, `./clusters/dev`, `prune: true`) + will then **delete the `apps` Kustomization object**. Deleting a Flux Kustomization that + has `prune: true` runs its finalizer, which **garbage-collects every resource it owns** — + all ~130 app resources, **including the `nextcloud-db` / `semaphore-db` CNPG Clusters**. + Deleting a CNPG `Cluster` cascade-deletes its PVCs → **the databases are destroyed**, even + though the new waves were about to adopt them. +2. **Ordering race.** Even setting that aside, if the old Kustomization's prune and the new + waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart + for stateless apps, and unacceptable for the stateful DBs. + +The fix is to make the old Kustomization **release** its resources (orphan-alive, no GC) so the +waves **adopt** them by identity — never a delete/recreate window. + +### 5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime) + +Split the change into two PRs so the release happens before the split, entirely through git. + +**Pre-flight (no changes):** +- `flux get kustomizations` → `apps` is `Ready`. Cluster healthy. +- Record the stateful state you must not lose: + `kubectl get cluster -A`, `kubectl get pvc -n nextcloud-db`, `kubectl get pvc -n semaphore-db` + (note names + `AGE`). Take a Longhorn/Velero backup of the `nextcloud-db` and `semaphore-db` + namespaces (defense in depth). + +**PR 1 — neutralize GC on the old `apps` Kustomization (one-line change):** +- Edit `clusters/base/apps.yaml`: `prune: true` → `prune: false`. Merge. +- Wait until `flux get kustomization apps` reconciles with the new spec + (`kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}'` → `false`). +- Effect: when `apps` is later deleted, its finalizer will **not** garbage-collect its + resources — they survive as orphaned-but-alive objects. + +**PR 2 — this wave-split PR.** On merge, the root Kustomization reconciles `./clusters/dev`: +- `apps` is gone from the build → root prunes the **`apps` Kustomization object**. Because + `apps` now has `prune: false` (from PR 1), **no app resources are deleted**; they are + orphaned, still running. +- `apps-wave-0/1/2` are created. Each does a server-side apply of the **same, byte-identical** + manifests, **adopting** the live objects (matched by GVK + namespace + name) and relabeling + `kustomize.toolkit.fluxcd.io/name` to its wave. Because content is unchanged, SSA is a no-op + on `spec` — **no pod restart, no PVC churn, no DB re-init**. Same field manager + (`kustomize-controller`) → no ownership conflict. +- Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted + and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and + corrupt the Secret. + +**Verify:** +``` +flux get kustomizations # apps-wave-0/1/2 Ready; apps absent +kubectl get cluster -A # DBs present, ORIGINAL age (not recreated) +kubectl get pvc -n nextcloud-db -n semaphore-db # same PVCs, same age +kubectl get deploy -A -o wide # apps not restarted (AGE/restarts unchanged) +kubectl get -n \ + -o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}' # == apps-wave- +``` + +### 5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1) + +If you would rather merge this PR alone, perform PR 1's effect out-of-band **first**, then +merge: +``` +flux -n flux-system suspend kustomization apps # stop it racing the waves +kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}' +``` +Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of +which Kustomization ends up owning them during the window): +``` +kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune=disabled +kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune=disabled +``` +Then merge this PR and verify as in 5.2. **Cleanup once the waves own the DBs** (restore normal +prune semantics): +``` +kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune- +kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune- +``` + +### 5.4 Rollback + +Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the +resources are still alive (orphaned). `flux suspend` the waves, revert the merge (re-creating +`apps`, which re-adopts by identity), and investigate. No data is lost because the cutover +never issues a delete. + +### 5.5 Why adoption is safe here (summary of the invariant) + +- The old owner **releases without GC** (`prune: false` before it is deleted) → no delete. +- The new owners **adopt by identity** via SSA on byte-identical manifests → no recreate. +- There is **no window** in which a resource is owned by nothing *and* pruned: it is + continuously alive, transitioning label ownership from `apps` → `apps-wave-`. + +--- + +## 6. Risks / open items for the PM + +1. **The GC hazard above is the whole ball game.** A plain merge of this PR onto a live cluster + whose `apps` Kustomization still has `prune: true` will destroy the CNPG databases. §5.2 / + §5.3 must be followed. (In dev this has never been applied — prototype only.) +2. **`prune` blast radius within a wave is unchanged.** Splitting reduces cross-wave coupling, + but a wave is still `wait: true` — a crashloop in `gitea-actions` (wave 1) will block wave 2 + (`seerr`) until it recovers or its `timeout` elapses. That is the intended linear-wave + trade-off; if the PM wants `gitea-actions` (flaky privileged DinD) to not gate `seerr`, + consider giving CI plumbing its own non-gating Kustomization branch (future work, not in + this PR). +3. **New-deepest-wave onboarding needs two one-line manual edits** (`clusters/base/kustomization.yaml` + + a `clusters/dev` decryption patch). `render.py --emit-waves` generates the CR but does not + edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing + in an existing wave is fully automatic. +4. **Image automation.** `platform/image-automation/image-update-automation.yaml` scans + `path: ./apps` (recursive) for `$imagepolicy` setters — `apps/wave-/` is under `./apps`, + so per-service image markers are still covered. No change needed; noted so it isn't a + surprise. +5. **Decryption drift.** Each wave must carry the SOPS decryption patch (done in this PR for + waves 0–2). If a future wave is added and the `clusters/dev` patch is forgotten, that wave + will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and + §4 call this out; a lint check that "every apps-wave- has a decryption patch" would be a + good hardening follow-up. +6. **`EXTRA_UNITS` is a small hand-maintained table** (only `gitea-actions` today). Any future + non-catalog app-tier unit must be added there or it will not be placed into a wave. diff --git a/apps/README.md b/apps/README.md index 00727d5..23137dc 100644 --- a/apps/README.md +++ b/apps/README.md @@ -1,3 +1,16 @@ # apps -_Placeholder — populated in Phase 0. See ../home-network-k3s (read-only spec)._ +App-tier manifests, rendered from `catalog/*.yaml` ServiceDefinitions by `catalog/render.py` +and split into dependency-ordered **waves** (`wave-/`). + +- Each `wave-/` is reconciled by its own Flux Kustomization `apps-wave-` + (`clusters/base/apps-wave-.yaml`), which `dependsOn` wave N-1 (wave 0 dependsOn + `platform-config`) and health-gates (`wait: true`) ONLY that wave. A crashlooping workload + in a later wave can no longer stall an earlier wave or block new revisions. +- `wave-/kustomization.yaml` and `clusters/base/apps-wave-.yaml` are **generated** — run + `python3 catalog/render.py --emit-waves` after onboarding a service; do not hand-edit them. +- Wave assignment = longest dependency-path depth over the catalog `dependencies:` graph; a + co-located CNPG database `-db` is a leaf and always lands strictly below its consumer. + +See [`../DESIGN-waves.md`](../DESIGN-waves.md) for the wave table, Flux topology, and the +cutover runbook. diff --git a/apps/kustomization.yaml b/apps/kustomization.yaml deleted file mode 100644 index 80bd8ad..0000000 --- a/apps/kustomization.yaml +++ /dev/null @@ -1,41 +0,0 @@ ---- -# Onboarded services (WS-APPS). Each app is rendered from its catalog/ ServiceDefinition by -# catalog/render.py and committed here; the `apps` Flux Kustomization (clusters/base/apps.yaml, -# dependsOn platform-config) reconciles this directory. Image-automation CRs are omitted for now -# (--no-image-automation) until the Flux image controllers are installed. -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: - - pihole.yaml - - gitea.yaml - - gitea-bootstrap-secret.sops.yaml # SOPS: fixed SECRET_KEY + break-glass admin creds (decrypted by Flux) - - gitea-actions.yaml # Phase 6 CI/CD: in-cluster act_runner (ns gitea-actions) + DinD - - gitea-actions-runner-secret.sops.yaml # SOPS: act_runner registration token (decrypted by Flux) - - sonarr.yaml - - radarr.yaml - - plex.yaml - - qbittorrent.yaml - - jackett.yaml - - seerr.yaml - - nextcloud.yaml - - nextcloud-db.yaml # CNPG PostgreSQL Cluster (ns nextcloud-db) — first CNPG-DB app - - nextcloud-db-app.sops.yaml # SOPS: CNPG owner credentials (basic-auth) for initdb - - nextcloud-bootstrap-secret.sops.yaml # SOPS: break-glass admin + mirrored db-password (ns nextcloud) - # RE-ENABLED: the two crashloop root causes are now fixed (both verified against the live image). - # (1) permission-denied writing /etc/semaphore/config.json — the image runs as non-root UID 1001; - # fixed by podSecurityContext.fsGroup=1001 (schema 1.3.0) so the mounted Longhorn PVC is - # group-writable (verified: the pod got past config write and successfully pinged the DB). - # (2) config panic "field 'Port' is not valid: tcp://:3000" — the kubelet injects a - # Docker-link SEMAPHORE_PORT for the same-named Service, colliding with Semaphore's own listen - # port; fixed by an explicit SEMAPHORE_PORT=":3000" env that shadows the service-link var. - # The env-driven config IS non-interactive (the wizard text is the wrapper echoing while auto- - # answering from env). OIDC remains deferred (break-glass local admin), same as gitea/nextcloud. - - semaphore.yaml - - semaphore-db.yaml # CNPG PostgreSQL Cluster (ns semaphore-db) — second CNPG-DB app - - semaphore-db-app.sops.yaml # SOPS: CNPG owner credentials (basic-auth) for initdb - - semaphore-bootstrap-secret.sops.yaml # SOPS: break-glass admin + mirrored db-password + access-key-encryption (ns semaphore) -# Gitea's headless-install env + break-glass admin postStart hook are now first-class in the -# ServiceDefinition (catalog/gitea.yaml env/lifecycle, schema >=1.2.0) and rendered inline into -# apps/gitea.yaml, so the former gitea-bootstrap-patch.yaml strategic-merge patch is retired. -# gitea-bootstrap-secret.sops.yaml is KEPT — the env still sources the fixed SECRET_KEY + admin -# creds from it via secretKeyRef. diff --git a/apps/gitea-bootstrap-secret.sops.yaml b/apps/wave-0/gitea-bootstrap-secret.sops.yaml similarity index 100% rename from apps/gitea-bootstrap-secret.sops.yaml rename to apps/wave-0/gitea-bootstrap-secret.sops.yaml diff --git a/apps/gitea.yaml b/apps/wave-0/gitea.yaml similarity index 100% rename from apps/gitea.yaml rename to apps/wave-0/gitea.yaml diff --git a/apps/jackett.yaml b/apps/wave-0/jackett.yaml similarity index 100% rename from apps/jackett.yaml rename to apps/wave-0/jackett.yaml diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml new file mode 100644 index 0000000..b3d539a --- /dev/null +++ b/apps/wave-0/kustomization.yaml @@ -0,0 +1,18 @@ +--- +# Wave 0 of the app tier — GENERATED by `catalog/render.py --emit-waves`. +# Reconciled by the Flux Kustomization `apps-wave-0` (clusters/base/apps-wave-0.yaml), which dependsOn `platform-config` and health-gates +# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave. +# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- gitea-bootstrap-secret.sops.yaml +- gitea.yaml +- jackett.yaml +- nextcloud-db-app.sops.yaml +- nextcloud-db.yaml +- pihole.yaml +- plex.yaml +- qbittorrent.yaml +- semaphore-db-app.sops.yaml +- semaphore-db.yaml diff --git a/apps/nextcloud-db-app.sops.yaml b/apps/wave-0/nextcloud-db-app.sops.yaml similarity index 100% rename from apps/nextcloud-db-app.sops.yaml rename to apps/wave-0/nextcloud-db-app.sops.yaml diff --git a/apps/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml similarity index 100% rename from apps/nextcloud-db.yaml rename to apps/wave-0/nextcloud-db.yaml diff --git a/apps/pihole.yaml b/apps/wave-0/pihole.yaml similarity index 100% rename from apps/pihole.yaml rename to apps/wave-0/pihole.yaml diff --git a/apps/plex.yaml b/apps/wave-0/plex.yaml similarity index 100% rename from apps/plex.yaml rename to apps/wave-0/plex.yaml diff --git a/apps/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml similarity index 100% rename from apps/qbittorrent.yaml rename to apps/wave-0/qbittorrent.yaml diff --git a/apps/semaphore-db-app.sops.yaml b/apps/wave-0/semaphore-db-app.sops.yaml similarity index 100% rename from apps/semaphore-db-app.sops.yaml rename to apps/wave-0/semaphore-db-app.sops.yaml diff --git a/apps/semaphore-db.yaml b/apps/wave-0/semaphore-db.yaml similarity index 100% rename from apps/semaphore-db.yaml rename to apps/wave-0/semaphore-db.yaml diff --git a/apps/gitea-actions-runner-secret.sops.yaml b/apps/wave-1/gitea-actions-runner-secret.sops.yaml similarity index 100% rename from apps/gitea-actions-runner-secret.sops.yaml rename to apps/wave-1/gitea-actions-runner-secret.sops.yaml diff --git a/apps/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml similarity index 100% rename from apps/gitea-actions.yaml rename to apps/wave-1/gitea-actions.yaml diff --git a/apps/wave-1/kustomization.yaml b/apps/wave-1/kustomization.yaml new file mode 100644 index 0000000..a0c54d7 --- /dev/null +++ b/apps/wave-1/kustomization.yaml @@ -0,0 +1,16 @@ +--- +# Wave 1 of the app tier — GENERATED by `catalog/render.py --emit-waves`. +# Reconciled by the Flux Kustomization `apps-wave-1` (clusters/base/apps-wave-1.yaml), which dependsOn `apps-wave-0` and health-gates +# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave. +# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- gitea-actions-runner-secret.sops.yaml +- gitea-actions.yaml +- nextcloud-bootstrap-secret.sops.yaml +- nextcloud.yaml +- radarr.yaml +- semaphore-bootstrap-secret.sops.yaml +- semaphore.yaml +- sonarr.yaml diff --git a/apps/nextcloud-bootstrap-secret.sops.yaml b/apps/wave-1/nextcloud-bootstrap-secret.sops.yaml similarity index 100% rename from apps/nextcloud-bootstrap-secret.sops.yaml rename to apps/wave-1/nextcloud-bootstrap-secret.sops.yaml diff --git a/apps/nextcloud.yaml b/apps/wave-1/nextcloud.yaml similarity index 100% rename from apps/nextcloud.yaml rename to apps/wave-1/nextcloud.yaml diff --git a/apps/radarr.yaml b/apps/wave-1/radarr.yaml similarity index 100% rename from apps/radarr.yaml rename to apps/wave-1/radarr.yaml diff --git a/apps/semaphore-bootstrap-secret.sops.yaml b/apps/wave-1/semaphore-bootstrap-secret.sops.yaml similarity index 100% rename from apps/semaphore-bootstrap-secret.sops.yaml rename to apps/wave-1/semaphore-bootstrap-secret.sops.yaml diff --git a/apps/semaphore.yaml b/apps/wave-1/semaphore.yaml similarity index 100% rename from apps/semaphore.yaml rename to apps/wave-1/semaphore.yaml diff --git a/apps/sonarr.yaml b/apps/wave-1/sonarr.yaml similarity index 100% rename from apps/sonarr.yaml rename to apps/wave-1/sonarr.yaml diff --git a/apps/wave-2/kustomization.yaml b/apps/wave-2/kustomization.yaml new file mode 100644 index 0000000..116be7a --- /dev/null +++ b/apps/wave-2/kustomization.yaml @@ -0,0 +1,9 @@ +--- +# Wave 2 of the app tier — GENERATED by `catalog/render.py --emit-waves`. +# Reconciled by the Flux Kustomization `apps-wave-2` (clusters/base/apps-wave-2.yaml), which dependsOn `apps-wave-1` and health-gates +# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave. +# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- seerr.yaml diff --git a/apps/seerr.yaml b/apps/wave-2/seerr.yaml similarity index 100% rename from apps/seerr.yaml rename to apps/wave-2/seerr.yaml diff --git a/bootstrap/flux/patches/apps-sops-decryption.yaml b/bootstrap/flux/patches/apps-sops-decryption.yaml index 408e8d7..32c1656 100644 --- a/bootstrap/flux/patches/apps-sops-decryption.yaml +++ b/bootstrap/flux/patches/apps-sops-decryption.yaml @@ -1,18 +1,19 @@ --- -# Enable SOPS decryption on the `apps` Flux Kustomization (ADR-0007, R-SEC-01). +# Enable SOPS decryption on the app-tier Flux Kustomizations (ADR-0007, R-SEC-01). # -# REALISES the commented TODO in clusters/base/apps.yaml (same pattern as the platform patch). -# Strategic-merge patch — adds the decryption stanza to the base `apps` Kustomization without -# forking it (R-ENV-02). `sops-age` Secret referenced BY NAME ONLY (created out-of-band). +# The app tier is now dependency-ordered WAVES (apps-wave-0..N), not a single `apps` +# Kustomization — so decryption is layered onto EACH wave, by name. clusters/dev inlines one +# such stanza per wave (see clusters/dev/kustomization.yaml); this file documents the shape. +# Strategic-merge patch — adds the decryption stanza without forking the base (R-ENV-02). +# `sops-age` Secret referenced BY NAME ONLY (created out-of-band). # -# WIRING (clusters owner / PM): add to clusters/dev/kustomization.yaml `patches:` alongside the -# platform patch, e.g. -# patches: -# - path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml +# WIRING (clusters owner / PM): the dev overlay carries one inlined patch per wave. When +# `render.py --emit-waves` introduces a NEW deepest wave, add a matching stanza there +# (metadata.name: apps-wave-). Below is the template for a single wave (wave 0 shown): apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: - name: apps + name: apps-wave-0 namespace: flux-system spec: decryption: diff --git a/catalog/render.py b/catalog/render.py index a125531..d74c0b8 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -935,6 +935,239 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N return emit(render(sd, image_automation=image_automation, inbound=inbound)) +# ============================================================================= +# Wave partitioning — dependency-ordered app-tier delivery (WS-APPS wave delivery) +# ============================================================================= +# A single monolithic `apps` Flux Kustomization health-gates (wait:true) the ENTIRE app +# set, so one crashlooping workload blocks ALL apps and every new revision. Instead we split +# the tier into dependency-ordered WAVES: each wave is its own Flux Kustomization +# `apps-wave-` that dependsOn wave N-1 (wave 0 dependsOn platform-config), so a failure in +# a later wave can no longer stall an earlier one, while everything WITHIN a wave still +# reconciles in parallel. This module computes the wave assignment from the declared catalog +# dependency graph and emits the per-wave kustomize + Flux Kustomization CRs. It changes only +# WHERE a rendered manifest is placed — never its bytes (the L1 goldens stay green). + +CO_DB_SUFFIX = "-db" # a co-located CNPG database for `` is the node `-db`. + +# Non-catalog app-tier units (they carry NO ServiceDefinition, so compute_waves can't see +# them). Each is anchored to a catalog service; `above=True` forces it into the wave STRICTLY +# ABOVE its anchor because it is a hard runtime prerequisite it cannot start without (e.g. the +# Gitea Actions act_runner registers against + clones from Gitea, so Gitea must be healthy +# first); `above=False` would co-locate it in the anchor's own wave. +EXTRA_UNITS: dict = { + "gitea-actions": { + "anchor": "gitea", + "above": True, + "files": [ + "gitea-actions.yaml", + "gitea-actions-runner-secret.sops.yaml", + ], + }, +} + + +def compute_waves(sds: list) -> dict: + """Longest-dependency-path wave per node over the catalog dependency DAG. + + Nodes = every catalog service ∪ every dependency target. A dependency target that has NO + ServiceDefinition (a co-located `-db` CNPG database) is a leaf. wave(n) = 0 if n + depends on nothing, else 1 + max(wave(dep)) over its declared dependencies. Because a + co-located DB is a leaf and its consumer declares a dependency ON it, the consumer always + lands in a strictly higher wave — the "DB strictly below its consumer" guarantee falls + straight out of the longest-path rule (no special case). Raises ValueError on a cycle. + """ + edges: dict = {} # node -> set(targets it depends on) + nodes: set = set() + for sd in sds: + if not isinstance(sd, dict) or "name" not in sd: + continue # tolerate non-ServiceDefinition siblings (e.g. render.py itself) + s = apply_defaults(sd) + name = s["name"] + nodes.add(name) + edges.setdefault(name, set()) + for dep in s["dependencies"]: + edges[name].add(dep["to"]) + nodes.add(dep["to"]) + edges.setdefault(dep["to"], set()) + + wave: dict = {} + visiting: set = set() + + def depth(n: str) -> int: + if n in wave: + return wave[n] + if n in visiting: + raise ValueError(f"dependency cycle through {n!r}") + visiting.add(n) + d = 0 + for t in sorted(edges.get(n, ())): + d = max(d, depth(t) + 1) + visiting.discard(n) + wave[n] = d + return d + + for n in sorted(nodes): + depth(n) + return wave + + +def _catalog_by_name(sds: list) -> dict: + out = {} + for sd in sds: + if isinstance(sd, dict) and "name" in sd: + s = apply_defaults(sd) + out[s["name"]] = s + return out + + +def service_wave(sds: list, name: str) -> int: + """The wave a single service's rendered manifest belongs in (for wave-aware `-o`).""" + return compute_waves(sds).get(name, 0) + + +def plan_apps_waves(sds: list, present_files: set) -> dict: + """Map the ONBOARDED app-tier file set into waves -> sorted resource basenames. + + `present_files` is the set of manifest basenames currently committed under apps/ (flat or + already in wave-/ subdirs), e.g. {"pihole.yaml", "nextcloud-db.yaml", + "nextcloud-db-app.sops.yaml", ...}. A catalog service is ONBOARDED iff `.yaml` is + present (the catalog also holds not-yet-onboarded SDs like keycloak/vm-demo, which must + NOT be placed). Each onboarded unit's sidecar files inherit the correct wave by naming + convention: + - `.yaml` -> wave(svc) + - `-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply) + - `-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB) + - `-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret) + Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset. + Only files that actually exist in `present_files` are placed, so the plan is robust to a + service having no bootstrap secret or no DB. Deterministic (sorted). + """ + waves = compute_waves(sds) + by_name = _catalog_by_name(sds) + plan: dict = {} + + def add(w: int, f: str) -> None: + if f in present_files: + plan.setdefault(w, set()).add(f) + + onboarded = {n for n in by_name if f"{n}.yaml" in present_files} + for name in sorted(onboarded): + w = waves[name] + add(w, f"{name}.yaml") + add(w, f"{name}-bootstrap-secret.sops.yaml") + db = f"{name}{CO_DB_SUFFIX}" + if any(dep["to"] == db for dep in by_name[name]["dependencies"]): + wdb = waves[db] + add(wdb, f"{db}.yaml") + add(wdb, f"{db}-app.sops.yaml") + + for unit, cfg in EXTRA_UNITS.items(): + anchor = cfg["anchor"] + if anchor not in onboarded: + continue + w = waves[anchor] + (1 if cfg.get("above") else 0) + for f in cfg["files"]: + add(w, f) + + return {w: sorted(fs) for w, fs in plan.items()} + + +def wave_kustomization_text(wave: int, files: list) -> str: + """The kustomize Kustomization for one wave directory (apps/wave-/kustomization.yaml).""" + dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}" + header = ( + f"# Wave {wave} of the app tier — GENERATED by `catalog/render.py --emit-waves`.\n" + f"# Reconciled by the Flux Kustomization `apps-wave-{wave}` " + f"(clusters/base/apps-wave-{wave}.yaml), which dependsOn `{dep}` and health-gates\n" + f"# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.\n" + f"# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service." + ) + body = { + "apiVersion": "kustomize.config.k8s.io/v1beta1", + "kind": "Kustomization", + "resources": list(files), + } + dumped = yaml.dump(body, sort_keys=False, default_flow_style=False, width=4096) + return f"---\n{header}\n{dumped}" + + +def flux_wave_kustomization_text(wave: int) -> str: + """The Flux Kustomization CR for one wave (clusters/base/apps-wave-.yaml). + + dependsOn wave N-1 (wave 0 dependsOn platform-config); prune+wait match the retired + monolith. SOPS decryption is added by the environment overlay patch, IDENTICALLY to the + former single `apps` Kustomization (clusters/dev patches each apps-wave- by name) — the + base stays env-agnostic and references the out-of-band `sops-age` Secret by name only. + """ + dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}" + header = ( + f"# Flux Kustomization: app-tier WAVE {wave}. Reconciles ./apps/wave-{wave} AFTER " + f"`{dep}` is healthy.\n" + f"# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its " + f"own wave + downstream:\n" + f"# wait:true health-gates ONLY this wave's resources, so a later wave can no longer " + f"stall an earlier one.\n" + f"# SOPS decryption is layered on per-environment (clusters/dev patches this " + f"Kustomization by name), identical to the retired `apps` Kustomization." + ) + cr = { + "apiVersion": "kustomize.toolkit.fluxcd.io/v1", + "kind": "Kustomization", + "metadata": {"name": f"apps-wave-{wave}", "namespace": "flux-system"}, + "spec": { + "interval": "10m", + "retryInterval": "2m", + "timeout": "5m", + "prune": True, + "wait": True, + "dependsOn": [{"name": dep}], + "sourceRef": {"kind": "GitRepository", "name": "flux-system"}, + "path": f"./apps/wave-{wave}", + "postBuild": { + "substituteFrom": [{"kind": "ConfigMap", "name": "cluster-vars"}] + }, + }, + } + dumped = yaml.dump(cr, sort_keys=False, default_flow_style=False, width=4096) + return f"---\n{header}\n{dumped}" + + +def _discover_app_files(apps_dir: Path) -> set: + """Basenames of every committed app-tier manifest under apps/ (flat OR in wave-/). + + Excludes the generated per-wave kustomization.yaml and any README. Deterministic. + """ + present = set() + for p in apps_dir.rglob("*.yaml"): + if p.name == "kustomization.yaml": + continue + present.add(p.name) + return present + + +def emit_waves(catalog_dir: Path, apps_dir: Path, clusters_dir: Path) -> dict: + """Write apps/wave-/kustomization.yaml + clusters/base/apps-wave-.yaml. + + Computes the wave plan from the catalog DAG and the app files present under apps/, then + (re)generates the per-wave kustomize files and the Flux Kustomization CRs. Idempotent: + it never moves or rewrites a service manifest (only its placement changes, done in git), + so the rendered bytes — and the L1 goldens — are untouched. Returns the wave plan. + """ + sds = _load_catalog(catalog_dir) + present = _discover_app_files(apps_dir) + plan = plan_apps_waves(sds, present) + for wave in sorted(plan): + wdir = apps_dir / f"wave-{wave}" + wdir.mkdir(parents=True, exist_ok=True) + (wdir / "kustomization.yaml").write_text( + wave_kustomization_text(wave, plan[wave]) + ) + (clusters_dir / f"apps-wave-{wave}.yaml").write_text( + flux_wave_kustomization_text(wave) + ) + return plan + + # ============================================================================= # Self-test (deterministic, no cluster) — `render.py --selftest` # ============================================================================= @@ -1187,6 +1420,74 @@ def _selftest() -> int: "an SD without podSecurityContext must not gain a pod securityContext", ) + # 7) wave partitioning: longest-path over the dependency DAG, CNPG-DB strictly below. + wsds = [ + {"name": "a", "source": {"image": "x:1"}}, # leaf + {"name": "b", "source": {"image": "x:1"}}, # leaf + {"name": "c", "source": {"image": "x:1"}, + "dependencies": [{"to": "a", "port": 1}, {"to": "b", "port": 1}]}, # wave 1 + {"name": "d", "source": {"image": "x:1"}, + "dependencies": [{"to": "c", "port": 1}]}, # wave 2 + {"name": "svc", "source": {"image": "x:1"}, + "dependencies": [{"to": "svc-db", "port": 5432}]}, # DB consumer + ] + w = compute_waves(wsds) + check(w["a"] == 0 and w["b"] == 0, "leaf services must be wave 0") + check(w["c"] == 1, "a service depending on wave-0 nodes must be wave 1") + check(w["d"] == 2, "longest-path depth must be transitive (d -> c -> {a,b})") + check( + w["svc-db"] == 0 and w["svc"] == 1 and w["svc-db"] < w["svc"], + "a co-located -db must be a leaf STRICTLY BELOW its consumer (CNPG-DB rule)", + ) + # cycle detection. + try: + compute_waves([ + {"name": "p", "source": {"image": "x:1"}, "dependencies": [{"to": "q", "port": 1}]}, + {"name": "q", "source": {"image": "x:1"}, "dependencies": [{"to": "p", "port": 1}]}, + ]) + check(False, "compute_waves must raise on a dependency cycle") + except ValueError: + pass + + # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped; + # EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed. + present = { + "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml", + "svc-db.yaml", "svc-db-app.sops.yaml", "gitea.yaml", + "gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml", + } + plan_sds = wsds + [{"name": "gitea", "source": {"image": "x:1"}}, + {"name": "notonboarded", "source": {"image": "x:1"}}] + plan = plan_apps_waves(plan_sds, present) + flat = {f: wv for wv, fs in plan.items() for f in fs} + check(flat.get("a.yaml") == 0, "leaf app must land in wave 0") + check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0, + "CNPG DB manifest + its owner secret must be in the DB's (lower) wave") + check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1, + "a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB") + check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0") + check(flat.get("gitea-actions.yaml") == 1 + and flat.get("gitea-actions-runner-secret.sops.yaml") == 1, + "gitea-actions (EXTRA_UNITS, above=True) must be one wave above its anchor gitea") + check("notonboarded.yaml" not in flat, + "a catalog SD with no committed manifest must NOT be placed (not onboarded)") + check("b.yaml" not in flat, "a file absent from the tree must never be placed") + + # 7b) generated Flux Kustomization CR wiring (dependsOn chain + prune/wait/path). + import yaml as _yaml # already imported at module top; local alias for clarity + fw0 = _yaml.safe_load(flux_wave_kustomization_text(0)) + fw2 = _yaml.safe_load(flux_wave_kustomization_text(2)) + check(fw0["metadata"]["name"] == "apps-wave-0" + and fw0["spec"]["dependsOn"] == [{"name": "platform-config"}] + and fw0["spec"]["path"] == "./apps/wave-0", + "wave 0 Flux Kustomization must dependsOn platform-config and point at ./apps/wave-0") + check(fw2["spec"]["dependsOn"] == [{"name": "apps-wave-1"}] + and fw2["spec"]["prune"] is True and fw2["spec"]["wait"] is True, + "wave N Flux Kustomization must dependsOn wave N-1 with prune+wait (health-gated)") + kt = _yaml.safe_load(wave_kustomization_text(1, ["svc.yaml", "sonarr.yaml"])) + check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"], + "wave kustomization must list exactly the wave's resources") + if failures: for f in failures: sys.stderr.write(f"SELFTEST FAIL: {f}\n") @@ -1204,30 +1505,72 @@ def main(argv=None) -> int: ap.add_argument("--no-image-automation", action="store_true", help="Omit Flux ImageRepository/ImagePolicy CRs (for onboarding before the " "image-reflector/-automation controllers are installed)") + ap.add_argument("--wave-layout", action="store_true", + help="With -o: place each rendered file under /wave-/.yaml, " + "N = its dependency-ordered wave (compute_waves over the catalog)") + ap.add_argument("--emit-waves", action="store_true", + help="Generate the per-wave kustomize + Flux Kustomization CRs from the " + "catalog DAG and the committed apps tree (no manifest is rewritten)") + ap.add_argument("--print-waves", action="store_true", + help="Print the computed wave table (node -> wave) for the catalog and exit") + ap.add_argument("--catalog", default="catalog", + help="Catalog dir for --emit-waves/--print-waves/--wave-layout (default: catalog)") + ap.add_argument("--apps-dir", default="apps", + help="App-tier dir for --emit-waves (default: apps)") + ap.add_argument("--clusters-dir", default="clusters/base", + help="Dir to write clusters/base/apps-wave-.yaml (default: clusters/base)") args = ap.parse_args(argv) if args.selftest: return _selftest() + if args.print_waves: + waves = compute_waves(_load_catalog(Path(args.catalog))) + for node in sorted(waves, key=lambda n: (waves[n], n)): + sys.stdout.write(f"wave {waves[node]} {node}\n") + return 0 + + if args.emit_waves: + plan = emit_waves(Path(args.catalog), Path(args.apps_dir), Path(args.clusters_dir)) + for wave in sorted(plan): + sys.stderr.write(f"[emit-waves] wave {wave}: {', '.join(plan[wave])}\n") + sys.stderr.write( + f"[emit-waves] wrote {len(plan)} wave kustomization(s) under {args.apps_dir}/ " + f"and clusters CRs under {args.clusters_dir}/\n" + ) + return 0 + image_automation = not args.no_image_automation + def _out_path(outdir: Path, name: str, sds: list) -> Path: + if args.wave_layout: + wdir = outdir / f"wave-{service_wave(sds, name)}" + wdir.mkdir(parents=True, exist_ok=True) + return wdir / f"{name}.yaml" + outdir.mkdir(parents=True, exist_ok=True) + return outdir / f"{name}.yaml" + if args.all: src = Path(args.all) files = sorted(p for p in src.glob("*.yaml") if p.name != "render.py") + sds = _load_catalog(src) if args.wave_layout else [] for f in files: text = render_file(f, image_automation=image_automation) if args.out_dir: - outdir = Path(args.out_dir) - outdir.mkdir(parents=True, exist_ok=True) - name = load_sd(f)["name"] - (outdir / f"{name}.yaml").write_text(text) + _out_path(Path(args.out_dir), load_sd(f)["name"], sds).write_text(text) else: sys.stdout.write(text) return 0 if not args.input: - ap.error("provide a ServiceDefinition file, --all DIR, or --selftest") - sys.stdout.write(render_file(Path(args.input), image_automation=image_automation)) + ap.error("provide a ServiceDefinition file, --all DIR, --emit-waves, or --selftest") + inp = Path(args.input) + text = render_file(inp, image_automation=image_automation) + if args.out_dir: + sds = _load_catalog(Path(args.catalog)) if args.wave_layout else [] + _out_path(Path(args.out_dir), load_sd(inp)["name"], sds).write_text(text) + else: + sys.stdout.write(text) return 0 diff --git a/clusters/README.md b/clusters/README.md index 479d715..b72633f 100644 --- a/clusters/README.md +++ b/clusters/README.md @@ -16,7 +16,9 @@ clusters/ base/ shared Flux objects (one authoring, both environments) flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea) platform.yaml Flux Kustomization → ./platform (reconciles first) - apps.yaml Flux Kustomization → ./apps (dependsOn platform) + apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config) + apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0) + apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1) cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults) kustomization.yaml dev/ ACTIVE build target — patches cluster-vars with dev values only @@ -60,10 +62,20 @@ GitRepository (flux-system) Kustomization: platform (./platform, substituteFrom cluster-vars) │ healthy ▼ -Kustomization: apps (./apps, dependsOn platform, substituteFrom cluster-vars) +Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars) + │ healthy + ▼ +Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0) + │ healthy + ▼ +Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1) ``` -`apps` `dependsOn` `platform` so CNI/storage/TLS/DNS/gateway exist before any workload reconciles. +Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway +exist before any workload reconciles. The app tier is split into dependency-ordered WAVES +(`catalog/render.py --emit-waves`): each `apps-wave-` health-gates (wait:true) ONLY its own +wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions. +See [DESIGN-waves.md](../DESIGN-waves.md) for the wave table + cutover runbook. ## TODOs (later phases — do not action in Phase 0) diff --git a/clusters/base/apps-wave-0.yaml b/clusters/base/apps-wave-0.yaml new file mode 100644 index 0000000..fd20bcf --- /dev/null +++ b/clusters/base/apps-wave-0.yaml @@ -0,0 +1,26 @@ +--- +# Flux Kustomization: app-tier WAVE 0. Reconciles ./apps/wave-0 AFTER `platform-config` is healthy. +# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream: +# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one. +# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: apps-wave-0 + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: true + dependsOn: + - name: platform-config + sourceRef: + kind: GitRepository + name: flux-system + path: ./apps/wave-0 + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/base/apps-wave-1.yaml b/clusters/base/apps-wave-1.yaml new file mode 100644 index 0000000..a827c11 --- /dev/null +++ b/clusters/base/apps-wave-1.yaml @@ -0,0 +1,26 @@ +--- +# Flux Kustomization: app-tier WAVE 1. Reconciles ./apps/wave-1 AFTER `apps-wave-0` is healthy. +# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream: +# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one. +# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: apps-wave-1 + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: true + dependsOn: + - name: apps-wave-0 + sourceRef: + kind: GitRepository + name: flux-system + path: ./apps/wave-1 + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/base/apps-wave-2.yaml b/clusters/base/apps-wave-2.yaml new file mode 100644 index 0000000..e710138 --- /dev/null +++ b/clusters/base/apps-wave-2.yaml @@ -0,0 +1,26 @@ +--- +# Flux Kustomization: app-tier WAVE 2. Reconciles ./apps/wave-2 AFTER `apps-wave-1` is healthy. +# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream: +# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one. +# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: apps-wave-2 + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: true + dependsOn: + - name: apps-wave-1 + sourceRef: + kind: GitRepository + name: flux-system + path: ./apps/wave-2 + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/base/apps.yaml b/clusters/base/apps.yaml deleted file mode 100644 index b55cff2..0000000 --- a/clusters/base/apps.yaml +++ /dev/null @@ -1,31 +0,0 @@ ---- -# Flux Kustomization: APPLICATION workloads. Reconciles ./apps AFTER platform is healthy. -# Phase 0 STUB. apps/ is owned by WS-APPS; this entrypoint is the GitOps contract that renders it. -apiVersion: kustomize.toolkit.fluxcd.io/v1 -kind: Kustomization -metadata: - name: apps - namespace: flux-system -spec: - interval: 10m - retryInterval: 2m - timeout: 5m - # CUTOVER STEP 1 (temporary): prune disabled so that when this `apps` Kustomization object is - # removed in the wave-split commit, Flux does NOT garbage-collect the resources it owns (notably - # the CNPG database Clusters -> PVCs). The apps-wave-* Kustomizations then adopt them in place. - # This whole file is deleted in the follow-up wave-split commit. - prune: false - wait: true - # Ordering: the full platform (controllers + their configs) must be healthy first. - # platform-config dependsOn platform, so depending on platform-config transitively waits for both. - dependsOn: - - name: platform-config - sourceRef: - kind: GitRepository - name: flux-system - path: ./apps - postBuild: - substituteFrom: - - kind: ConfigMap - name: cluster-vars - # TODO(WS-SEC, Phase 1): enable SOPS decryption (ADR-0007) — reference age secret by name only. diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 76f7466..8d308b4 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -7,5 +7,12 @@ resources: - flux-system.yaml # GitRepository (reconciliation source) - platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs) - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform) - - apps.yaml # Flux Kustomization -> ./apps (dependsOn platform-config) + # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization). + # Each apps-wave- dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates + # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one. + # GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave + # is introduced (a service deeper than any existing wave). See DESIGN-waves.md. + - apps-wave-0.yaml # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config) + - apps-wave-1.yaml # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0) + - apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index c78ba39..38f1fd2 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -26,11 +26,37 @@ patches: provider: sops secretRef: name: sops-age + # SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps` + # Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by + # name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave + # (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only). - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: - name: apps + name: apps-wave-0 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-1 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-2 namespace: flux-system spec: decryption: