refactor(apps): split monolithic apps Kustomization into dependency waves

Replaces the single `apps` Flux Kustomization with dependency-ordered waves (apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream instead of stalling the entire app tier. Parallel within a wave, ordered across. - catalog/render.py: compute waves by longest dependency-path depth over the ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its consumer lands strictly higher — the "DB below consumer" rule falls out of longest-path, no special case). `--emit-waves` generates the per-wave kustomizations + Flux CRs; `--wave-layout` places renders into the right dir. wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db wave 1: sonarr radarr nextcloud semaphore gitea-actions wave 2: seerr - apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact). - clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each). - clusters/dev: one SOPS decryption patch per wave (load-bearing). - DESIGN-waves.md: wave table, topology, cutover runbook. Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that object here does not GC its resources; the wave Kustomizations SSA-adopt the byte-identical manifests in place (no restart, no PVC churn). Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:19:00 -05:00
1 parent 1bfb0db6f1
commit 36a1a4e1f9
@@ -16,7 +16,9 @@ clusters/
  base/                 shared Flux objects (one authoring, both environments)
    flux-system.yaml    GitRepository — the reconciliation source (bootstrap → Gitea)
    platform.yaml       Flux Kustomization → ./platform   (reconciles first)
-    apps.yaml           Flux Kustomization → ./apps        (dependsOn platform)
+    apps-wave-0.yaml    Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
+    apps-wave-1.yaml    Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
+    apps-wave-2.yaml    Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
    cluster-vars.yaml   ConfigMap of overlay knobs (BASE defaults)
    kustomization.yaml
  dev/                  ACTIVE build target — patches cluster-vars with dev values only
@@ -60,10 +62,20 @@ GitRepository (flux-system)
 Kustomization: platform  (./platform, substituteFrom cluster-vars)
        │  healthy
        ▼
-Kustomization: apps      (./apps, dependsOn platform, substituteFrom cluster-vars)
+Kustomization: apps-wave-0  (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
+        │  healthy
+        ▼
+Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
+        │  healthy
+        ▼
+Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```

-`apps` `dependsOn` `platform` so CNI/storage/TLS/DNS/gateway exist before any workload reconciles.
+Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
+exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
+(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
+wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
+See [DESIGN-waves.md](../DESIGN-waves.md) for the wave table + cutover runbook.

 ## TODOs (later phases — do not action in Phase 0)

@@ -0,0 +1,26 @@
+---
+# Flux Kustomization: app-tier WAVE 0. Reconciles ./apps/wave-0 AFTER `platform-config` is healthy.
+# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
+# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
+# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: apps-wave-0
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+  - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./apps/wave-0
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-vars
@@ -0,0 +1,26 @@
+---
+# Flux Kustomization: app-tier WAVE 1. Reconciles ./apps/wave-1 AFTER `apps-wave-0` is healthy.
+# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
+# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
+# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: apps-wave-1
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+  - name: apps-wave-0
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./apps/wave-1
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-vars
@@ -0,0 +1,26 @@
+---
+# Flux Kustomization: app-tier WAVE 2. Reconciles ./apps/wave-2 AFTER `apps-wave-1` is healthy.
+# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
+# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
+# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: apps-wave-2
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+  - name: apps-wave-1
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./apps/wave-2
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-vars
@@ -1,31 +0,0 @@
---
-# Flux Kustomization: APPLICATION workloads. Reconciles ./apps AFTER platform is healthy.
-# Phase 0 STUB. apps/ is owned by WS-APPS; this entrypoint is the GitOps contract that renders it.
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: apps
-  namespace: flux-system
-spec:
-  interval: 10m
-  retryInterval: 2m
-  timeout: 5m
-  # CUTOVER STEP 1 (temporary): prune disabled so that when this `apps` Kustomization object is
-  # removed in the wave-split commit, Flux does NOT garbage-collect the resources it owns (notably
-  # the CNPG database Clusters -> PVCs). The apps-wave-* Kustomizations then adopt them in place.
-  # This whole file is deleted in the follow-up wave-split commit.
-  prune: false
-  wait: true
-  # Ordering: the full platform (controllers + their configs) must be healthy first.
-  # platform-config dependsOn platform, so depending on platform-config transitively waits for both.
-  dependsOn:
-    - name: platform-config
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./apps
-  postBuild:
-    substituteFrom:
-      - kind: ConfigMap
-        name: cluster-vars
-  # TODO(WS-SEC, Phase 1): enable SOPS decryption (ADR-0007) — reference age secret by name only.
@@ -7,5 +7,12 @@ resources:
  - flux-system.yaml     # GitRepository (reconciliation source)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
-  - apps.yaml            # Flux Kustomization -> ./apps (dependsOn platform-config)
+  # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
+  # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
+  # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
+  # GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave
+  # is introduced (a service deeper than any existing wave). See DESIGN-waves.md.
+  - apps-wave-0.yaml     # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config)
+  - apps-wave-1.yaml     # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0)
+  - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
  - cluster-vars.yaml    # overlay knob surface (base defaults)
@@ -26,11 +26,37 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
+  # SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
+  # Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
+  # name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
+  # (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
-        name: apps
+        name: apps-wave-0
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-1
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-2
        namespace: flux-system
      spec:
        decryption: