refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
1bfb0db6f1
commit
36a1a4e1f9
34 files changed
+793
-93
No files matched your search
+249
@@ -0,0 +1,249 @@
|
|||||||
|
# Wave-based app-tier delivery (PROTOTYPE — for PM review)
|
||||||
|
|
||||||
|
**Problem.** Today one Flux Kustomization `apps` (clusters/base/apps.yaml, `dependsOn:
|
||||||
|
platform-config`, `wait: true`, `prune: true`) reconciles all of `apps/` — ~10 services + CNPG
|
||||||
|
databases + SOPS secrets. Because `wait: true` health-gates the **entire** set, a single
|
||||||
|
crashlooping workload blocks **all** apps and every new revision (this has bitten twice).
|
||||||
|
|
||||||
|
**Solution.** Split the tier into dependency-ordered **waves**. Each wave is its own Flux
|
||||||
|
Kustomization `apps-wave-<N>` that `dependsOn` wave N-1 (wave 0 `dependsOn` `platform-config`),
|
||||||
|
with `wait: true` + `prune: true` and `path: ./apps/wave-<N>`. `wait: true` now health-gates
|
||||||
|
**only that wave**, so a crashloop isolates to its own wave and everything downstream, while
|
||||||
|
services **within** a wave still reconcile in parallel. Wave assignment is computed by
|
||||||
|
`catalog/render.py` from the declared dependency graph, so future onboarding auto-places a
|
||||||
|
service in the correct wave.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Wave assignment algorithm
|
||||||
|
|
||||||
|
`wave(node)` = **longest dependency-path depth** over the catalog `dependencies:` edges
|
||||||
|
(service → service). A node with no dependencies is wave 0; a node that depends on a wave-k node
|
||||||
|
is wave ≥ k+1. Implemented in `catalog/render.py :: compute_waves()`.
|
||||||
|
|
||||||
|
**Co-located CNPG database rule.** A service `<svc>` that declares a dependency `to: <svc>-db`
|
||||||
|
(e.g. `nextcloud` → `nextcloud-db`, `semaphore` → `semaphore-db`) has a hard DB prerequisite.
|
||||||
|
The `<svc>-db` node carries **no** ServiceDefinition, so it is a **leaf** (wave 0); because the
|
||||||
|
consumer declares a dependency on it, the consumer lands in a strictly higher wave. **The
|
||||||
|
"DB strictly below its consumer" guarantee therefore falls straight out of the longest-path
|
||||||
|
rule — no special case.** The DB manifest (`<svc>-db.yaml`) and its CNPG `initdb` owner secret
|
||||||
|
(`<svc>-db-app.sops.yaml`) are placed in the DB's (lower) wave; the app manifest and its
|
||||||
|
`<svc>-bootstrap-secret.sops.yaml` are placed in the consumer's wave.
|
||||||
|
|
||||||
|
**Non-catalog units.** A few app-tier files carry no ServiceDefinition (`compute_waves` can't
|
||||||
|
see them). They are anchored explicitly in `render.py :: EXTRA_UNITS`. Today the only one is
|
||||||
|
**gitea-actions** (the in-cluster `act_runner` + its DinD sidecar), anchored to `gitea` with
|
||||||
|
`above: true` — the runner registers against and clones from Gitea, so Gitea must be healthy
|
||||||
|
first → it lands one wave **above** gitea.
|
||||||
|
|
||||||
|
Cycle detection: `compute_waves` raises on any dependency cycle.
|
||||||
|
|
||||||
|
## 2. Computed WAVE TABLE
|
||||||
|
|
||||||
|
Verify with `python3 catalog/render.py --print-waves` (nodes) and `--emit-waves` (file plan).
|
||||||
|
|
||||||
|
| Wave | File | Kind | Reason (dependency path) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **0** | `pihole.yaml` | app | no dependencies (leaf) |
|
||||||
|
| 0 | `gitea.yaml` (+ `gitea-bootstrap-secret.sops.yaml`) | app + secret | no dependencies (leaf) |
|
||||||
|
| 0 | `jackett.yaml` | app | no dependencies (leaf) |
|
||||||
|
| 0 | `plex.yaml` | app | no dependencies (leaf) |
|
||||||
|
| 0 | `qbittorrent.yaml` | app | no dependencies (leaf) |
|
||||||
|
| 0 | `nextcloud-db.yaml` (+ `nextcloud-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `nextcloud` |
|
||||||
|
| 0 | `semaphore-db.yaml` (+ `semaphore-db-app.sops.yaml`) | CNPG DB + secret | co-located DB leaf; strictly below `semaphore` |
|
||||||
|
| **1** | `sonarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
|
||||||
|
| 1 | `radarr.yaml` | app | → `qbittorrent` (w0), `jackett` (w0) |
|
||||||
|
| 1 | `nextcloud.yaml` (+ `nextcloud-bootstrap-secret.sops.yaml`) | app + secret | → `nextcloud-db` (w0) |
|
||||||
|
| 1 | `semaphore.yaml` (+ `semaphore-bootstrap-secret.sops.yaml`) | app + secret | → `semaphore-db` (w0) |
|
||||||
|
| 1 | `gitea-actions.yaml` (+ `gitea-actions-runner-secret.sops.yaml`) | CI runner + secret | EXTRA_UNITS: anchor `gitea` (w0), `above` → one wave up |
|
||||||
|
| **2** | `seerr.yaml` | app | → `sonarr` (w1), `radarr` (w1) |
|
||||||
|
|
||||||
|
Catalog services **not yet onboarded** (no committed manifest) are computed but **not placed**:
|
||||||
|
`keycloak` (w1, → `keycloak-db` w0) and `vm-demo` (w0). They join a wave automatically when
|
||||||
|
onboarded.
|
||||||
|
|
||||||
|
Depth check: 62 (wave 0) + 55 (wave 1) + 13 (wave 2) = **130 resources**, byte-for-byte the
|
||||||
|
same set the monolithic `apps` Kustomization built before the split (validated with
|
||||||
|
`kubectl kustomize`).
|
||||||
|
|
||||||
|
## 3. Flux Kustomization topology
|
||||||
|
|
||||||
|
```
|
||||||
|
platform (controllers/CRDs)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
platform-config (cluster CRs)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
apps-wave-0 → ./apps/wave-0 dependsOn platform-config wait:true prune:true (leaves + CNPG DBs)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
apps-wave-1 → ./apps/wave-1 dependsOn apps-wave-0 wait:true prune:true (arr, DB-backed apps, gitea-actions)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
apps-wave-2 → ./apps/wave-2 dependsOn apps-wave-1 wait:true prune:true (seerr)
|
||||||
|
```
|
||||||
|
|
||||||
|
Every wave Kustomization is identical to the retired `apps` one except `name`, `path`, and
|
||||||
|
`dependsOn`: same `interval/retryInterval/timeout`, `prune: true`, `wait: true`,
|
||||||
|
`postBuild.substituteFrom cluster-vars`, and the **same** SOPS decryption (added per-environment
|
||||||
|
in `clusters/dev/kustomization.yaml` — one inlined patch per `apps-wave-<N>`, referencing the
|
||||||
|
out-of-band `sops-age` Secret by name only, exactly as the single `apps` Kustomization did).
|
||||||
|
|
||||||
|
**Failure isolation gained:** a crashloop in `seerr` (wave 2) no longer blocks `apps-wave-0`
|
||||||
|
or `apps-wave-1`, nor their new revisions. A crashloop in wave 1 blocks only wave 2 (its
|
||||||
|
downstream), never wave 0. Before, any one crashloop stalled the whole tier.
|
||||||
|
|
||||||
|
## 4. What `render.py` now does
|
||||||
|
|
||||||
|
- `compute_waves(sds)` — longest-path wave per node (services + DB leaf nodes); cycle-safe.
|
||||||
|
- `plan_apps_waves(sds, present_files)` — maps the committed app file set → `{wave: [files]}`
|
||||||
|
by naming convention (`<svc>`, `<svc>-bootstrap-secret`, `<svc>-db`, `<svc>-db-app`) plus
|
||||||
|
`EXTRA_UNITS`; only files actually present are placed (not-yet-onboarded SDs are skipped).
|
||||||
|
- `--emit-waves` — (re)generates every `apps/wave-<N>/kustomization.yaml` and
|
||||||
|
`clusters/base/apps-wave-<N>.yaml`. Idempotent; **never rewrites a service manifest** (only
|
||||||
|
placement changes, done in git), so rendered bytes — and the L1 goldens — are untouched.
|
||||||
|
- `--print-waves` — prints the node→wave table.
|
||||||
|
- `-o <dir> --wave-layout` — future onboarding writes `<dir>/wave-<N>/<svc>.yaml` directly.
|
||||||
|
- Self-test extended (`--selftest`): longest-path depth, CNPG-DB-below-consumer, cycle
|
||||||
|
detection, sidecar/EXTRA_UNITS placement, and the generated CR wiring (dependsOn/prune/wait).
|
||||||
|
|
||||||
|
**Onboarding a new service now:** add `catalog/<svc>.yaml`, render it
|
||||||
|
(`render.py catalog/<svc>.yaml -o apps --wave-layout [--no-image-automation]`), then
|
||||||
|
`render.py --emit-waves`. If it lands in an existing wave, no clusters/base edit is needed. If
|
||||||
|
it creates a **new deepest** wave, add one line to `clusters/base/kustomization.yaml` and one
|
||||||
|
decryption patch to `clusters/dev/kustomization.yaml` (both flagged in-file).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. CUTOVER RUNBOOK — Flux resource adoption without data loss
|
||||||
|
|
||||||
|
> **Read this before merge.** Moving a manifest from the `apps` Kustomization to an
|
||||||
|
> `apps-wave-<N>` Kustomization is a Flux **ownership transfer** (adoption). Done wrong, one
|
||||||
|
> Kustomization can **prune** (delete) a resource another is adopting — and for the CNPG
|
||||||
|
> databases that is **data loss**.
|
||||||
|
|
||||||
|
### 5.1 Why the naive "just merge" is dangerous
|
||||||
|
|
||||||
|
Flux marks every resource it applies with `kustomize.toolkit.fluxcd.io/name` +
|
||||||
|
`…/namespace`, and with `prune: true` it **garbage-collects** any owned resource that leaves
|
||||||
|
its build. Two hazards on a plain merge of this PR:
|
||||||
|
|
||||||
|
1. **The live `apps` Kustomization has `prune: true`.** This PR removes `apps.yaml` from
|
||||||
|
`clusters/base`. The root Kustomization (`flux-system`, `./clusters/dev`, `prune: true`)
|
||||||
|
will then **delete the `apps` Kustomization object**. Deleting a Flux Kustomization that
|
||||||
|
has `prune: true` runs its finalizer, which **garbage-collects every resource it owns** —
|
||||||
|
all ~130 app resources, **including the `nextcloud-db` / `semaphore-db` CNPG Clusters**.
|
||||||
|
Deleting a CNPG `Cluster` cascade-deletes its PVCs → **the databases are destroyed**, even
|
||||||
|
though the new waves were about to adopt them.
|
||||||
|
2. **Ordering race.** Even setting that aside, if the old Kustomization's prune and the new
|
||||||
|
waves' adoption interleave, a resource can be deleted-then-recreated — a needless restart
|
||||||
|
for stateless apps, and unacceptable for the stateful DBs.
|
||||||
|
|
||||||
|
The fix is to make the old Kustomization **release** its resources (orphan-alive, no GC) so the
|
||||||
|
waves **adopt** them by identity — never a delete/recreate window.
|
||||||
|
|
||||||
|
### 5.2 Preferred sequence — pure GitOps, two merges (no data loss, no downtime)
|
||||||
|
|
||||||
|
Split the change into two PRs so the release happens before the split, entirely through git.
|
||||||
|
|
||||||
|
**Pre-flight (no changes):**
|
||||||
|
- `flux get kustomizations` → `apps` is `Ready`. Cluster healthy.
|
||||||
|
- Record the stateful state you must not lose:
|
||||||
|
`kubectl get cluster -A`, `kubectl get pvc -n nextcloud-db`, `kubectl get pvc -n semaphore-db`
|
||||||
|
(note names + `AGE`). Take a Longhorn/Velero backup of the `nextcloud-db` and `semaphore-db`
|
||||||
|
namespaces (defense in depth).
|
||||||
|
|
||||||
|
**PR 1 — neutralize GC on the old `apps` Kustomization (one-line change):**
|
||||||
|
- Edit `clusters/base/apps.yaml`: `prune: true` → `prune: false`. Merge.
|
||||||
|
- Wait until `flux get kustomization apps` reconciles with the new spec
|
||||||
|
(`kubectl -n flux-system get kustomization apps -o jsonpath='{.spec.prune}'` → `false`).
|
||||||
|
- Effect: when `apps` is later deleted, its finalizer will **not** garbage-collect its
|
||||||
|
resources — they survive as orphaned-but-alive objects.
|
||||||
|
|
||||||
|
**PR 2 — this wave-split PR.** On merge, the root Kustomization reconciles `./clusters/dev`:
|
||||||
|
- `apps` is gone from the build → root prunes the **`apps` Kustomization object**. Because
|
||||||
|
`apps` now has `prune: false` (from PR 1), **no app resources are deleted**; they are
|
||||||
|
orphaned, still running.
|
||||||
|
- `apps-wave-0/1/2` are created. Each does a server-side apply of the **same, byte-identical**
|
||||||
|
manifests, **adopting** the live objects (matched by GVK + namespace + name) and relabeling
|
||||||
|
`kustomize.toolkit.fluxcd.io/name` to its wave. Because content is unchanged, SSA is a no-op
|
||||||
|
on `spec` — **no pod restart, no PVC churn, no DB re-init**. Same field manager
|
||||||
|
(`kustomize-controller`) → no ownership conflict.
|
||||||
|
- Decryption is already wired on every wave (this PR), so the SOPS Secrets re-apply decrypted
|
||||||
|
and identical — critical: a wave WITHOUT decryption would apply the encrypted YAML and
|
||||||
|
corrupt the Secret.
|
||||||
|
|
||||||
|
**Verify:**
|
||||||
|
```
|
||||||
|
flux get kustomizations # apps-wave-0/1/2 Ready; apps absent
|
||||||
|
kubectl get cluster -A # DBs present, ORIGINAL age (not recreated)
|
||||||
|
kubectl get pvc -n nextcloud-db -n semaphore-db # same PVCs, same age
|
||||||
|
kubectl get deploy -A -o wide # apps not restarted (AGE/restarts unchanged)
|
||||||
|
kubectl get <any-app-deploy> -n <ns> \
|
||||||
|
-o jsonpath='{.metadata.labels.kustomize\.toolkit\.fluxcd\.io/name}' # == apps-wave-<N>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.3 Single-PR variant (if a live pre-step is acceptable instead of PR 1)
|
||||||
|
|
||||||
|
If you would rather merge this PR alone, perform PR 1's effect out-of-band **first**, then
|
||||||
|
merge:
|
||||||
|
```
|
||||||
|
flux -n flux-system suspend kustomization apps # stop it racing the waves
|
||||||
|
kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"prune":false}}'
|
||||||
|
```
|
||||||
|
Belt-and-suspenders for the databases (a per-object opt-out Flux always honors, regardless of
|
||||||
|
which Kustomization ends up owning them during the window):
|
||||||
|
```
|
||||||
|
kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune=disabled
|
||||||
|
kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune=disabled
|
||||||
|
```
|
||||||
|
Then merge this PR and verify as in 5.2. **Cleanup once the waves own the DBs** (restore normal
|
||||||
|
prune semantics):
|
||||||
|
```
|
||||||
|
kubectl -n nextcloud-db annotate cluster/nextcloud-db kustomize.toolkit.fluxcd.io/prune-
|
||||||
|
kubectl -n semaphore-db annotate cluster/semaphore-db kustomize.toolkit.fluxcd.io/prune-
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.4 Rollback
|
||||||
|
|
||||||
|
Nothing is ever deleted, so rollback is safe at every step. If a wave fails to adopt: the
|
||||||
|
resources are still alive (orphaned). `flux suspend` the waves, revert the merge (re-creating
|
||||||
|
`apps`, which re-adopts by identity), and investigate. No data is lost because the cutover
|
||||||
|
never issues a delete.
|
||||||
|
|
||||||
|
### 5.5 Why adoption is safe here (summary of the invariant)
|
||||||
|
|
||||||
|
- The old owner **releases without GC** (`prune: false` before it is deleted) → no delete.
|
||||||
|
- The new owners **adopt by identity** via SSA on byte-identical manifests → no recreate.
|
||||||
|
- There is **no window** in which a resource is owned by nothing *and* pruned: it is
|
||||||
|
continuously alive, transitioning label ownership from `apps` → `apps-wave-<N>`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Risks / open items for the PM
|
||||||
|
|
||||||
|
1. **The GC hazard above is the whole ball game.** A plain merge of this PR onto a live cluster
|
||||||
|
whose `apps` Kustomization still has `prune: true` will destroy the CNPG databases. §5.2 /
|
||||||
|
§5.3 must be followed. (In dev this has never been applied — prototype only.)
|
||||||
|
2. **`prune` blast radius within a wave is unchanged.** Splitting reduces cross-wave coupling,
|
||||||
|
but a wave is still `wait: true` — a crashloop in `gitea-actions` (wave 1) will block wave 2
|
||||||
|
(`seerr`) until it recovers or its `timeout` elapses. That is the intended linear-wave
|
||||||
|
trade-off; if the PM wants `gitea-actions` (flaky privileged DinD) to not gate `seerr`,
|
||||||
|
consider giving CI plumbing its own non-gating Kustomization branch (future work, not in
|
||||||
|
this PR).
|
||||||
|
3. **New-deepest-wave onboarding needs two one-line manual edits** (`clusters/base/kustomization.yaml`
|
||||||
|
+ a `clusters/dev` decryption patch). `render.py --emit-waves` generates the CR but does not
|
||||||
|
edit those hand-maintained, comment-carrying files; both spots are flagged in-file. Landing
|
||||||
|
in an existing wave is fully automatic.
|
||||||
|
4. **Image automation.** `platform/image-automation/image-update-automation.yaml` scans
|
||||||
|
`path: ./apps` (recursive) for `$imagepolicy` setters — `apps/wave-<N>/` is under `./apps`,
|
||||||
|
so per-service image markers are still covered. No change needed; noted so it isn't a
|
||||||
|
surprise.
|
||||||
|
5. **Decryption drift.** Each wave must carry the SOPS decryption patch (done in this PR for
|
||||||
|
waves 0–2). If a future wave is added and the `clusters/dev` patch is forgotten, that wave
|
||||||
|
will apply encrypted SOPS files verbatim and corrupt its Secrets. The in-file reminders and
|
||||||
|
§4 call this out; a lint check that "every apps-wave-<N> has a decryption patch" would be a
|
||||||
|
good hardening follow-up.
|
||||||
|
6. **`EXTRA_UNITS` is a small hand-maintained table** (only `gitea-actions` today). Any future
|
||||||
|
non-catalog app-tier unit must be added there or it will not be placed into a wave.
|
||||||
+14
-1
@@ -1,3 +1,16 @@
|
|||||||
# apps
|
# apps
|
||||||
|
|
||||||
_Placeholder — populated in Phase 0. See ../home-network-k3s (read-only spec)._
|
App-tier manifests, rendered from `catalog/*.yaml` ServiceDefinitions by `catalog/render.py`
|
||||||
|
and split into dependency-ordered **waves** (`wave-<N>/`).
|
||||||
|
|
||||||
|
- Each `wave-<N>/` is reconciled by its own Flux Kustomization `apps-wave-<N>`
|
||||||
|
(`clusters/base/apps-wave-<N>.yaml`), which `dependsOn` wave N-1 (wave 0 dependsOn
|
||||||
|
`platform-config`) and health-gates (`wait: true`) ONLY that wave. A crashlooping workload
|
||||||
|
in a later wave can no longer stall an earlier wave or block new revisions.
|
||||||
|
- `wave-<N>/kustomization.yaml` and `clusters/base/apps-wave-<N>.yaml` are **generated** — run
|
||||||
|
`python3 catalog/render.py --emit-waves` after onboarding a service; do not hand-edit them.
|
||||||
|
- Wave assignment = longest dependency-path depth over the catalog `dependencies:` graph; a
|
||||||
|
co-located CNPG database `<svc>-db` is a leaf and always lands strictly below its consumer.
|
||||||
|
|
||||||
|
See [`../DESIGN-waves.md`](../DESIGN-waves.md) for the wave table, Flux topology, and the
|
||||||
|
cutover runbook.
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
---
|
|
||||||
# Onboarded services (WS-APPS). Each app is rendered from its catalog/ ServiceDefinition by
|
|
||||||
# catalog/render.py and committed here; the `apps` Flux Kustomization (clusters/base/apps.yaml,
|
|
||||||
# dependsOn platform-config) reconciles this directory. Image-automation CRs are omitted for now
|
|
||||||
# (--no-image-automation) until the Flux image controllers are installed.
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- pihole.yaml
|
|
||||||
- gitea.yaml
|
|
||||||
- gitea-bootstrap-secret.sops.yaml # SOPS: fixed SECRET_KEY + break-glass admin creds (decrypted by Flux)
|
|
||||||
- gitea-actions.yaml # Phase 6 CI/CD: in-cluster act_runner (ns gitea-actions) + DinD
|
|
||||||
- gitea-actions-runner-secret.sops.yaml # SOPS: act_runner registration token (decrypted by Flux)
|
|
||||||
- sonarr.yaml
|
|
||||||
- radarr.yaml
|
|
||||||
- plex.yaml
|
|
||||||
- qbittorrent.yaml
|
|
||||||
- jackett.yaml
|
|
||||||
- seerr.yaml
|
|
||||||
- nextcloud.yaml
|
|
||||||
- nextcloud-db.yaml # CNPG PostgreSQL Cluster (ns nextcloud-db) — first CNPG-DB app
|
|
||||||
- nextcloud-db-app.sops.yaml # SOPS: CNPG owner credentials (basic-auth) for initdb
|
|
||||||
- nextcloud-bootstrap-secret.sops.yaml # SOPS: break-glass admin + mirrored db-password (ns nextcloud)
|
|
||||||
# RE-ENABLED: the two crashloop root causes are now fixed (both verified against the live image).
|
|
||||||
# (1) permission-denied writing /etc/semaphore/config.json — the image runs as non-root UID 1001;
|
|
||||||
# fixed by podSecurityContext.fsGroup=1001 (schema 1.3.0) so the mounted Longhorn PVC is
|
|
||||||
# group-writable (verified: the pod got past config write and successfully pinged the DB).
|
|
||||||
# (2) config panic "field 'Port' is not valid: tcp://<svcIP>:3000" — the kubelet injects a
|
|
||||||
# Docker-link SEMAPHORE_PORT for the same-named Service, colliding with Semaphore's own listen
|
|
||||||
# port; fixed by an explicit SEMAPHORE_PORT=":3000" env that shadows the service-link var.
|
|
||||||
# The env-driven config IS non-interactive (the wizard text is the wrapper echoing while auto-
|
|
||||||
# answering from env). OIDC remains deferred (break-glass local admin), same as gitea/nextcloud.
|
|
||||||
- semaphore.yaml
|
|
||||||
- semaphore-db.yaml # CNPG PostgreSQL Cluster (ns semaphore-db) — second CNPG-DB app
|
|
||||||
- semaphore-db-app.sops.yaml # SOPS: CNPG owner credentials (basic-auth) for initdb
|
|
||||||
- semaphore-bootstrap-secret.sops.yaml # SOPS: break-glass admin + mirrored db-password + access-key-encryption (ns semaphore)
|
|
||||||
# Gitea's headless-install env + break-glass admin postStart hook are now first-class in the
|
|
||||||
# ServiceDefinition (catalog/gitea.yaml env/lifecycle, schema >=1.2.0) and rendered inline into
|
|
||||||
# apps/gitea.yaml, so the former gitea-bootstrap-patch.yaml strategic-merge patch is retired.
|
|
||||||
# gitea-bootstrap-secret.sops.yaml is KEPT — the env still sources the fixed SECRET_KEY + admin
|
|
||||||
# creds from it via secretKeyRef.
|
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,18 @@
|
|||||||
|
---
|
||||||
|
# Wave 0 of the app tier — GENERATED by `catalog/render.py --emit-waves`.
|
||||||
|
# Reconciled by the Flux Kustomization `apps-wave-0` (clusters/base/apps-wave-0.yaml), which dependsOn `platform-config` and health-gates
|
||||||
|
# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.
|
||||||
|
# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- gitea-bootstrap-secret.sops.yaml
|
||||||
|
- gitea.yaml
|
||||||
|
- jackett.yaml
|
||||||
|
- nextcloud-db-app.sops.yaml
|
||||||
|
- nextcloud-db.yaml
|
||||||
|
- pihole.yaml
|
||||||
|
- plex.yaml
|
||||||
|
- qbittorrent.yaml
|
||||||
|
- semaphore-db-app.sops.yaml
|
||||||
|
- semaphore-db.yaml
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,16 @@
|
|||||||
|
---
|
||||||
|
# Wave 1 of the app tier — GENERATED by `catalog/render.py --emit-waves`.
|
||||||
|
# Reconciled by the Flux Kustomization `apps-wave-1` (clusters/base/apps-wave-1.yaml), which dependsOn `apps-wave-0` and health-gates
|
||||||
|
# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.
|
||||||
|
# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- gitea-actions-runner-secret.sops.yaml
|
||||||
|
- gitea-actions.yaml
|
||||||
|
- nextcloud-bootstrap-secret.sops.yaml
|
||||||
|
- nextcloud.yaml
|
||||||
|
- radarr.yaml
|
||||||
|
- semaphore-bootstrap-secret.sops.yaml
|
||||||
|
- semaphore.yaml
|
||||||
|
- sonarr.yaml
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
# Wave 2 of the app tier — GENERATED by `catalog/render.py --emit-waves`.
|
||||||
|
# Reconciled by the Flux Kustomization `apps-wave-2` (clusters/base/apps-wave-2.yaml), which dependsOn `apps-wave-1` and health-gates
|
||||||
|
# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.
|
||||||
|
# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- seerr.yaml
|
||||||
File renamed without changes.
@@ -1,18 +1,19 @@
|
|||||||
---
|
---
|
||||||
# Enable SOPS decryption on the `apps` Flux Kustomization (ADR-0007, R-SEC-01).
|
# Enable SOPS decryption on the app-tier Flux Kustomizations (ADR-0007, R-SEC-01).
|
||||||
#
|
#
|
||||||
# REALISES the commented TODO in clusters/base/apps.yaml (same pattern as the platform patch).
|
# The app tier is now dependency-ordered WAVES (apps-wave-0..N), not a single `apps`
|
||||||
# Strategic-merge patch — adds the decryption stanza to the base `apps` Kustomization without
|
# Kustomization — so decryption is layered onto EACH wave, by name. clusters/dev inlines one
|
||||||
# forking it (R-ENV-02). `sops-age` Secret referenced BY NAME ONLY (created out-of-band).
|
# such stanza per wave (see clusters/dev/kustomization.yaml); this file documents the shape.
|
||||||
|
# Strategic-merge patch — adds the decryption stanza without forking the base (R-ENV-02).
|
||||||
|
# `sops-age` Secret referenced BY NAME ONLY (created out-of-band).
|
||||||
#
|
#
|
||||||
# WIRING (clusters owner / PM): add to clusters/dev/kustomization.yaml `patches:` alongside the
|
# WIRING (clusters owner / PM): the dev overlay carries one inlined patch per wave. When
|
||||||
# platform patch, e.g.
|
# `render.py --emit-waves` introduces a NEW deepest wave, add a matching stanza there
|
||||||
# patches:
|
# (metadata.name: apps-wave-<N>). Below is the template for a single wave (wave 0 shown):
|
||||||
# - path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
name: apps
|
name: apps-wave-0
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
decryption:
|
decryption:
|
||||||
|
|||||||
+349
-6
@@ -935,6 +935,239 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
|
|||||||
return emit(render(sd, image_automation=image_automation, inbound=inbound))
|
return emit(render(sd, image_automation=image_automation, inbound=inbound))
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Wave partitioning — dependency-ordered app-tier delivery (WS-APPS wave delivery)
|
||||||
|
# =============================================================================
|
||||||
|
# A single monolithic `apps` Flux Kustomization health-gates (wait:true) the ENTIRE app
|
||||||
|
# set, so one crashlooping workload blocks ALL apps and every new revision. Instead we split
|
||||||
|
# the tier into dependency-ordered WAVES: each wave is its own Flux Kustomization
|
||||||
|
# `apps-wave-<N>` that dependsOn wave N-1 (wave 0 dependsOn platform-config), so a failure in
|
||||||
|
# a later wave can no longer stall an earlier one, while everything WITHIN a wave still
|
||||||
|
# reconciles in parallel. This module computes the wave assignment from the declared catalog
|
||||||
|
# dependency graph and emits the per-wave kustomize + Flux Kustomization CRs. It changes only
|
||||||
|
# WHERE a rendered manifest is placed — never its bytes (the L1 goldens stay green).
|
||||||
|
|
||||||
|
CO_DB_SUFFIX = "-db" # a co-located CNPG database for `<svc>` is the node `<svc>-db`.
|
||||||
|
|
||||||
|
# Non-catalog app-tier units (they carry NO ServiceDefinition, so compute_waves can't see
|
||||||
|
# them). Each is anchored to a catalog service; `above=True` forces it into the wave STRICTLY
|
||||||
|
# ABOVE its anchor because it is a hard runtime prerequisite it cannot start without (e.g. the
|
||||||
|
# Gitea Actions act_runner registers against + clones from Gitea, so Gitea must be healthy
|
||||||
|
# first); `above=False` would co-locate it in the anchor's own wave.
|
||||||
|
EXTRA_UNITS: dict = {
|
||||||
|
"gitea-actions": {
|
||||||
|
"anchor": "gitea",
|
||||||
|
"above": True,
|
||||||
|
"files": [
|
||||||
|
"gitea-actions.yaml",
|
||||||
|
"gitea-actions-runner-secret.sops.yaml",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def compute_waves(sds: list) -> dict:
|
||||||
|
"""Longest-dependency-path wave per node over the catalog dependency DAG.
|
||||||
|
|
||||||
|
Nodes = every catalog service ∪ every dependency target. A dependency target that has NO
|
||||||
|
ServiceDefinition (a co-located `<svc>-db` CNPG database) is a leaf. wave(n) = 0 if n
|
||||||
|
depends on nothing, else 1 + max(wave(dep)) over its declared dependencies. Because a
|
||||||
|
co-located DB is a leaf and its consumer declares a dependency ON it, the consumer always
|
||||||
|
lands in a strictly higher wave — the "DB strictly below its consumer" guarantee falls
|
||||||
|
straight out of the longest-path rule (no special case). Raises ValueError on a cycle.
|
||||||
|
"""
|
||||||
|
edges: dict = {} # node -> set(targets it depends on)
|
||||||
|
nodes: set = set()
|
||||||
|
for sd in sds:
|
||||||
|
if not isinstance(sd, dict) or "name" not in sd:
|
||||||
|
continue # tolerate non-ServiceDefinition siblings (e.g. render.py itself)
|
||||||
|
s = apply_defaults(sd)
|
||||||
|
name = s["name"]
|
||||||
|
nodes.add(name)
|
||||||
|
edges.setdefault(name, set())
|
||||||
|
for dep in s["dependencies"]:
|
||||||
|
edges[name].add(dep["to"])
|
||||||
|
nodes.add(dep["to"])
|
||||||
|
edges.setdefault(dep["to"], set())
|
||||||
|
|
||||||
|
wave: dict = {}
|
||||||
|
visiting: set = set()
|
||||||
|
|
||||||
|
def depth(n: str) -> int:
|
||||||
|
if n in wave:
|
||||||
|
return wave[n]
|
||||||
|
if n in visiting:
|
||||||
|
raise ValueError(f"dependency cycle through {n!r}")
|
||||||
|
visiting.add(n)
|
||||||
|
d = 0
|
||||||
|
for t in sorted(edges.get(n, ())):
|
||||||
|
d = max(d, depth(t) + 1)
|
||||||
|
visiting.discard(n)
|
||||||
|
wave[n] = d
|
||||||
|
return d
|
||||||
|
|
||||||
|
for n in sorted(nodes):
|
||||||
|
depth(n)
|
||||||
|
return wave
|
||||||
|
|
||||||
|
|
||||||
|
def _catalog_by_name(sds: list) -> dict:
|
||||||
|
out = {}
|
||||||
|
for sd in sds:
|
||||||
|
if isinstance(sd, dict) and "name" in sd:
|
||||||
|
s = apply_defaults(sd)
|
||||||
|
out[s["name"]] = s
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def service_wave(sds: list, name: str) -> int:
|
||||||
|
"""The wave a single service's rendered manifest belongs in (for wave-aware `-o`)."""
|
||||||
|
return compute_waves(sds).get(name, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||||
|
"""Map the ONBOARDED app-tier file set into waves -> sorted resource basenames.
|
||||||
|
|
||||||
|
`present_files` is the set of manifest basenames currently committed under apps/ (flat or
|
||||||
|
already in wave-<N>/ subdirs), e.g. {"pihole.yaml", "nextcloud-db.yaml",
|
||||||
|
"nextcloud-db-app.sops.yaml", ...}. A catalog service is ONBOARDED iff `<svc>.yaml` is
|
||||||
|
present (the catalog also holds not-yet-onboarded SDs like keycloak/vm-demo, which must
|
||||||
|
NOT be placed). Each onboarded unit's sidecar files inherit the correct wave by naming
|
||||||
|
convention:
|
||||||
|
- `<svc>.yaml` -> wave(svc)
|
||||||
|
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||||
|
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||||
|
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||||
|
Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
|
||||||
|
Only files that actually exist in `present_files` are placed, so the plan is robust to a
|
||||||
|
service having no bootstrap secret or no DB. Deterministic (sorted).
|
||||||
|
"""
|
||||||
|
waves = compute_waves(sds)
|
||||||
|
by_name = _catalog_by_name(sds)
|
||||||
|
plan: dict = {}
|
||||||
|
|
||||||
|
def add(w: int, f: str) -> None:
|
||||||
|
if f in present_files:
|
||||||
|
plan.setdefault(w, set()).add(f)
|
||||||
|
|
||||||
|
onboarded = {n for n in by_name if f"{n}.yaml" in present_files}
|
||||||
|
for name in sorted(onboarded):
|
||||||
|
w = waves[name]
|
||||||
|
add(w, f"{name}.yaml")
|
||||||
|
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||||
|
db = f"{name}{CO_DB_SUFFIX}"
|
||||||
|
if any(dep["to"] == db for dep in by_name[name]["dependencies"]):
|
||||||
|
wdb = waves[db]
|
||||||
|
add(wdb, f"{db}.yaml")
|
||||||
|
add(wdb, f"{db}-app.sops.yaml")
|
||||||
|
|
||||||
|
for unit, cfg in EXTRA_UNITS.items():
|
||||||
|
anchor = cfg["anchor"]
|
||||||
|
if anchor not in onboarded:
|
||||||
|
continue
|
||||||
|
w = waves[anchor] + (1 if cfg.get("above") else 0)
|
||||||
|
for f in cfg["files"]:
|
||||||
|
add(w, f)
|
||||||
|
|
||||||
|
return {w: sorted(fs) for w, fs in plan.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def wave_kustomization_text(wave: int, files: list) -> str:
|
||||||
|
"""The kustomize Kustomization for one wave directory (apps/wave-<N>/kustomization.yaml)."""
|
||||||
|
dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}"
|
||||||
|
header = (
|
||||||
|
f"# Wave {wave} of the app tier — GENERATED by `catalog/render.py --emit-waves`.\n"
|
||||||
|
f"# Reconciled by the Flux Kustomization `apps-wave-{wave}` "
|
||||||
|
f"(clusters/base/apps-wave-{wave}.yaml), which dependsOn `{dep}` and health-gates\n"
|
||||||
|
f"# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.\n"
|
||||||
|
f"# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service."
|
||||||
|
)
|
||||||
|
body = {
|
||||||
|
"apiVersion": "kustomize.config.k8s.io/v1beta1",
|
||||||
|
"kind": "Kustomization",
|
||||||
|
"resources": list(files),
|
||||||
|
}
|
||||||
|
dumped = yaml.dump(body, sort_keys=False, default_flow_style=False, width=4096)
|
||||||
|
return f"---\n{header}\n{dumped}"
|
||||||
|
|
||||||
|
|
||||||
|
def flux_wave_kustomization_text(wave: int) -> str:
|
||||||
|
"""The Flux Kustomization CR for one wave (clusters/base/apps-wave-<N>.yaml).
|
||||||
|
|
||||||
|
dependsOn wave N-1 (wave 0 dependsOn platform-config); prune+wait match the retired
|
||||||
|
monolith. SOPS decryption is added by the environment overlay patch, IDENTICALLY to the
|
||||||
|
former single `apps` Kustomization (clusters/dev patches each apps-wave-<N> by name) — the
|
||||||
|
base stays env-agnostic and references the out-of-band `sops-age` Secret by name only.
|
||||||
|
"""
|
||||||
|
dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}"
|
||||||
|
header = (
|
||||||
|
f"# Flux Kustomization: app-tier WAVE {wave}. Reconciles ./apps/wave-{wave} AFTER "
|
||||||
|
f"`{dep}` is healthy.\n"
|
||||||
|
f"# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its "
|
||||||
|
f"own wave + downstream:\n"
|
||||||
|
f"# wait:true health-gates ONLY this wave's resources, so a later wave can no longer "
|
||||||
|
f"stall an earlier one.\n"
|
||||||
|
f"# SOPS decryption is layered on per-environment (clusters/dev patches this "
|
||||||
|
f"Kustomization by name), identical to the retired `apps` Kustomization."
|
||||||
|
)
|
||||||
|
cr = {
|
||||||
|
"apiVersion": "kustomize.toolkit.fluxcd.io/v1",
|
||||||
|
"kind": "Kustomization",
|
||||||
|
"metadata": {"name": f"apps-wave-{wave}", "namespace": "flux-system"},
|
||||||
|
"spec": {
|
||||||
|
"interval": "10m",
|
||||||
|
"retryInterval": "2m",
|
||||||
|
"timeout": "5m",
|
||||||
|
"prune": True,
|
||||||
|
"wait": True,
|
||||||
|
"dependsOn": [{"name": dep}],
|
||||||
|
"sourceRef": {"kind": "GitRepository", "name": "flux-system"},
|
||||||
|
"path": f"./apps/wave-{wave}",
|
||||||
|
"postBuild": {
|
||||||
|
"substituteFrom": [{"kind": "ConfigMap", "name": "cluster-vars"}]
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
dumped = yaml.dump(cr, sort_keys=False, default_flow_style=False, width=4096)
|
||||||
|
return f"---\n{header}\n{dumped}"
|
||||||
|
|
||||||
|
|
||||||
|
def _discover_app_files(apps_dir: Path) -> set:
|
||||||
|
"""Basenames of every committed app-tier manifest under apps/ (flat OR in wave-<N>/).
|
||||||
|
|
||||||
|
Excludes the generated per-wave kustomization.yaml and any README. Deterministic.
|
||||||
|
"""
|
||||||
|
present = set()
|
||||||
|
for p in apps_dir.rglob("*.yaml"):
|
||||||
|
if p.name == "kustomization.yaml":
|
||||||
|
continue
|
||||||
|
present.add(p.name)
|
||||||
|
return present
|
||||||
|
|
||||||
|
|
||||||
|
def emit_waves(catalog_dir: Path, apps_dir: Path, clusters_dir: Path) -> dict:
|
||||||
|
"""Write apps/wave-<N>/kustomization.yaml + clusters/base/apps-wave-<N>.yaml.
|
||||||
|
|
||||||
|
Computes the wave plan from the catalog DAG and the app files present under apps/, then
|
||||||
|
(re)generates the per-wave kustomize files and the Flux Kustomization CRs. Idempotent:
|
||||||
|
it never moves or rewrites a service manifest (only its placement changes, done in git),
|
||||||
|
so the rendered bytes — and the L1 goldens — are untouched. Returns the wave plan.
|
||||||
|
"""
|
||||||
|
sds = _load_catalog(catalog_dir)
|
||||||
|
present = _discover_app_files(apps_dir)
|
||||||
|
plan = plan_apps_waves(sds, present)
|
||||||
|
for wave in sorted(plan):
|
||||||
|
wdir = apps_dir / f"wave-{wave}"
|
||||||
|
wdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
(wdir / "kustomization.yaml").write_text(
|
||||||
|
wave_kustomization_text(wave, plan[wave])
|
||||||
|
)
|
||||||
|
(clusters_dir / f"apps-wave-{wave}.yaml").write_text(
|
||||||
|
flux_wave_kustomization_text(wave)
|
||||||
|
)
|
||||||
|
return plan
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Self-test (deterministic, no cluster) — `render.py --selftest`
|
# Self-test (deterministic, no cluster) — `render.py --selftest`
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -1187,6 +1420,74 @@ def _selftest() -> int:
|
|||||||
"an SD without podSecurityContext must not gain a pod securityContext",
|
"an SD without podSecurityContext must not gain a pod securityContext",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# 7) wave partitioning: longest-path over the dependency DAG, CNPG-DB strictly below.
|
||||||
|
wsds = [
|
||||||
|
{"name": "a", "source": {"image": "x:1"}}, # leaf
|
||||||
|
{"name": "b", "source": {"image": "x:1"}}, # leaf
|
||||||
|
{"name": "c", "source": {"image": "x:1"},
|
||||||
|
"dependencies": [{"to": "a", "port": 1}, {"to": "b", "port": 1}]}, # wave 1
|
||||||
|
{"name": "d", "source": {"image": "x:1"},
|
||||||
|
"dependencies": [{"to": "c", "port": 1}]}, # wave 2
|
||||||
|
{"name": "svc", "source": {"image": "x:1"},
|
||||||
|
"dependencies": [{"to": "svc-db", "port": 5432}]}, # DB consumer
|
||||||
|
]
|
||||||
|
w = compute_waves(wsds)
|
||||||
|
check(w["a"] == 0 and w["b"] == 0, "leaf services must be wave 0")
|
||||||
|
check(w["c"] == 1, "a service depending on wave-0 nodes must be wave 1")
|
||||||
|
check(w["d"] == 2, "longest-path depth must be transitive (d -> c -> {a,b})")
|
||||||
|
check(
|
||||||
|
w["svc-db"] == 0 and w["svc"] == 1 and w["svc-db"] < w["svc"],
|
||||||
|
"a co-located <svc>-db must be a leaf STRICTLY BELOW its consumer (CNPG-DB rule)",
|
||||||
|
)
|
||||||
|
# cycle detection.
|
||||||
|
try:
|
||||||
|
compute_waves([
|
||||||
|
{"name": "p", "source": {"image": "x:1"}, "dependencies": [{"to": "q", "port": 1}]},
|
||||||
|
{"name": "q", "source": {"image": "x:1"}, "dependencies": [{"to": "p", "port": 1}]},
|
||||||
|
])
|
||||||
|
check(False, "compute_waves must raise on a dependency cycle")
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||||
|
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||||
|
present = {
|
||||||
|
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||||
|
"svc-db.yaml", "svc-db-app.sops.yaml", "gitea.yaml",
|
||||||
|
"gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
|
||||||
|
}
|
||||||
|
plan_sds = wsds + [{"name": "gitea", "source": {"image": "x:1"}},
|
||||||
|
{"name": "notonboarded", "source": {"image": "x:1"}}]
|
||||||
|
plan = plan_apps_waves(plan_sds, present)
|
||||||
|
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||||
|
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||||
|
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0,
|
||||||
|
"CNPG DB manifest + its owner secret must be in the DB's (lower) wave")
|
||||||
|
check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
|
||||||
|
"a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
|
||||||
|
check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")
|
||||||
|
check(flat.get("gitea-actions.yaml") == 1
|
||||||
|
and flat.get("gitea-actions-runner-secret.sops.yaml") == 1,
|
||||||
|
"gitea-actions (EXTRA_UNITS, above=True) must be one wave above its anchor gitea")
|
||||||
|
check("notonboarded.yaml" not in flat,
|
||||||
|
"a catalog SD with no committed manifest must NOT be placed (not onboarded)")
|
||||||
|
check("b.yaml" not in flat, "a file absent from the tree must never be placed")
|
||||||
|
|
||||||
|
# 7b) generated Flux Kustomization CR wiring (dependsOn chain + prune/wait/path).
|
||||||
|
import yaml as _yaml # already imported at module top; local alias for clarity
|
||||||
|
fw0 = _yaml.safe_load(flux_wave_kustomization_text(0))
|
||||||
|
fw2 = _yaml.safe_load(flux_wave_kustomization_text(2))
|
||||||
|
check(fw0["metadata"]["name"] == "apps-wave-0"
|
||||||
|
and fw0["spec"]["dependsOn"] == [{"name": "platform-config"}]
|
||||||
|
and fw0["spec"]["path"] == "./apps/wave-0",
|
||||||
|
"wave 0 Flux Kustomization must dependsOn platform-config and point at ./apps/wave-0")
|
||||||
|
check(fw2["spec"]["dependsOn"] == [{"name": "apps-wave-1"}]
|
||||||
|
and fw2["spec"]["prune"] is True and fw2["spec"]["wait"] is True,
|
||||||
|
"wave N Flux Kustomization must dependsOn wave N-1 with prune+wait (health-gated)")
|
||||||
|
kt = _yaml.safe_load(wave_kustomization_text(1, ["svc.yaml", "sonarr.yaml"]))
|
||||||
|
check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
|
||||||
|
"wave kustomization must list exactly the wave's resources")
|
||||||
|
|
||||||
if failures:
|
if failures:
|
||||||
for f in failures:
|
for f in failures:
|
||||||
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
|
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
|
||||||
@@ -1204,30 +1505,72 @@ def main(argv=None) -> int:
|
|||||||
ap.add_argument("--no-image-automation", action="store_true",
|
ap.add_argument("--no-image-automation", action="store_true",
|
||||||
help="Omit Flux ImageRepository/ImagePolicy CRs (for onboarding before the "
|
help="Omit Flux ImageRepository/ImagePolicy CRs (for onboarding before the "
|
||||||
"image-reflector/-automation controllers are installed)")
|
"image-reflector/-automation controllers are installed)")
|
||||||
|
ap.add_argument("--wave-layout", action="store_true",
|
||||||
|
help="With -o: place each rendered file under <out>/wave-<N>/<name>.yaml, "
|
||||||
|
"N = its dependency-ordered wave (compute_waves over the catalog)")
|
||||||
|
ap.add_argument("--emit-waves", action="store_true",
|
||||||
|
help="Generate the per-wave kustomize + Flux Kustomization CRs from the "
|
||||||
|
"catalog DAG and the committed apps tree (no manifest is rewritten)")
|
||||||
|
ap.add_argument("--print-waves", action="store_true",
|
||||||
|
help="Print the computed wave table (node -> wave) for the catalog and exit")
|
||||||
|
ap.add_argument("--catalog", default="catalog",
|
||||||
|
help="Catalog dir for --emit-waves/--print-waves/--wave-layout (default: catalog)")
|
||||||
|
ap.add_argument("--apps-dir", default="apps",
|
||||||
|
help="App-tier dir for --emit-waves (default: apps)")
|
||||||
|
ap.add_argument("--clusters-dir", default="clusters/base",
|
||||||
|
help="Dir to write clusters/base/apps-wave-<N>.yaml (default: clusters/base)")
|
||||||
args = ap.parse_args(argv)
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
if args.selftest:
|
if args.selftest:
|
||||||
return _selftest()
|
return _selftest()
|
||||||
|
|
||||||
|
if args.print_waves:
|
||||||
|
waves = compute_waves(_load_catalog(Path(args.catalog)))
|
||||||
|
for node in sorted(waves, key=lambda n: (waves[n], n)):
|
||||||
|
sys.stdout.write(f"wave {waves[node]} {node}\n")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if args.emit_waves:
|
||||||
|
plan = emit_waves(Path(args.catalog), Path(args.apps_dir), Path(args.clusters_dir))
|
||||||
|
for wave in sorted(plan):
|
||||||
|
sys.stderr.write(f"[emit-waves] wave {wave}: {', '.join(plan[wave])}\n")
|
||||||
|
sys.stderr.write(
|
||||||
|
f"[emit-waves] wrote {len(plan)} wave kustomization(s) under {args.apps_dir}/ "
|
||||||
|
f"and clusters CRs under {args.clusters_dir}/\n"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
image_automation = not args.no_image_automation
|
image_automation = not args.no_image_automation
|
||||||
|
|
||||||
|
def _out_path(outdir: Path, name: str, sds: list) -> Path:
|
||||||
|
if args.wave_layout:
|
||||||
|
wdir = outdir / f"wave-{service_wave(sds, name)}"
|
||||||
|
wdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
return wdir / f"{name}.yaml"
|
||||||
|
outdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
return outdir / f"{name}.yaml"
|
||||||
|
|
||||||
if args.all:
|
if args.all:
|
||||||
src = Path(args.all)
|
src = Path(args.all)
|
||||||
files = sorted(p for p in src.glob("*.yaml") if p.name != "render.py")
|
files = sorted(p for p in src.glob("*.yaml") if p.name != "render.py")
|
||||||
|
sds = _load_catalog(src) if args.wave_layout else []
|
||||||
for f in files:
|
for f in files:
|
||||||
text = render_file(f, image_automation=image_automation)
|
text = render_file(f, image_automation=image_automation)
|
||||||
if args.out_dir:
|
if args.out_dir:
|
||||||
outdir = Path(args.out_dir)
|
_out_path(Path(args.out_dir), load_sd(f)["name"], sds).write_text(text)
|
||||||
outdir.mkdir(parents=True, exist_ok=True)
|
|
||||||
name = load_sd(f)["name"]
|
|
||||||
(outdir / f"{name}.yaml").write_text(text)
|
|
||||||
else:
|
else:
|
||||||
sys.stdout.write(text)
|
sys.stdout.write(text)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
if not args.input:
|
if not args.input:
|
||||||
ap.error("provide a ServiceDefinition file, --all DIR, or --selftest")
|
ap.error("provide a ServiceDefinition file, --all DIR, --emit-waves, or --selftest")
|
||||||
sys.stdout.write(render_file(Path(args.input), image_automation=image_automation))
|
inp = Path(args.input)
|
||||||
|
text = render_file(inp, image_automation=image_automation)
|
||||||
|
if args.out_dir:
|
||||||
|
sds = _load_catalog(Path(args.catalog)) if args.wave_layout else []
|
||||||
|
_out_path(Path(args.out_dir), load_sd(inp)["name"], sds).write_text(text)
|
||||||
|
else:
|
||||||
|
sys.stdout.write(text)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+15
-3
@@ -16,7 +16,9 @@ clusters/
|
|||||||
base/ shared Flux objects (one authoring, both environments)
|
base/ shared Flux objects (one authoring, both environments)
|
||||||
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
|
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
|
||||||
platform.yaml Flux Kustomization → ./platform (reconciles first)
|
platform.yaml Flux Kustomization → ./platform (reconciles first)
|
||||||
apps.yaml Flux Kustomization → ./apps (dependsOn platform)
|
apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
|
||||||
|
apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
|
||||||
|
apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
|
||||||
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
|
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
|
||||||
kustomization.yaml
|
kustomization.yaml
|
||||||
dev/ ACTIVE build target — patches cluster-vars with dev values only
|
dev/ ACTIVE build target — patches cluster-vars with dev values only
|
||||||
@@ -60,10 +62,20 @@ GitRepository (flux-system)
|
|||||||
Kustomization: platform (./platform, substituteFrom cluster-vars)
|
Kustomization: platform (./platform, substituteFrom cluster-vars)
|
||||||
│ healthy
|
│ healthy
|
||||||
▼
|
▼
|
||||||
Kustomization: apps (./apps, dependsOn platform, substituteFrom cluster-vars)
|
Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
||||||
|
│ healthy
|
||||||
|
▼
|
||||||
|
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||||
```
|
```
|
||||||
|
|
||||||
`apps` `dependsOn` `platform` so CNI/storage/TLS/DNS/gateway exist before any workload reconciles.
|
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||||
|
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||||
|
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||||
|
wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
|
||||||
|
See [DESIGN-waves.md](../DESIGN-waves.md) for the wave table + cutover runbook.
|
||||||
|
|
||||||
## TODOs (later phases — do not action in Phase 0)
|
## TODOs (later phases — do not action in Phase 0)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization: app-tier WAVE 0. Reconciles ./apps/wave-0 AFTER `platform-config` is healthy.
|
||||||
|
# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
|
||||||
|
# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
|
||||||
|
# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: apps-wave-0
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 10m
|
||||||
|
retryInterval: 2m
|
||||||
|
timeout: 5m
|
||||||
|
prune: true
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
- name: platform-config
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./apps/wave-0
|
||||||
|
postBuild:
|
||||||
|
substituteFrom:
|
||||||
|
- kind: ConfigMap
|
||||||
|
name: cluster-vars
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization: app-tier WAVE 1. Reconciles ./apps/wave-1 AFTER `apps-wave-0` is healthy.
|
||||||
|
# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
|
||||||
|
# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
|
||||||
|
# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: apps-wave-1
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 10m
|
||||||
|
retryInterval: 2m
|
||||||
|
timeout: 5m
|
||||||
|
prune: true
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
- name: apps-wave-0
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./apps/wave-1
|
||||||
|
postBuild:
|
||||||
|
substituteFrom:
|
||||||
|
- kind: ConfigMap
|
||||||
|
name: cluster-vars
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization: app-tier WAVE 2. Reconciles ./apps/wave-2 AFTER `apps-wave-1` is healthy.
|
||||||
|
# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its own wave + downstream:
|
||||||
|
# wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one.
|
||||||
|
# SOPS decryption is layered on per-environment (clusters/dev patches this Kustomization by name), identical to the retired `apps` Kustomization.
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: apps-wave-2
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 10m
|
||||||
|
retryInterval: 2m
|
||||||
|
timeout: 5m
|
||||||
|
prune: true
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
- name: apps-wave-1
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./apps/wave-2
|
||||||
|
postBuild:
|
||||||
|
substituteFrom:
|
||||||
|
- kind: ConfigMap
|
||||||
|
name: cluster-vars
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
---
|
|
||||||
# Flux Kustomization: APPLICATION workloads. Reconciles ./apps AFTER platform is healthy.
|
|
||||||
# Phase 0 STUB. apps/ is owned by WS-APPS; this entrypoint is the GitOps contract that renders it.
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: apps
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
interval: 10m
|
|
||||||
retryInterval: 2m
|
|
||||||
timeout: 5m
|
|
||||||
# CUTOVER STEP 1 (temporary): prune disabled so that when this `apps` Kustomization object is
|
|
||||||
# removed in the wave-split commit, Flux does NOT garbage-collect the resources it owns (notably
|
|
||||||
# the CNPG database Clusters -> PVCs). The apps-wave-* Kustomizations then adopt them in place.
|
|
||||||
# This whole file is deleted in the follow-up wave-split commit.
|
|
||||||
prune: false
|
|
||||||
wait: true
|
|
||||||
# Ordering: the full platform (controllers + their configs) must be healthy first.
|
|
||||||
# platform-config dependsOn platform, so depending on platform-config transitively waits for both.
|
|
||||||
dependsOn:
|
|
||||||
- name: platform-config
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
path: ./apps
|
|
||||||
postBuild:
|
|
||||||
substituteFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: cluster-vars
|
|
||||||
# TODO(WS-SEC, Phase 1): enable SOPS decryption (ADR-0007) — reference age secret by name only.
|
|
||||||
@@ -7,5 +7,12 @@ resources:
|
|||||||
- flux-system.yaml # GitRepository (reconciliation source)
|
- flux-system.yaml # GitRepository (reconciliation source)
|
||||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||||
- apps.yaml # Flux Kustomization -> ./apps (dependsOn platform-config)
|
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||||
|
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||||
|
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||||
|
# GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave
|
||||||
|
# is introduced (a service deeper than any existing wave). See DESIGN-waves.md.
|
||||||
|
- apps-wave-0.yaml # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config)
|
||||||
|
- apps-wave-1.yaml # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0)
|
||||||
|
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||||
@@ -26,11 +26,37 @@ patches:
|
|||||||
provider: sops
|
provider: sops
|
||||||
secretRef:
|
secretRef:
|
||||||
name: sops-age
|
name: sops-age
|
||||||
|
# SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
|
||||||
|
# Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
|
||||||
|
# name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
|
||||||
|
# (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
|
||||||
- patch: |-
|
- patch: |-
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
name: apps
|
name: apps-wave-0
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
decryption:
|
||||||
|
provider: sops
|
||||||
|
secretRef:
|
||||||
|
name: sops-age
|
||||||
|
- patch: |-
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: apps-wave-1
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
decryption:
|
||||||
|
provider: sops
|
||||||
|
secretRef:
|
||||||
|
name: sops-age
|
||||||
|
- patch: |-
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: apps-wave-2
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
decryption:
|
decryption:
|
||||||
|
|||||||
Reference in new issue
Block a user