Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
apps
App-tier manifests, rendered from catalog/*.yaml ServiceDefinitions by catalog/render.py
and split into dependency-ordered waves (wave-<N>/).
- Each
wave-<N>/is reconciled by its own Flux Kustomizationapps-wave-<N>(clusters/base/apps-wave-<N>.yaml), whichdependsOnwave N-1 (wave 0 dependsOnplatform-config) and health-gates (wait: true) ONLY that wave. A crashlooping workload in a later wave can no longer stall an earlier wave or block new revisions. wave-<N>/kustomization.yamlandclusters/base/apps-wave-<N>.yamlare generated — runpython3 catalog/render.py --emit-wavesafter onboarding a service; do not hand-edit them.- Wave assignment = longest dependency-path depth over the catalog
dependencies:graph; a co-located CNPG database<svc>-dbis a leaf and always lands strictly below its consumer.
See ../DESIGN-waves.md for the wave table, Flux topology, and the
cutover runbook.