jakman85
cac413e87a
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)
apps
App-tier manifests, rendered from catalog/*.yaml ServiceDefinitions by catalog/render.py
and split into dependency-ordered waves (wave-<N>/).
- Each
wave-<N>/is reconciled by its own Flux Kustomizationapps-wave-<N>(clusters/base/apps-wave-<N>.yaml), whichdependsOnwave N-1 (wave 0 dependsOnplatform-config) and health-gates (wait: true) ONLY that wave. A crashlooping workload in a later wave can no longer stall an earlier wave or block new revisions. wave-<N>/kustomization.yamlandclusters/base/apps-wave-<N>.yamlare generated — runpython3 catalog/render.py --emit-wavesafter onboarding a service; do not hand-edit them.- Wave assignment = longest dependency-path depth over the catalog
dependencies:graph; a co-located CNPG database<svc>-dbis a leaf and always lands strictly below its consumer.
See ../DESIGN-waves.md for the wave table, Flux topology, and the
cutover runbook.