From f338e91770e1af4a0d8d7eda798f19f0e1add6f3 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 23:33:40 -0500 Subject: [PATCH] =?UTF-8?q?feat(cnpg):=20instances=203=20on=20all=20three?= =?UTF-8?q?=20Clusters=20=E2=80=94=20a=20standby=20to=20fail=20over=20to?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02) --- apps/wave-0/nextcloud-db.yaml | 15 ++++++++--- apps/wave-0/semaphore-db.yaml | 15 ++++++++--- platform/keycloak/postgres-cluster.yaml | 33 ++++++++++++++++++++++--- 3 files changed, 51 insertions(+), 12 deletions(-) diff --git a/apps/wave-0/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml index 754ba03..2e5b51f 100644 --- a/apps/wave-0/nextcloud-db.yaml +++ b/apps/wave-0/nextcloud-db.yaml @@ -37,10 +37,17 @@ metadata: name: nextcloud-db namespace: nextcloud-db spec: - # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as - # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level - # HA as a future hardening. - instances: 1 + # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see + # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way + # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG + # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and + # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service. + # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real + # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup. + # + # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by + # pg_basebackup from the running primary, which is neither restarted nor disconnected. + instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:16.4 # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02). diff --git a/apps/wave-0/semaphore-db.yaml b/apps/wave-0/semaphore-db.yaml index 3991548..65e364f 100644 --- a/apps/wave-0/semaphore-db.yaml +++ b/apps/wave-0/semaphore-db.yaml @@ -38,10 +38,17 @@ metadata: name: semaphore-db namespace: semaphore-db spec: - # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as - # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for - # Postgres-level HA as a future hardening. - instances: 1 + # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db / + # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: + # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with + # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore + # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint + # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True + # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup. + # + # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by + # pg_basebackup from the running primary, which is neither restarted nor disconnected. + instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:16.4 # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02). diff --git a/platform/keycloak/postgres-cluster.yaml b/platform/keycloak/postgres-cluster.yaml index bcfd330..a50e61e 100644 --- a/platform/keycloak/postgres-cluster.yaml +++ b/platform/keycloak/postgres-cluster.yaml @@ -12,10 +12,35 @@ metadata: name: keycloak-db namespace: keycloak spec: - # Single Postgres instance on 3-replica Longhorn is acceptable for the home scale (ADR-0014: - # "single replica with DB on Longhorn + backups is acceptable"). Raise to 3 for Postgres-level HA - # as a future hardening (prod) — the storage is already 3-way replicated by Longhorn. - instances: 1 + # THREE instances — one primary + two hot standbys, one per node (2026-09-07). + # + # WAS `instances: 1`, on ADR-0014's "single replica with DB on Longhorn + backups is acceptable". + # Longhorn's 3-way replication saves the DATA when a node dies; it does NOT save the SERVICE. With + # one instance, losing the node means rescheduling the pod, re-attaching the volume and replaying + # WAL before Postgres will accept a connection again — minutes, and only if scheduling cooperates. + # + # WHY NOW (the 2026-09-07 incident): all three CNPG primaries — keycloak-db, nextcloud-db and + # semaphore-db — were co-resident on k3s-prod-2, because with `instances: 1` nothing in the spec + # ever said they must not be. One node took out SSO, Nextcloud and Semaphore together, and with + # SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give this + # Cluster something to fail OVER to: CNPG promotes a streaming standby in seconds and repoints the + # -rw Service, with no volume re-attach and no restore. The `affinity.podAntiAffinityType: + # required` below is what makes the three land on three DIFFERENT nodes. + # + # UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Replicas copy corruption and deletions + # faithfully; only the Barman archive below is a backup. Adding standbys while archiving was + # broken would have multiplied the disk-full clock described further down, so this change waited + # for all three Clusters to report ContinuousArchiving=True with a real firstRecoverabilityPoint. + # + # COST: two more 5Gi PVCs here (Longhorn thin-provisions; the live pgdata is ~600 MiB), ~30 GiB + # PROVISIONED across all three DBs and well under 3 GiB actually consumed. Longhorn allows 200% + # over-provisioning on both clusters and both have headroom. + # + # APPLYING THIS IS NON-DISRUPTIVE: CNPG scales out by creating -2 then -3 sequentially, each + # seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are + # not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES + # cause a switchover — see it for the blast radius.) + instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:16.4 storage: storageClass: longhorn