feat(cnpg): instances 3 on all three Clusters — a standby to fail over to

All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.

With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.

Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.

UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).

Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.

DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).

Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.

Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)

apps/wave-0/nextcloud-db.yaml

@@ -37,10 +37,17 @@ metadata:
   name: nextcloud-db
   namespace: nextcloud-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
-  # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
-  # HA as a future hardening.
-  instances: 1
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
+  # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
+  # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
+  # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
+  # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
+  # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
+  # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).

apps/wave-0/semaphore-db.yaml

@@ -38,10 +38,17 @@ metadata:
   name: semaphore-db
   namespace: semaphore-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
-  # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
-  # Postgres-level HA as a future hardening.
-  instances: 1
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
+  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
+  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
+  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
+  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
+  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
+  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).

platform/keycloak/postgres-cluster.yaml

@@ -12,10 +12,35 @@ metadata:
   name: keycloak-db
   namespace: keycloak
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable for the home scale (ADR-0014:
-  # "single replica with DB on Longhorn + backups is acceptable"). Raise to 3 for Postgres-level HA
-  # as a future hardening (prod) — the storage is already 3-way replicated by Longhorn.
-  instances: 1
+  # THREE instances — one primary + two hot standbys, one per node (2026-09-07).
+  #
+  # WAS `instances: 1`, on ADR-0014's "single replica with DB on Longhorn + backups is acceptable".
+  # Longhorn's 3-way replication saves the DATA when a node dies; it does NOT save the SERVICE. With
+  # one instance, losing the node means rescheduling the pod, re-attaching the volume and replaying
+  # WAL before Postgres will accept a connection again — minutes, and only if scheduling cooperates.
+  #
+  # WHY NOW (the 2026-09-07 incident): all three CNPG primaries — keycloak-db, nextcloud-db and
+  # semaphore-db — were co-resident on k3s-prod-2, because with `instances: 1` nothing in the spec
+  # ever said they must not be. One node took out SSO, Nextcloud and Semaphore together, and with
+  # SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give this
+  # Cluster something to fail OVER to: CNPG promotes a streaming standby in seconds and repoints the
+  # -rw Service, with no volume re-attach and no restore. The `affinity.podAntiAffinityType:
+  # required` below is what makes the three land on three DIFFERENT nodes.
+  #
+  # UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Replicas copy corruption and deletions
+  # faithfully; only the Barman archive below is a backup. Adding standbys while archiving was
+  # broken would have multiplied the disk-full clock described further down, so this change waited
+  # for all three Clusters to report ContinuousArchiving=True with a real firstRecoverabilityPoint.
+  #
+  # COST: two more 5Gi PVCs here (Longhorn thin-provisions; the live pgdata is ~600 MiB), ~30 GiB
+  # PROVISIONED across all three DBs and well under 3 GiB actually consumed. Longhorn allows 200%
+  # over-provisioning on both clusters and both have headroom.
+  #
+  # APPLYING THIS IS NON-DISRUPTIVE: CNPG scales out by creating -2 then -3 sequentially, each
+  # seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are
+  # not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES
+  # cause a switchover — see it for the blast radius.)
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   storage:
     storageClass: longhorn