feat(cnpg): instances 3 on all three Clusters — a standby to fail over to

All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.

With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.

Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.

UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).

Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.

DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).

Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.

Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)

apps/wave-0/nextcloud-db.yaml

@@ -37,10 +37,17 @@ metadata:
   name: nextcloud-db
   namespace: nextcloud-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
-  # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
-  # HA as a future hardening.
-  instances: 1
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
+  # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
+  # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
+  # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
+  # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
+  # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
+  # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).

apps/wave-0/semaphore-db.yaml

@@ -38,10 +38,17 @@ metadata:
   name: semaphore-db
   namespace: semaphore-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
-  # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
-  # Postgres-level HA as a future hardening.
-  instances: 1
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
+  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
+  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
+  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
+  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
+  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
+  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).