From e8d8601ca682fe343b4ba39ff7256211aa45d261 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 23:35:47 -0500 Subject: [PATCH] feat(cnpg): required pod anti-affinity + switchover primary rolls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `instances: 3` alone does not keep the three instances apart. All three Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a scheduler HINT — two instances may share a node whenever that scores better, and they WILL on the path that matters: when CNPG recreates an instance after a node loss, the surviving nodes are the only candidates and one of them already runs an instance. `preferred` is how a 3-instance cluster decays back into the single-node failure domain the previous commit exists to remove. `required` makes it a scheduling constraint. On a 3-node cluster with no spare, the cost is that an instance whose node is gone stays Pending until a node returns. That is correct: a replacement squeezed onto an already-occupied node dies with that node while making the cluster LOOK three-way replicated. 2 of 3 instances still serve and CNPG still promotes. ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the affinity changes the instance pod spec, so CNPG performs a rolling update, and the CRD default for primaryUpdateMethod is `restart` (verified against the live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. With standbys now present, `switchover` promotes an already-updated streaming replica instead, reducing the roll to a connection reset. Known constraint documented in the manifest: under `switchover` the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile. Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have 3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a Longhorn replica on all three (replica-soft-anti-affinity=false), so any instance can attach from any node. DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling (249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0, still inside the ceiling, and only ~600 MiB per volume is really consumed. Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling. THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile, after the scale-out has produced two Ready standbys per Cluster, so the blast radius is one clean switchover per database rather than a scale-out and a roll at once. Expect a brief SSO interruption for keycloak-db. Verified: yamllint clean; both tiers render the new fields; field names, types and enum values checked against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical on prod and dev). Co-Authored-By: Claude Opus 5 (cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e) --- apps/wave-0/nextcloud-db.yaml | 15 +++++++++++ apps/wave-0/semaphore-db.yaml | 15 +++++++++++ platform/keycloak/postgres-cluster.yaml | 33 +++++++++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/apps/wave-0/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml index 2e5b51f..8a70f78 100644 --- a/apps/wave-0/nextcloud-db.yaml +++ b/apps/wave-0/nextcloud-db.yaml @@ -54,6 +54,21 @@ spec: inheritedMetadata: labels: app.kubernetes.io/name: nextcloud-db + # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the + # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and + # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short + # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance + # after a node loss it may stack two instances on one node — which is how `instances: 3` decays + # back into the single-node failure domain this change removes. `required` makes it a constraint; + # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still + # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes. + # + # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one + # switchover. Expect a single brief connection reset for Nextcloud. + affinity: + enablePodAntiAffinity: true + podAntiAffinityType: required + primaryUpdateMethod: switchover storage: storageClass: longhorn size: 5Gi diff --git a/apps/wave-0/semaphore-db.yaml b/apps/wave-0/semaphore-db.yaml index 65e364f..111bb7e 100644 --- a/apps/wave-0/semaphore-db.yaml +++ b/apps/wave-0/semaphore-db.yaml @@ -55,6 +55,21 @@ spec: inheritedMetadata: labels: app.kubernetes.io/name: semaphore-db + # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the + # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and + # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short + # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance + # after a node loss it may stack two instances on one node — which is how `instances: 3` decays + # back into the single-node failure domain this change removes. `required` makes it a constraint; + # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still + # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes. + # + # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one + # switchover. Expect a single brief connection reset for Semaphore. + affinity: + enablePodAntiAffinity: true + podAntiAffinityType: required + primaryUpdateMethod: switchover storage: storageClass: longhorn size: 5Gi diff --git a/platform/keycloak/postgres-cluster.yaml b/platform/keycloak/postgres-cluster.yaml index a50e61e..5e9fd3a 100644 --- a/platform/keycloak/postgres-cluster.yaml +++ b/platform/keycloak/postgres-cluster.yaml @@ -42,6 +42,39 @@ spec: # cause a switchover — see it for the blast radius.) instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:16.4 + # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes (2026-09-07). + # + # CNPG's default is `preferred`, which is only a scheduler HINT: the scheduler may put two + # instances on one node whenever that scores better, and it WILL do so on the path that matters — + # when CNPG recreates an instance after losing a node, the surviving nodes are the only candidates + # and one of them already runs an instance. So `preferred` is precisely how `instances: 3` decays + # back into the single-node failure domain this whole change exists to remove; nothing pins the + # good layout in place. `required` makes it a scheduling CONSTRAINT instead of a preference. + # + # THE TRADE, STATED PLAINLY: on a 3-node cluster with no spare node, an instance whose node is + # gone stays Pending until a node is available. That is the CORRECT behaviour here. A replacement + # squeezed onto a node that already hosts an instance buys nothing — both die with that node — + # while making the cluster LOOK three-way replicated. Availability is unaffected either way: + # 2 of 3 instances still serve and CNPG promotes one of them. + # + # Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes (checked + # read-only before merge). topologyKey defaults to kubernetes.io/hostname — the failure domain + # that matters here. enablePodAntiAffinity defaults to true; stated explicitly so this block can + # be read without knowing the default. + affinity: + enablePodAntiAffinity: true + podAntiAffinityType: required + # Roll the PRIMARY by SWITCHOVER, not by restarting it in place. CNPG's default for + # primaryUpdateMethod is `restart`, which deletes and recreates the primary pod — tens of seconds + # with no Postgres serving, and on THIS database that is an SSO outage. With standbys present + # (instances: 3 above) a switchover promotes an already-updated, already-streaming replica and + # demotes the old primary, so the interruption is a connection reset rather than a cold start. + # This is the field that makes the anti-affinity roll below a switchover rather than an outage. + # + # KNOWN CONSTRAINT: under `switchover` the operator REJECTS an update that changes `imageName` + # AND a postgresql parameter in the same reconcile. Renovate bumping the image alone is fine; + # never combine an image bump with a postgresql.parameters change in one commit. + primaryUpdateMethod: switchover storage: storageClass: longhorn size: 5Gi