diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 90af052..9c745b2 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2561,3 +2561,130 @@ own status conditions and find out why the list is lying before acting. **Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`, `prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete. +## 2026-09-13 — CNPG primaries spread across nodes, and the honest answer: **there is no declarative fix** (branch `fix/cnpg-primary-spread`) + +The original ask was *"fix the database primaries being solely on prod 2"*. The instance half of +that was already done (`instances: 3` + `affinity.podAntiAffinityType: required` + +`primaryUpdateMethod: switchover`, commits `f338e91`/`e8d8601`/`cac413e`): every Cluster now has +exactly one pod per node and a synchronous standby, so no single node loss can take a database +down. The primary half was **not** done — and worse, that roll had *moved* the concentration +rather than removed it: + +``` +BEFORE (2026-09-13 04:19 UTC) AFTER (04:26 UTC) +keycloak-db-2 -> k3s-prod-1 primary keycloak-db-1 -> k3s-prod-2 primary +nextcloud-db-2 -> k3s-prod-1 primary nextcloud-db-1 -> k3s-prod-3 primary +semaphore-db-2 -> k3s-prod-1 primary semaphore-db-2 -> k3s-prod-1 primary +``` + +All three primaries had landed on `k3s-prod-1` — the node whose failing drive was just replaced and +whose etcd WAL shares a 2011 SSD with Longhorn. Losing it meant three simultaneous failovers. + +### The judgment call, made BEFORE touching anything + +**CNPG 1.30.0 has no primary-placement primitive, and the obvious workaround is worse than +nothing.** Established by walking the openAPIV3Schema of the CRD *actually installed on prod* +(`clusters.postgresql.cnpg.io/v1`, controller-gen v0.21.0) for +`/primar|promot|spread|topolog|prefer.*node|switchover/`, not from memory: + +- `spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey, + additionalPodAffinity,additionalPodAntiAffinity}` — every one is documented as applying to *"all + the cluster's pods"*. None is role-aware. +- `spec.topologySpreadConstraints` — spreads a Cluster's **own** pods; says nothing about how two + independent Clusters' primaries relate. +- `spec.replica.primary` — names which **Cluster object** is primary in a distributed + (`externalClusters`) topology. Not a node. +- `spec.primaryUpdateMethod` / `primaryUpdateStrategy` / `switchoverDelay` / `primaryLease` / + `postgresql.promotionTimeout` — govern **how and when** a promotion happens, never **where**. +- No other CNPG CRD is about scheduling (`backups`, `clusterimagecatalogs`, `databaseroles`, + `databases`, `failoverquorums`, `imagecatalogs`, `poolers`, `publications`, `scheduledbackups`, + `subscriptions`). +- **`targetPrimary` exists only under `status`, never `spec`** (confirmed against the live schema). + So `cnpg promote` is an imperative status patch by construction — there is nothing to write in git. + +And `additionalPodAntiAffinity` against `cnpg.io/instanceRole=primary` fails **twice**: (1) Kubernetes +pod anti-affinity is only ever `...IgnoredDuringExecution` — there is no `RequiredDuringExecution` +variant — while CNPG promotes by **relabelling an already-running pod**, so a scheduling-time +constraint cannot bind a promotion at all; (2) even at scheduling time, with 3 instances on 3 nodes +and `required` intra-cluster anti-affinity, excluding the node holding another primary leaves the +third instance **unschedulable** — trading blast radius for a permanent `readyInstances: 2`. + +**So option (a) — "accept it, every cluster already survives node loss" — was rejected as +incomplete, and a git-only "fix" was rejected as impossible.** What was done instead: reduce the +blast radius today, and make the drift **visible** so it is a known cheap runbook step rather than +something found by hand months later. The placement below is **current, not durable** — say so. + +### Why the drift is not hypothetical + +It already happened, from a routine declarative change: the `required` anti-affinity roll is +precisely what re-promoted all three clusters onto one node. `keycloak-db` reported +`Primary promotion time: 2026-09-08` — a re-promotion nobody asked for. No prod outage was staged to +"prove" non-durability; the repo's own history is the evidence. + +### The rebalance: two switchovers, not three + +`semaphore-db` was **deliberately left** with its primary on `k3s-prod-1`. One primary per node is +the goal, and semaphore is by far the lightest writer (its WAL backlog sat at 3 segments because it +writes almost nothing), so it is the right tenant for the node with the known etcd/Longhorn I/O +contention. Two switchovers instead of three = one less write interruption. + +**Disruption, measured — not estimated.** A `select txid_current()` probe (a real write: it assigns +an xid) ran against each `-rw` Service from a pod that stayed a replica throughout, password fed via +stdin so it never reached argv: + +| cluster | target | write outage | detail | +|---|---|---|---| +| `keycloak-db` (SSO) | `keycloak-db-1` on k3s-prod-2 | **~9.4 s** | last OK 04:20:24.634 → fail burst to 04:20:33.731; healthy at t+30s | +| `nextcloud-db` | `nextcloud-db-1` on k3s-prod-3 | **9.43 s** | 1402 samples @44 ms; ONE burst 04:25:46.854 → 04:25:56.287; healthy at t+25s | +| `semaphore-db` | *(none — kept on k3s-prod-1)* | **0 s** | stayed `readyInstances: 3` throughout | + +One contiguous failure burst each, no flapping, no second dip. Note the 30 s scrape interval showed +**no gap at all** for keycloak-db — the `-rw` endpoint was `ready` at every sample. A ~9 s write +outage is invisible to Prometheus at this resolution; only the sub-second probe sees it. + +After: all three `readyInstances: 3`, `Cluster in healthy state`, synchronous replication intact — +`synchronous_standby_names = ANY 1 (...)` in force on each primary with both standbys +`state=streaming sync_state=quorum`. + +### The durable part: `CNPGPrimariesCoLocated` + +Added to `platform/cnpg/alerts.yaml` (already wired via `platform-config/kustomization.yaml`, so +Flux reconciles it on merge — no hand-apply). + +**Neither metric family can answer the question alone**, which is the whole difficulty: +`cnpg_*` metrics carry `namespace`/`pod` but **no node**; and `kube_pod_labels` is **empty** on this +cluster (kube-state-metrics has no `--metric-labels-allowlist`), so `cnpg.io/instanceRole` is not in +Prometheus at all. The rule instead reads **CNPG's own definition of "who is primary"** — the +`-rw` Service selects the primary and nothing else, and +`kube_endpointslice_endpoints` carries both `endpoint_nodename` and `targetref_name`. Two +`label_replace`s lift those into `node`/`pod`; `* on (namespace, pod) group_left () cnpg_collector_up` +restricts the count to real CNPG instances so an unrelated `*-rw` Service cannot trigger it. +`cnpg_collector_up` is the right-hand side deliberately — it exists for **every** instance, unlike +the `cnpg_pg_stat_*` family which is missing entirely for `nextcloud-db`. + +**A Ready-node gate keeps it from becoming the next permanently-firing alert.** With three clusters +on three nodes, losing a node *necessarily* co-locates two primaries — correct behaviour, not a +defect — so the rule only speaks while every node is `Ready`, and re-arms itself when the node +returns. `for: 30m` rides out a switchover (seconds) and a full rolling update (minutes). +`severity: warning`, like everything else in that file: a resilience posture, never a page. + +**Proven in both directions, because this repo has been burned by both.** Live: the expression +returned `{node="k3s-prod-1"} => 3` before, and an empty vector after (while the inner join still +resolved all three primaries — quiet because the condition is false, not because the query went +blind). Loaded into prod's Prometheus as a throwaway twin (`for: 1m`), it was observed going +`pending` at +70 s and `firing` at +130 s, then deleted. And `platform/cnpg/alerts.promtool-test.yaml` +(`./run-alert-tests.sh`) pins all four behaviours under promtool 3.14.0 — the exact Prometheus +version on prod. **The tests were mutation-checked:** removing the `cnpg_collector_up` join, +removing the readiness gate, or shortening `for:` each makes a test *fail*, so they are not vacuous. + +### Debt / follow-ups + +- **Placement is current, not durable.** The next failover, node reboot or CNPG rolling update can + re-concentrate the primaries. That is now *reported*, not *prevented*. Rebalancing is a ~9 s + write interruption per cluster. +- **`promtool` is not in the baked toolchain** (`infra/devcontainer/versions.env`), so + `run-alert-tests.sh` fetches it on demand and no CI stage runs the alert unit tests. Adding + `PROMTOOL_VERSION` to `ci/versions.env` + the Containerfile would let `make -C ci gate` run them. +- **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`. + `cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description` + carries the full actionable procedure so the missing page is not load-bearing. diff --git a/platform/cnpg/alerts.promtool-test.yaml b/platform/cnpg/alerts.promtool-test.yaml new file mode 100644 index 0000000..2d18971 --- /dev/null +++ b/platform/cnpg/alerts.promtool-test.yaml @@ -0,0 +1,324 @@ +--- +# promtool unit tests for the CNPGPrimariesCoLocated rule in platform/cnpg/alerts.yaml. +# +# WHY THIS FILE EXISTS. This repo has been burned in both directions by alerts nobody tested: +# a permanently-firing `critical` (the VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml) +# and, worse, silence that looked like health (the 2026-09-06 CNPG archiving incident — 38h of +# ContinuousArchiving=False with no alert of any kind). CNPGPrimariesCoLocated is a JOIN across two +# metric families with a label_replace on each side and a node-readiness gate; "it looks right" is +# not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin +# BOTH directions plus the two ways the rule is meant to stay quiet. +# +# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule +# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN}, +# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer +# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the +# Prometheus actually running on prod (v3.14.0) if it is not already on PATH. +# +# WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares +# annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in +# here and re-pasting it on every wording change. Querying the synthetic `ALERTS` series instead +# proves exactly what matters — the alert reaches `pending` and then `firing` across the `for: 30m` +# boundary, with the right labels — and stays decoupled from annotation prose. +# +# THE SERIES BELOW ARE REAL SHAPES, not invented ones: every label on +# kube_endpointslice_endpoints / cnpg_collector_up / kube_node_status_condition / kube_node_info was +# read off live prod Prometheus on 2026-09-12 before being copied here. +rule_files: + - alerts.rules.yaml +evaluation_interval: 1m + +tests: + # ------------------------------------------------------------------------------------------- + # 1. THE BAD STATE THAT PROMPTED THIS RULE, exactly as prod looked on 2026-09-12: all three + # primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) on k3s-prod-1, all nodes Ready. + # Asserts the rule CAN fire, that it counts 3 (not 1, which a broken join would give), and + # that `for: 30m` is honoured — pending at 29m, firing at 31m. + - interval: 1m + name: three primaries on one node fires after 30m + input_series: + - series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.142",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.126",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.201",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + # All nine instances publish cnpg_collector_up — the join must pick out only the three that + # the -rw EndpointSlices point at, so the replicas are present here on purpose. + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-3",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-3",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-1",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-3",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # The join resolves each -rw endpoint to (namespace, primary pod, node) — the drill-down an + # operator actually wants. All three land on k3s-prod-1. + - expr: | + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + eval_time: 10m + exp_samples: + - labels: '{namespace="keycloak", node="k3s-prod-1", pod="keycloak-db-2"}' + value: 1 + - labels: '{namespace="nextcloud-db", node="k3s-prod-1", pod="nextcloud-db-2"}' + value: 1 + - labels: '{namespace="semaphore-db", node="k3s-prod-1", pod="semaphore-db-2"}' + value: 1 + # ... and the alert's own expression counts 3 of them on that one node. + - expr: | + count by (node) ( + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + ) > 1 + and on () + count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info) + eval_time: 10m + exp_samples: + - labels: '{node="k3s-prod-1"}' + value: 3 + # NOT PERMANENTLY FIRING, part 1: `for: 30m` really is waited out. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 29m + exp_samples: + - labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}' + value: 1 + # NOT PERMANENTLY INERT: it does reach `firing`. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 31m + exp_samples: + - labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}' + value: 1 + + # ------------------------------------------------------------------------------------------- + # 2. THE GOOD STATE — one primary per node, which is what the rebalancing switchovers on + # 2026-09-12 produced. The rule must be SILENT here, and silent for the right reason: the + # first assertion proves the join still resolves all three primaries (an alert that is quiet + # because its query returns nothing is the failure mode this repo keeps hitting), the second + # proves the >1 test then finds no over-loaded node. + - interval: 1m + name: one primary per node is silent + input_series: + - series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-1",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-1",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-3",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # Quiet for the RIGHT reason: the join is alive and finds all three primaries, one per node. + - expr: | + count by (node) ( + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + ) + eval_time: 10m + exp_samples: + - labels: '{node="k3s-prod-1"}' + value: 1 + - labels: '{node="k3s-prod-2"}' + value: 1 + - labels: '{node="k3s-prod-3"}' + value: 1 + # ... and no alert is ever raised, at any point in the hour. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 31m + exp_samples: [] + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 59m + exp_samples: [] + + # ------------------------------------------------------------------------------------------- + # 3. A NODE IS DOWN, so co-location is UNAVOIDABLE and correct — the rule must not nag during + # an outage. k3s-prod-3 is NotReady for the first 10 minutes and nextcloud-db has failed over + # onto k3s-prod-1, which now holds two primaries. Then the node comes back at t=11m and the + # co-location is NOT cleaned up, which is precisely the drift worth reporting: the rule must + # RE-ARM by itself and fire 30m later. This is the test that keeps the readiness gate from + # turning into a permanent mute. + - interval: 1m + name: node down suppresses, then re-arms when the node returns + input_series: + - series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x90' + - series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x90' + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x90' + - series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}' + values: '1x90' + - series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}' + values: '1x90' + - series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}' + values: '1x90' + # k3s-prod-3: NotReady (0) for t=0..10, Ready (1) from t=11 onwards. + - series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}' + values: '0x10 1x79' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x90' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x90' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x90' + promql_expr_test: + # Two primaries genuinely do share k3s-prod-1 at t=5m... + - expr: | + count by (node) ( + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + ) > 1 + eval_time: 5m + exp_samples: + - labels: '{node="k3s-prod-1"}' + value: 2 + # ... but the readiness gate suppresses the alert entirely while a node is down: not even pending. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 5m + exp_samples: [] + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 10m + exp_samples: [] + # Node back at t=11m; the `for` timer starts from scratch, so still only pending at t=25m. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 25m + exp_samples: + - labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}' + value: 1 + # RE-ARMED: 30m of full-cluster health with the co-location still there -> firing. + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 45m + exp_samples: + - labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}' + value: 1 + + # ------------------------------------------------------------------------------------------- + # 4. THE FALSE-POSITIVE GUARD. `endpointslice=~".+-rw-.+"` on its own is a loose match — any + # Service in the cluster named `something-rw` produces an EndpointSlice that matches it. Two + # such endpoints on one node must NOT be counted as CNPG primaries, which is what the + # `* on (namespace, pod) group_left () cnpg_collector_up` factor is for. Without that factor + # this test fires, so it is the thing that pins the join instead of a bare regex. + - interval: 1m + name: non-CNPG rw services are not counted + input_series: + - series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="myapp-rw-aaaaa",targetref_kind="Pod",targetref_name="myapp-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="other-rw-bbbbb",targetref_kind="Pod",targetref_name="other-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + # One real CNPG primary, on a different node — present so the test cannot pass merely because + # the whole query is empty. + - series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}' + values: '1x60' + - series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-1"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-2"}' + values: '1x60' + - series: 'kube_node_info{node="k3s-prod-3"}' + values: '1x60' + promql_expr_test: + # Only the CNPG primary survives the join — the two decoy `*-rw` endpoints are dropped. + - expr: | + count by (node) ( + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + ) + eval_time: 10m + exp_samples: + - labels: '{node="k3s-prod-2"}' + value: 1 + - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}' + eval_time: 45m + exp_samples: [] diff --git a/platform/cnpg/alerts.yaml b/platform/cnpg/alerts.yaml index 3c0621d..bdbb13a 100644 --- a/platform/cnpg/alerts.yaml +++ b/platform/cnpg/alerts.yaml @@ -265,3 +265,116 @@ spec: summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}" description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database." runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter" + # --------------------------------------------------------------------------------------- + # BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE. + # + # WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12). + # The operator's original ask was "the database primaries are all on one node". `instances: 3` + # + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has + # exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the + # PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2, + # semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous + # failovers — survivable, but the worst single-node blast radius the cluster can have. + # + # THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here + # (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking + # the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/: + # * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey, + # additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all + # the cluster's pods". None is role-aware: there is no way to say "this constraint is for + # whichever pod is currently primary". + # * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one + # Cluster that is the job podAntiAffinityType already does; it says nothing about how two + # INDEPENDENT Clusters' primaries relate. + # * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology + # (externalClusters / cross-k8s replica clusters). Not a node, not applicable here. + # * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease / + # postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the + # primary lands. + # Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about + # scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums, + # imagecatalogs, poolers, publications, scheduledbackups, subscriptions). + # + # AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled + # cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod + # (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no + # RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod + # (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time + # constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be + # actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity, + # excluding the node that holds another cluster's primary leaves the third instance + # UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2. + # + # SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches + # status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live) + # rebalances today and drifts again on the next failover, node reboot or rolling update. + # A switchover that silently un-does itself is not a fix, so what is committed here is the + # part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is + # a known, cheap runbook step instead of something discovered by hand months later. + # + # WHY THIS PROMQL. Neither family alone can answer the question: + # * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live). + # * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no + # --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all. + # The join below instead reads CNPG's OWN definition of "who is primary": the `-rw` + # Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint + # and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and + # `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the + # `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods + # that really are CNPG instances — so an unrelated Service that happens to be named `*-rw` + # cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by + # the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which + # is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above). + # + # THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With + # three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that + # is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the + # rule only speaks when every node is Ready, i.e. when the spread is actually achievable and + # the co-location is therefore avoidable drift. It re-arms by itself once the node returns. + # + # for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so + # 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while + # still reporting real drift the same hour. severity: warning, like every rule in this file: + # this is a resilience posture to correct at leisure, never a page. + # + # PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot + # fire is decoration and one that cannot stop is fatigue: + # * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned + # {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a + # throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and + # observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin + # was deleted afterwards; it was never committed. + # * GOES QUIET once rebalanced. After the two switchovers the same expression returned an + # empty vector, while the inner join still resolved all three primaries — semaphore-db-2 + # on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because + # the condition is false, NOT because the query went blind. + # * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with + # ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus + # version running here: co-located fires after 30m, one-per-node is silent, a NotReady + # node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw` + # Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the + # readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly. + - alert: CNPGPrimariesCoLocated + expr: | + count by (node) ( + max by (namespace, pod, node) ( + label_replace( + label_replace( + kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"}, + "pod", "$1", "targetref_name", "(.+)" + ), + "node", "$1", "endpoint_nodename", "(.+)" + ) + ) + * on (namespace, pod) group_left () + max by (namespace, pod) (cnpg_collector_up) + ) > 1 + and on () + count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info) + for: 30m + labels: + severity: warning + annotations: + summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}" + description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n ` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting." + runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread" diff --git a/platform/cnpg/run-alert-tests.sh b/platform/cnpg/run-alert-tests.sh new file mode 100755 index 0000000..c190c5d --- /dev/null +++ b/platform/cnpg/run-alert-tests.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +############################################################################### +# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml. +# +# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches: +# 1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus +# rules file (`groups:` at the top level). So `.spec` is extracted first. +# 2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not +# resolved in git. promtool does not care what the string says, but leaving a raw +# `${...}` in a rendered rules file is exactly the kind of thing that gets copied +# somewhere it DOES matter, so it is substituted with a placeholder here. +# Both derived files go to a temp dir — nothing generated is left in the worktree, because +# a generated file sitting next to its source is how a stale copy becomes the thing under +# test. +# +# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env). +# If it is not on PATH this fetches the version that matches the Prometheus actually +# running on prod, so the unit tests are evaluated by the same engine that will evaluate +# the rules for real. Override with PROMTOOL=/path/to/promtool. +# +# ./run-alert-tests.sh +############################################################################### +set -euo pipefail + +# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless +# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`). +PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}" + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +promtool="${PROMTOOL:-$(command -v promtool || true)}" +if [[ -z "$promtool" ]]; then + echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)" + tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64" + curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \ + | tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool" + promtool="$tmp/promtool" +fi +# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would +# break there, and it did during development. +promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")" +echo "==> $("$promtool" --version 2>&1 | head -1)" + +# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved. +sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \ + | python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \ + > "$tmp/alerts.rules.yaml" +cp "$here/alerts.promtool-test.yaml" "$tmp/" + +# 2. Syntax gate first — a rules file that does not parse would make every unit test +# "pass" vacuously by finding no rules at all. +echo "==> promtool check rules" +"$promtool" check rules "$tmp/alerts.rules.yaml" + +# 3. The unit tests. +echo "==> promtool test rules" +cd "$tmp" +"$promtool" test rules alerts.promtool-test.yaml