feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
df77affc23
commit
505083fded
4 files changed
+624
No files matched your search
@@ -0,0 +1,324 @@
|
||||
---
|
||||
# promtool unit tests for the CNPGPrimariesCoLocated rule in platform/cnpg/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. This repo has been burned in both directions by alerts nobody tested:
|
||||
# a permanently-firing `critical` (the VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml)
|
||||
# and, worse, silence that looked like health (the 2026-09-06 CNPG archiving incident — 38h of
|
||||
# ContinuousArchiving=False with no alert of any kind). CNPGPrimariesCoLocated is a JOIN across two
|
||||
# metric families with a label_replace on each side and a node-readiness gate; "it looks right" is
|
||||
# not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin
|
||||
# BOTH directions plus the two ways the rule is meant to stay quiet.
|
||||
#
|
||||
# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule
|
||||
# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN},
|
||||
# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer
|
||||
# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the
|
||||
# Prometheus actually running on prod (v3.14.0) if it is not already on PATH.
|
||||
#
|
||||
# WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares
|
||||
# annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in
|
||||
# here and re-pasting it on every wording change. Querying the synthetic `ALERTS` series instead
|
||||
# proves exactly what matters — the alert reaches `pending` and then `firing` across the `for: 30m`
|
||||
# boundary, with the right labels — and stays decoupled from annotation prose.
|
||||
#
|
||||
# THE SERIES BELOW ARE REAL SHAPES, not invented ones: every label on
|
||||
# kube_endpointslice_endpoints / cnpg_collector_up / kube_node_status_condition / kube_node_info was
|
||||
# read off live prod Prometheus on 2026-09-12 before being copied here.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 1. THE BAD STATE THAT PROMPTED THIS RULE, exactly as prod looked on 2026-09-12: all three
|
||||
# primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) on k3s-prod-1, all nodes Ready.
|
||||
# Asserts the rule CAN fire, that it counts 3 (not 1, which a broken join would give), and
|
||||
# that `for: 30m` is honoured — pending at 29m, firing at 31m.
|
||||
- interval: 1m
|
||||
name: three primaries on one node fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.142",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.126",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.201",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# All nine instances publish cnpg_collector_up — the join must pick out only the three that
|
||||
# the -rw EndpointSlices point at, so the replicas are present here on purpose.
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-3",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-3",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-1",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-3",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The join resolves each -rw endpoint to (namespace, primary pod, node) — the drill-down an
|
||||
# operator actually wants. All three land on k3s-prod-1.
|
||||
- expr: |
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{namespace="keycloak", node="k3s-prod-1", pod="keycloak-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="nextcloud-db", node="k3s-prod-1", pod="nextcloud-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="semaphore-db", node="k3s-prod-1", pod="semaphore-db-2"}'
|
||||
value: 1
|
||||
# ... and the alert's own expression counts 3 of them on that one node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 3
|
||||
# NOT PERMANENTLY FIRING, part 1: `for: 30m` really is waited out.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 29m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# NOT PERMANENTLY INERT: it does reach `firing`.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 2. THE GOOD STATE — one primary per node, which is what the rebalancing switchovers on
|
||||
# 2026-09-12 produced. The rule must be SILENT here, and silent for the right reason: the
|
||||
# first assertion proves the join still resolves all three primaries (an alert that is quiet
|
||||
# because its query returns nothing is the failure mode this repo keeps hitting), the second
|
||||
# proves the >1 test then finds no over-loaded node.
|
||||
- interval: 1m
|
||||
name: one primary per node is silent
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-1",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-1",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-3",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Quiet for the RIGHT reason: the join is alive and finds all three primaries, one per node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-3"}'
|
||||
value: 1
|
||||
# ... and no alert is ever raised, at any point in the hour.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 59m
|
||||
exp_samples: []
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 3. A NODE IS DOWN, so co-location is UNAVOIDABLE and correct — the rule must not nag during
|
||||
# an outage. k3s-prod-3 is NotReady for the first 10 minutes and nextcloud-db has failed over
|
||||
# onto k3s-prod-1, which now holds two primaries. Then the node comes back at t=11m and the
|
||||
# co-location is NOT cleaned up, which is precisely the drift worth reporting: the rule must
|
||||
# RE-ARM by itself and fire 30m later. This is the test that keeps the readiness gate from
|
||||
# turning into a permanent mute.
|
||||
- interval: 1m
|
||||
name: node down suppresses, then re-arms when the node returns
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
# k3s-prod-3: NotReady (0) for t=0..10, Ready (1) from t=11 onwards.
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '0x10 1x79'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x90'
|
||||
promql_expr_test:
|
||||
# Two primaries genuinely do share k3s-prod-1 at t=5m...
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 2
|
||||
# ... but the readiness gate suppresses the alert entirely while a node is down: not even pending.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 5m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 10m
|
||||
exp_samples: []
|
||||
# Node back at t=11m; the `for` timer starts from scratch, so still only pending at t=25m.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# RE-ARMED: 30m of full-cluster health with the co-location still there -> firing.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 4. THE FALSE-POSITIVE GUARD. `endpointslice=~".+-rw-.+"` on its own is a loose match — any
|
||||
# Service in the cluster named `something-rw` produces an EndpointSlice that matches it. Two
|
||||
# such endpoints on one node must NOT be counted as CNPG primaries, which is what the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor is for. Without that factor
|
||||
# this test fires, so it is the thing that pins the join instead of a bare regex.
|
||||
- interval: 1m
|
||||
name: non-CNPG rw services are not counted
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="myapp-rw-aaaaa",targetref_kind="Pod",targetref_name="myapp-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="other-rw-bbbbb",targetref_kind="Pod",targetref_name="other-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# One real CNPG primary, on a different node — present so the test cannot pass merely because
|
||||
# the whole query is empty.
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Only the CNPG primary survives the join — the two decoy `*-rw` endpoints are dropped.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples: []
|
||||
@@ -265,3 +265,116 @@ spec:
|
||||
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
|
||||
#
|
||||
# WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
|
||||
# The operator's original ask was "the database primaries are all on one node". `instances: 3`
|
||||
# + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
|
||||
# exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
|
||||
# PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
|
||||
# semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
|
||||
# failovers — survivable, but the worst single-node blast radius the cluster can have.
|
||||
#
|
||||
# THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
|
||||
# (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
|
||||
# the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
|
||||
# * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
||||
# additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
|
||||
# the cluster's pods". None is role-aware: there is no way to say "this constraint is for
|
||||
# whichever pod is currently primary".
|
||||
# * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
|
||||
# Cluster that is the job podAntiAffinityType already does; it says nothing about how two
|
||||
# INDEPENDENT Clusters' primaries relate.
|
||||
# * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
|
||||
# (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
|
||||
# * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
|
||||
# postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
|
||||
# primary lands.
|
||||
# Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
|
||||
# scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
|
||||
# imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
|
||||
#
|
||||
# AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
|
||||
# cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
|
||||
# (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
|
||||
# RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
|
||||
# (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
|
||||
# constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
|
||||
# actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
|
||||
# excluding the node that holds another cluster's primary leaves the third instance
|
||||
# UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
|
||||
#
|
||||
# SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
|
||||
# status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
|
||||
# rebalances today and drifts again on the next failover, node reboot or rolling update.
|
||||
# A switchover that silently un-does itself is not a fix, so what is committed here is the
|
||||
# part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
|
||||
# a known, cheap runbook step instead of something discovered by hand months later.
|
||||
#
|
||||
# WHY THIS PROMQL. Neither family alone can answer the question:
|
||||
# * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
|
||||
# * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
|
||||
# --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
|
||||
# The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
|
||||
# Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
|
||||
# and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
|
||||
# `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
|
||||
# that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
|
||||
# cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
|
||||
# the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
|
||||
# is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
|
||||
#
|
||||
# THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
|
||||
# three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
|
||||
# is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
|
||||
# rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
|
||||
# the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
|
||||
#
|
||||
# for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
|
||||
# 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
|
||||
# still reporting real drift the same hour. severity: warning, like every rule in this file:
|
||||
# this is a resilience posture to correct at leisure, never a page.
|
||||
#
|
||||
# PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
|
||||
# fire is decoration and one that cannot stop is fatigue:
|
||||
# * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
|
||||
# {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
|
||||
# throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
|
||||
# observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
|
||||
# was deleted afterwards; it was never committed.
|
||||
# * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
|
||||
# empty vector, while the inner join still resolved all three primaries — semaphore-db-2
|
||||
# on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
|
||||
# the condition is false, NOT because the query went blind.
|
||||
# * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
|
||||
# ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
|
||||
# version running here: co-located fires after 30m, one-per-node is silent, a NotReady
|
||||
# node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
|
||||
# Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
|
||||
# readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
|
||||
- alert: CNPGPrimariesCoLocated
|
||||
expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
|
||||
description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml.
|
||||
#
|
||||
# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches:
|
||||
# 1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus
|
||||
# rules file (`groups:` at the top level). So `.spec` is extracted first.
|
||||
# 2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not
|
||||
# resolved in git. promtool does not care what the string says, but leaving a raw
|
||||
# `${...}` in a rendered rules file is exactly the kind of thing that gets copied
|
||||
# somewhere it DOES matter, so it is substituted with a placeholder here.
|
||||
# Both derived files go to a temp dir — nothing generated is left in the worktree, because
|
||||
# a generated file sitting next to its source is how a stale copy becomes the thing under
|
||||
# test.
|
||||
#
|
||||
# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env).
|
||||
# If it is not on PATH this fetches the version that matches the Prometheus actually
|
||||
# running on prod, so the unit tests are evaluated by the same engine that will evaluate
|
||||
# the rules for real. Override with PROMTOOL=/path/to/promtool.
|
||||
#
|
||||
# ./run-alert-tests.sh
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless
|
||||
# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`).
|
||||
PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}"
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
promtool="${PROMTOOL:-$(command -v promtool || true)}"
|
||||
if [[ -z "$promtool" ]]; then
|
||||
echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)"
|
||||
tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
|
||||
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \
|
||||
| tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool"
|
||||
promtool="$tmp/promtool"
|
||||
fi
|
||||
# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would
|
||||
# break there, and it did during development.
|
||||
promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")"
|
||||
echo "==> $("$promtool" --version 2>&1 | head -1)"
|
||||
|
||||
# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved.
|
||||
sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \
|
||||
| python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \
|
||||
> "$tmp/alerts.rules.yaml"
|
||||
cp "$here/alerts.promtool-test.yaml" "$tmp/"
|
||||
|
||||
# 2. Syntax gate first — a rules file that does not parse would make every unit test
|
||||
# "pass" vacuously by finding no rules at all.
|
||||
echo "==> promtool check rules"
|
||||
"$promtool" check rules "$tmp/alerts.rules.yaml"
|
||||
|
||||
# 3. The unit tests.
|
||||
echo "==> promtool test rules"
|
||||
cd "$tmp"
|
||||
"$promtool" test rules alerts.promtool-test.yaml
|
||||
Reference in new issue
Block a user