jakman85andClaude Opus 5
505083fded
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
325 lines
20 KiB
YAML
325 lines
20 KiB
YAML
---
|
|
# promtool unit tests for the CNPGPrimariesCoLocated rule in platform/cnpg/alerts.yaml.
|
|
#
|
|
# WHY THIS FILE EXISTS. This repo has been burned in both directions by alerts nobody tested:
|
|
# a permanently-firing `critical` (the VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml)
|
|
# and, worse, silence that looked like health (the 2026-09-06 CNPG archiving incident — 38h of
|
|
# ContinuousArchiving=False with no alert of any kind). CNPGPrimariesCoLocated is a JOIN across two
|
|
# metric families with a label_replace on each side and a node-readiness gate; "it looks right" is
|
|
# not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin
|
|
# BOTH directions plus the two ways the rule is meant to stay quiet.
|
|
#
|
|
# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule
|
|
# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN},
|
|
# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer
|
|
# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the
|
|
# Prometheus actually running on prod (v3.14.0) if it is not already on PATH.
|
|
#
|
|
# WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares
|
|
# annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in
|
|
# here and re-pasting it on every wording change. Querying the synthetic `ALERTS` series instead
|
|
# proves exactly what matters — the alert reaches `pending` and then `firing` across the `for: 30m`
|
|
# boundary, with the right labels — and stays decoupled from annotation prose.
|
|
#
|
|
# THE SERIES BELOW ARE REAL SHAPES, not invented ones: every label on
|
|
# kube_endpointslice_endpoints / cnpg_collector_up / kube_node_status_condition / kube_node_info was
|
|
# read off live prod Prometheus on 2026-09-12 before being copied here.
|
|
rule_files:
|
|
- alerts.rules.yaml
|
|
evaluation_interval: 1m
|
|
|
|
tests:
|
|
# -------------------------------------------------------------------------------------------
|
|
# 1. THE BAD STATE THAT PROMPTED THIS RULE, exactly as prod looked on 2026-09-12: all three
|
|
# primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) on k3s-prod-1, all nodes Ready.
|
|
# Asserts the rule CAN fire, that it counts 3 (not 1, which a broken join would give), and
|
|
# that `for: 30m` is honoured — pending at 29m, firing at 31m.
|
|
- interval: 1m
|
|
name: three primaries on one node fires after 30m
|
|
input_series:
|
|
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.142",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.126",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.201",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
# All nine instances publish cnpg_collector_up — the join must pick out only the three that
|
|
# the -rw EndpointSlices point at, so the replicas are present here on purpose.
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-3",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-3",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-1",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-3",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-1"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-2"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-3"}'
|
|
values: '1x60'
|
|
promql_expr_test:
|
|
# The join resolves each -rw endpoint to (namespace, primary pod, node) — the drill-down an
|
|
# operator actually wants. All three land on k3s-prod-1.
|
|
- expr: |
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
eval_time: 10m
|
|
exp_samples:
|
|
- labels: '{namespace="keycloak", node="k3s-prod-1", pod="keycloak-db-2"}'
|
|
value: 1
|
|
- labels: '{namespace="nextcloud-db", node="k3s-prod-1", pod="nextcloud-db-2"}'
|
|
value: 1
|
|
- labels: '{namespace="semaphore-db", node="k3s-prod-1", pod="semaphore-db-2"}'
|
|
value: 1
|
|
# ... and the alert's own expression counts 3 of them on that one node.
|
|
- expr: |
|
|
count by (node) (
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
) > 1
|
|
and on ()
|
|
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
|
eval_time: 10m
|
|
exp_samples:
|
|
- labels: '{node="k3s-prod-1"}'
|
|
value: 3
|
|
# NOT PERMANENTLY FIRING, part 1: `for: 30m` really is waited out.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 29m
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
|
value: 1
|
|
# NOT PERMANENTLY INERT: it does reach `firing`.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 31m
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
|
value: 1
|
|
|
|
# -------------------------------------------------------------------------------------------
|
|
# 2. THE GOOD STATE — one primary per node, which is what the rebalancing switchovers on
|
|
# 2026-09-12 produced. The rule must be SILENT here, and silent for the right reason: the
|
|
# first assertion proves the join still resolves all three primaries (an alert that is quiet
|
|
# because its query returns nothing is the failure mode this repo keeps hitting), the second
|
|
# proves the >1 test then finds no over-loaded node.
|
|
- interval: 1m
|
|
name: one primary per node is silent
|
|
input_series:
|
|
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-1",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-1",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-3",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-1"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-2"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-3"}'
|
|
values: '1x60'
|
|
promql_expr_test:
|
|
# Quiet for the RIGHT reason: the join is alive and finds all three primaries, one per node.
|
|
- expr: |
|
|
count by (node) (
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
)
|
|
eval_time: 10m
|
|
exp_samples:
|
|
- labels: '{node="k3s-prod-1"}'
|
|
value: 1
|
|
- labels: '{node="k3s-prod-2"}'
|
|
value: 1
|
|
- labels: '{node="k3s-prod-3"}'
|
|
value: 1
|
|
# ... and no alert is ever raised, at any point in the hour.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 31m
|
|
exp_samples: []
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 59m
|
|
exp_samples: []
|
|
|
|
# -------------------------------------------------------------------------------------------
|
|
# 3. A NODE IS DOWN, so co-location is UNAVOIDABLE and correct — the rule must not nag during
|
|
# an outage. k3s-prod-3 is NotReady for the first 10 minutes and nextcloud-db has failed over
|
|
# onto k3s-prod-1, which now holds two primaries. Then the node comes back at t=11m and the
|
|
# co-location is NOT cleaned up, which is precisely the drift worth reporting: the rule must
|
|
# RE-ARM by itself and fire 30m later. This is the test that keeps the readiness gate from
|
|
# turning into a permanent mute.
|
|
- interval: 1m
|
|
name: node down suppresses, then re-arms when the node returns
|
|
input_series:
|
|
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x90'
|
|
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x90'
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x90'
|
|
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
|
values: '1x90'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
|
values: '1x90'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
|
values: '1x90'
|
|
# k3s-prod-3: NotReady (0) for t=0..10, Ready (1) from t=11 onwards.
|
|
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
|
values: '0x10 1x79'
|
|
- series: 'kube_node_info{node="k3s-prod-1"}'
|
|
values: '1x90'
|
|
- series: 'kube_node_info{node="k3s-prod-2"}'
|
|
values: '1x90'
|
|
- series: 'kube_node_info{node="k3s-prod-3"}'
|
|
values: '1x90'
|
|
promql_expr_test:
|
|
# Two primaries genuinely do share k3s-prod-1 at t=5m...
|
|
- expr: |
|
|
count by (node) (
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
) > 1
|
|
eval_time: 5m
|
|
exp_samples:
|
|
- labels: '{node="k3s-prod-1"}'
|
|
value: 2
|
|
# ... but the readiness gate suppresses the alert entirely while a node is down: not even pending.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 5m
|
|
exp_samples: []
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 10m
|
|
exp_samples: []
|
|
# Node back at t=11m; the `for` timer starts from scratch, so still only pending at t=25m.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 25m
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
|
value: 1
|
|
# RE-ARMED: 30m of full-cluster health with the co-location still there -> firing.
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 45m
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
|
value: 1
|
|
|
|
# -------------------------------------------------------------------------------------------
|
|
# 4. THE FALSE-POSITIVE GUARD. `endpointslice=~".+-rw-.+"` on its own is a loose match — any
|
|
# Service in the cluster named `something-rw` produces an EndpointSlice that matches it. Two
|
|
# such endpoints on one node must NOT be counted as CNPG primaries, which is what the
|
|
# `* on (namespace, pod) group_left () cnpg_collector_up` factor is for. Without that factor
|
|
# this test fires, so it is the thing that pins the join instead of a bare regex.
|
|
- interval: 1m
|
|
name: non-CNPG rw services are not counted
|
|
input_series:
|
|
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="myapp-rw-aaaaa",targetref_kind="Pod",targetref_name="myapp-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="other-rw-bbbbb",targetref_kind="Pod",targetref_name="other-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
# One real CNPG primary, on a different node — present so the test cannot pass merely because
|
|
# the whole query is empty.
|
|
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
|
values: '1x60'
|
|
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-1"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-2"}'
|
|
values: '1x60'
|
|
- series: 'kube_node_info{node="k3s-prod-3"}'
|
|
values: '1x60'
|
|
promql_expr_test:
|
|
# Only the CNPG primary survives the join — the two decoy `*-rw` endpoints are dropped.
|
|
- expr: |
|
|
count by (node) (
|
|
max by (namespace, pod, node) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
|
"pod", "$1", "targetref_name", "(.+)"
|
|
),
|
|
"node", "$1", "endpoint_nodename", "(.+)"
|
|
)
|
|
)
|
|
* on (namespace, pod) group_left ()
|
|
max by (namespace, pod) (cnpg_collector_up)
|
|
)
|
|
eval_time: 10m
|
|
exp_samples:
|
|
- labels: '{node="k3s-prod-2"}'
|
|
value: 1
|
|
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
|
eval_time: 45m
|
|
exp_samples: []
|