merge: CNPG primaries spread one per node, and the condition is now reported
All three primaries had re-concentrated onto prod-1 (the node with known etcd/Longhorn I/O contention), so losing it meant three simultaneous failovers. Two switchovers rebalanced them; semaphore-db deliberately stays on prod-1 as the lightest writer, saving a third write interruption. CNPG offers no role-aware scheduling — targetPrimary exists only under status — so the placement is CURRENT, NOT DURABLE: the next failover or rolling update can re-concentrate it. What is durable is CNPGPrimariesCoLocated, which reports the condition instead of leaving it to be found by hand.
This commit is contained in:
jakman85 committed
commit
9e7d919d12
4 files changed
+624
No files matched your search
@@ -2561,3 +2561,130 @@ own status conditions and find out why the list is lying before acting.
|
||||
**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a
|
||||
backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`,
|
||||
`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.
|
||||
## 2026-09-13 — CNPG primaries spread across nodes, and the honest answer: **there is no declarative fix** (branch `fix/cnpg-primary-spread`)
|
||||
|
||||
The original ask was *"fix the database primaries being solely on prod 2"*. The instance half of
|
||||
that was already done (`instances: 3` + `affinity.podAntiAffinityType: required` +
|
||||
`primaryUpdateMethod: switchover`, commits `f338e91`/`e8d8601`/`cac413e`): every Cluster now has
|
||||
exactly one pod per node and a synchronous standby, so no single node loss can take a database
|
||||
down. The primary half was **not** done — and worse, that roll had *moved* the concentration
|
||||
rather than removed it:
|
||||
|
||||
```
|
||||
BEFORE (2026-09-13 04:19 UTC) AFTER (04:26 UTC)
|
||||
keycloak-db-2 -> k3s-prod-1 primary keycloak-db-1 -> k3s-prod-2 primary
|
||||
nextcloud-db-2 -> k3s-prod-1 primary nextcloud-db-1 -> k3s-prod-3 primary
|
||||
semaphore-db-2 -> k3s-prod-1 primary semaphore-db-2 -> k3s-prod-1 primary
|
||||
```
|
||||
|
||||
All three primaries had landed on `k3s-prod-1` — the node whose failing drive was just replaced and
|
||||
whose etcd WAL shares a 2011 SSD with Longhorn. Losing it meant three simultaneous failovers.
|
||||
|
||||
### The judgment call, made BEFORE touching anything
|
||||
|
||||
**CNPG 1.30.0 has no primary-placement primitive, and the obvious workaround is worse than
|
||||
nothing.** Established by walking the openAPIV3Schema of the CRD *actually installed on prod*
|
||||
(`clusters.postgresql.cnpg.io/v1`, controller-gen v0.21.0) for
|
||||
`/primar|promot|spread|topolog|prefer.*node|switchover/`, not from memory:
|
||||
|
||||
- `spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
||||
additionalPodAffinity,additionalPodAntiAffinity}` — every one is documented as applying to *"all
|
||||
the cluster's pods"*. None is role-aware.
|
||||
- `spec.topologySpreadConstraints` — spreads a Cluster's **own** pods; says nothing about how two
|
||||
independent Clusters' primaries relate.
|
||||
- `spec.replica.primary` — names which **Cluster object** is primary in a distributed
|
||||
(`externalClusters`) topology. Not a node.
|
||||
- `spec.primaryUpdateMethod` / `primaryUpdateStrategy` / `switchoverDelay` / `primaryLease` /
|
||||
`postgresql.promotionTimeout` — govern **how and when** a promotion happens, never **where**.
|
||||
- No other CNPG CRD is about scheduling (`backups`, `clusterimagecatalogs`, `databaseroles`,
|
||||
`databases`, `failoverquorums`, `imagecatalogs`, `poolers`, `publications`, `scheduledbackups`,
|
||||
`subscriptions`).
|
||||
- **`targetPrimary` exists only under `status`, never `spec`** (confirmed against the live schema).
|
||||
So `cnpg promote` is an imperative status patch by construction — there is nothing to write in git.
|
||||
|
||||
And `additionalPodAntiAffinity` against `cnpg.io/instanceRole=primary` fails **twice**: (1) Kubernetes
|
||||
pod anti-affinity is only ever `...IgnoredDuringExecution` — there is no `RequiredDuringExecution`
|
||||
variant — while CNPG promotes by **relabelling an already-running pod**, so a scheduling-time
|
||||
constraint cannot bind a promotion at all; (2) even at scheduling time, with 3 instances on 3 nodes
|
||||
and `required` intra-cluster anti-affinity, excluding the node holding another primary leaves the
|
||||
third instance **unschedulable** — trading blast radius for a permanent `readyInstances: 2`.
|
||||
|
||||
**So option (a) — "accept it, every cluster already survives node loss" — was rejected as
|
||||
incomplete, and a git-only "fix" was rejected as impossible.** What was done instead: reduce the
|
||||
blast radius today, and make the drift **visible** so it is a known cheap runbook step rather than
|
||||
something found by hand months later. The placement below is **current, not durable** — say so.
|
||||
|
||||
### Why the drift is not hypothetical
|
||||
|
||||
It already happened, from a routine declarative change: the `required` anti-affinity roll is
|
||||
precisely what re-promoted all three clusters onto one node. `keycloak-db` reported
|
||||
`Primary promotion time: 2026-09-08` — a re-promotion nobody asked for. No prod outage was staged to
|
||||
"prove" non-durability; the repo's own history is the evidence.
|
||||
|
||||
### The rebalance: two switchovers, not three
|
||||
|
||||
`semaphore-db` was **deliberately left** with its primary on `k3s-prod-1`. One primary per node is
|
||||
the goal, and semaphore is by far the lightest writer (its WAL backlog sat at 3 segments because it
|
||||
writes almost nothing), so it is the right tenant for the node with the known etcd/Longhorn I/O
|
||||
contention. Two switchovers instead of three = one less write interruption.
|
||||
|
||||
**Disruption, measured — not estimated.** A `select txid_current()` probe (a real write: it assigns
|
||||
an xid) ran against each `-rw` Service from a pod that stayed a replica throughout, password fed via
|
||||
stdin so it never reached argv:
|
||||
|
||||
| cluster | target | write outage | detail |
|
||||
|---|---|---|---|
|
||||
| `keycloak-db` (SSO) | `keycloak-db-1` on k3s-prod-2 | **~9.4 s** | last OK 04:20:24.634 → fail burst to 04:20:33.731; healthy at t+30s |
|
||||
| `nextcloud-db` | `nextcloud-db-1` on k3s-prod-3 | **9.43 s** | 1402 samples @44 ms; ONE burst 04:25:46.854 → 04:25:56.287; healthy at t+25s |
|
||||
| `semaphore-db` | *(none — kept on k3s-prod-1)* | **0 s** | stayed `readyInstances: 3` throughout |
|
||||
|
||||
One contiguous failure burst each, no flapping, no second dip. Note the 30 s scrape interval showed
|
||||
**no gap at all** for keycloak-db — the `-rw` endpoint was `ready` at every sample. A ~9 s write
|
||||
outage is invisible to Prometheus at this resolution; only the sub-second probe sees it.
|
||||
|
||||
After: all three `readyInstances: 3`, `Cluster in healthy state`, synchronous replication intact —
|
||||
`synchronous_standby_names = ANY 1 (...)` in force on each primary with both standbys
|
||||
`state=streaming sync_state=quorum`.
|
||||
|
||||
### The durable part: `CNPGPrimariesCoLocated`
|
||||
|
||||
Added to `platform/cnpg/alerts.yaml` (already wired via `platform-config/kustomization.yaml`, so
|
||||
Flux reconciles it on merge — no hand-apply).
|
||||
|
||||
**Neither metric family can answer the question alone**, which is the whole difficulty:
|
||||
`cnpg_*` metrics carry `namespace`/`pod` but **no node**; and `kube_pod_labels` is **empty** on this
|
||||
cluster (kube-state-metrics has no `--metric-labels-allowlist`), so `cnpg.io/instanceRole` is not in
|
||||
Prometheus at all. The rule instead reads **CNPG's own definition of "who is primary"** — the
|
||||
`<cluster>-rw` Service selects the primary and nothing else, and
|
||||
`kube_endpointslice_endpoints` carries both `endpoint_nodename` and `targetref_name`. Two
|
||||
`label_replace`s lift those into `node`/`pod`; `* on (namespace, pod) group_left () cnpg_collector_up`
|
||||
restricts the count to real CNPG instances so an unrelated `*-rw` Service cannot trigger it.
|
||||
`cnpg_collector_up` is the right-hand side deliberately — it exists for **every** instance, unlike
|
||||
the `cnpg_pg_stat_*` family which is missing entirely for `nextcloud-db`.
|
||||
|
||||
**A Ready-node gate keeps it from becoming the next permanently-firing alert.** With three clusters
|
||||
on three nodes, losing a node *necessarily* co-locates two primaries — correct behaviour, not a
|
||||
defect — so the rule only speaks while every node is `Ready`, and re-arms itself when the node
|
||||
returns. `for: 30m` rides out a switchover (seconds) and a full rolling update (minutes).
|
||||
`severity: warning`, like everything else in that file: a resilience posture, never a page.
|
||||
|
||||
**Proven in both directions, because this repo has been burned by both.** Live: the expression
|
||||
returned `{node="k3s-prod-1"} => 3` before, and an empty vector after (while the inner join still
|
||||
resolved all three primaries — quiet because the condition is false, not because the query went
|
||||
blind). Loaded into prod's Prometheus as a throwaway twin (`for: 1m`), it was observed going
|
||||
`pending` at +70 s and `firing` at +130 s, then deleted. And `platform/cnpg/alerts.promtool-test.yaml`
|
||||
(`./run-alert-tests.sh`) pins all four behaviours under promtool 3.14.0 — the exact Prometheus
|
||||
version on prod. **The tests were mutation-checked:** removing the `cnpg_collector_up` join,
|
||||
removing the readiness gate, or shortening `for:` each makes a test *fail*, so they are not vacuous.
|
||||
|
||||
### Debt / follow-ups
|
||||
|
||||
- **Placement is current, not durable.** The next failover, node reboot or CNPG rolling update can
|
||||
re-concentrate the primaries. That is now *reported*, not *prevented*. Rebalancing is a ~9 s
|
||||
write interruption per cluster.
|
||||
- **`promtool` is not in the baked toolchain** (`infra/devcontainer/versions.env`), so
|
||||
`run-alert-tests.sh` fetches it on demand and no CI stage runs the alert unit tests. Adding
|
||||
`PROMTOOL_VERSION` to `ci/versions.env` + the Containerfile would let `make -C ci gate` run them.
|
||||
- **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`.
|
||||
`cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description`
|
||||
carries the full actionable procedure so the missing page is not load-bearing.
|
||||
@@ -0,0 +1,324 @@
|
||||
---
|
||||
# promtool unit tests for the CNPGPrimariesCoLocated rule in platform/cnpg/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. This repo has been burned in both directions by alerts nobody tested:
|
||||
# a permanently-firing `critical` (the VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml)
|
||||
# and, worse, silence that looked like health (the 2026-09-06 CNPG archiving incident — 38h of
|
||||
# ContinuousArchiving=False with no alert of any kind). CNPGPrimariesCoLocated is a JOIN across two
|
||||
# metric families with a label_replace on each side and a node-readiness gate; "it looks right" is
|
||||
# not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin
|
||||
# BOTH directions plus the two ways the rule is meant to stay quiet.
|
||||
#
|
||||
# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule
|
||||
# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN},
|
||||
# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer
|
||||
# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the
|
||||
# Prometheus actually running on prod (v3.14.0) if it is not already on PATH.
|
||||
#
|
||||
# WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares
|
||||
# annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in
|
||||
# here and re-pasting it on every wording change. Querying the synthetic `ALERTS` series instead
|
||||
# proves exactly what matters — the alert reaches `pending` and then `firing` across the `for: 30m`
|
||||
# boundary, with the right labels — and stays decoupled from annotation prose.
|
||||
#
|
||||
# THE SERIES BELOW ARE REAL SHAPES, not invented ones: every label on
|
||||
# kube_endpointslice_endpoints / cnpg_collector_up / kube_node_status_condition / kube_node_info was
|
||||
# read off live prod Prometheus on 2026-09-12 before being copied here.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 1. THE BAD STATE THAT PROMPTED THIS RULE, exactly as prod looked on 2026-09-12: all three
|
||||
# primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) on k3s-prod-1, all nodes Ready.
|
||||
# Asserts the rule CAN fire, that it counts 3 (not 1, which a broken join would give), and
|
||||
# that `for: 30m` is honoured — pending at 29m, firing at 31m.
|
||||
- interval: 1m
|
||||
name: three primaries on one node fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.142",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.126",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.201",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# All nine instances publish cnpg_collector_up — the join must pick out only the three that
|
||||
# the -rw EndpointSlices point at, so the replicas are present here on purpose.
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-3",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-3",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-1",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-3",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The join resolves each -rw endpoint to (namespace, primary pod, node) — the drill-down an
|
||||
# operator actually wants. All three land on k3s-prod-1.
|
||||
- expr: |
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{namespace="keycloak", node="k3s-prod-1", pod="keycloak-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="nextcloud-db", node="k3s-prod-1", pod="nextcloud-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="semaphore-db", node="k3s-prod-1", pod="semaphore-db-2"}'
|
||||
value: 1
|
||||
# ... and the alert's own expression counts 3 of them on that one node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 3
|
||||
# NOT PERMANENTLY FIRING, part 1: `for: 30m` really is waited out.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 29m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# NOT PERMANENTLY INERT: it does reach `firing`.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 2. THE GOOD STATE — one primary per node, which is what the rebalancing switchovers on
|
||||
# 2026-09-12 produced. The rule must be SILENT here, and silent for the right reason: the
|
||||
# first assertion proves the join still resolves all three primaries (an alert that is quiet
|
||||
# because its query returns nothing is the failure mode this repo keeps hitting), the second
|
||||
# proves the >1 test then finds no over-loaded node.
|
||||
- interval: 1m
|
||||
name: one primary per node is silent
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-1",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-1",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-3",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Quiet for the RIGHT reason: the join is alive and finds all three primaries, one per node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-3"}'
|
||||
value: 1
|
||||
# ... and no alert is ever raised, at any point in the hour.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 59m
|
||||
exp_samples: []
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 3. A NODE IS DOWN, so co-location is UNAVOIDABLE and correct — the rule must not nag during
|
||||
# an outage. k3s-prod-3 is NotReady for the first 10 minutes and nextcloud-db has failed over
|
||||
# onto k3s-prod-1, which now holds two primaries. Then the node comes back at t=11m and the
|
||||
# co-location is NOT cleaned up, which is precisely the drift worth reporting: the rule must
|
||||
# RE-ARM by itself and fire 30m later. This is the test that keeps the readiness gate from
|
||||
# turning into a permanent mute.
|
||||
- interval: 1m
|
||||
name: node down suppresses, then re-arms when the node returns
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
# k3s-prod-3: NotReady (0) for t=0..10, Ready (1) from t=11 onwards.
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '0x10 1x79'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x90'
|
||||
promql_expr_test:
|
||||
# Two primaries genuinely do share k3s-prod-1 at t=5m...
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 2
|
||||
# ... but the readiness gate suppresses the alert entirely while a node is down: not even pending.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 5m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 10m
|
||||
exp_samples: []
|
||||
# Node back at t=11m; the `for` timer starts from scratch, so still only pending at t=25m.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# RE-ARMED: 30m of full-cluster health with the co-location still there -> firing.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 4. THE FALSE-POSITIVE GUARD. `endpointslice=~".+-rw-.+"` on its own is a loose match — any
|
||||
# Service in the cluster named `something-rw` produces an EndpointSlice that matches it. Two
|
||||
# such endpoints on one node must NOT be counted as CNPG primaries, which is what the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor is for. Without that factor
|
||||
# this test fires, so it is the thing that pins the join instead of a bare regex.
|
||||
- interval: 1m
|
||||
name: non-CNPG rw services are not counted
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="myapp-rw-aaaaa",targetref_kind="Pod",targetref_name="myapp-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="other-rw-bbbbb",targetref_kind="Pod",targetref_name="other-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# One real CNPG primary, on a different node — present so the test cannot pass merely because
|
||||
# the whole query is empty.
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Only the CNPG primary survives the join — the two decoy `*-rw` endpoints are dropped.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples: []
|
||||
@@ -265,3 +265,116 @@ spec:
|
||||
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
|
||||
#
|
||||
# WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
|
||||
# The operator's original ask was "the database primaries are all on one node". `instances: 3`
|
||||
# + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
|
||||
# exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
|
||||
# PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
|
||||
# semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
|
||||
# failovers — survivable, but the worst single-node blast radius the cluster can have.
|
||||
#
|
||||
# THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
|
||||
# (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
|
||||
# the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
|
||||
# * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
||||
# additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
|
||||
# the cluster's pods". None is role-aware: there is no way to say "this constraint is for
|
||||
# whichever pod is currently primary".
|
||||
# * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
|
||||
# Cluster that is the job podAntiAffinityType already does; it says nothing about how two
|
||||
# INDEPENDENT Clusters' primaries relate.
|
||||
# * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
|
||||
# (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
|
||||
# * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
|
||||
# postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
|
||||
# primary lands.
|
||||
# Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
|
||||
# scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
|
||||
# imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
|
||||
#
|
||||
# AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
|
||||
# cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
|
||||
# (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
|
||||
# RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
|
||||
# (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
|
||||
# constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
|
||||
# actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
|
||||
# excluding the node that holds another cluster's primary leaves the third instance
|
||||
# UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
|
||||
#
|
||||
# SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
|
||||
# status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
|
||||
# rebalances today and drifts again on the next failover, node reboot or rolling update.
|
||||
# A switchover that silently un-does itself is not a fix, so what is committed here is the
|
||||
# part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
|
||||
# a known, cheap runbook step instead of something discovered by hand months later.
|
||||
#
|
||||
# WHY THIS PROMQL. Neither family alone can answer the question:
|
||||
# * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
|
||||
# * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
|
||||
# --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
|
||||
# The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
|
||||
# Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
|
||||
# and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
|
||||
# `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
|
||||
# that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
|
||||
# cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
|
||||
# the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
|
||||
# is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
|
||||
#
|
||||
# THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
|
||||
# three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
|
||||
# is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
|
||||
# rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
|
||||
# the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
|
||||
#
|
||||
# for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
|
||||
# 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
|
||||
# still reporting real drift the same hour. severity: warning, like every rule in this file:
|
||||
# this is a resilience posture to correct at leisure, never a page.
|
||||
#
|
||||
# PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
|
||||
# fire is decoration and one that cannot stop is fatigue:
|
||||
# * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
|
||||
# {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
|
||||
# throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
|
||||
# observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
|
||||
# was deleted afterwards; it was never committed.
|
||||
# * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
|
||||
# empty vector, while the inner join still resolved all three primaries — semaphore-db-2
|
||||
# on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
|
||||
# the condition is false, NOT because the query went blind.
|
||||
# * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
|
||||
# ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
|
||||
# version running here: co-located fires after 30m, one-per-node is silent, a NotReady
|
||||
# node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
|
||||
# Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
|
||||
# readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
|
||||
- alert: CNPGPrimariesCoLocated
|
||||
expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
|
||||
description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml.
|
||||
#
|
||||
# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches:
|
||||
# 1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus
|
||||
# rules file (`groups:` at the top level). So `.spec` is extracted first.
|
||||
# 2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not
|
||||
# resolved in git. promtool does not care what the string says, but leaving a raw
|
||||
# `${...}` in a rendered rules file is exactly the kind of thing that gets copied
|
||||
# somewhere it DOES matter, so it is substituted with a placeholder here.
|
||||
# Both derived files go to a temp dir — nothing generated is left in the worktree, because
|
||||
# a generated file sitting next to its source is how a stale copy becomes the thing under
|
||||
# test.
|
||||
#
|
||||
# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env).
|
||||
# If it is not on PATH this fetches the version that matches the Prometheus actually
|
||||
# running on prod, so the unit tests are evaluated by the same engine that will evaluate
|
||||
# the rules for real. Override with PROMTOOL=/path/to/promtool.
|
||||
#
|
||||
# ./run-alert-tests.sh
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless
|
||||
# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`).
|
||||
PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}"
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
promtool="${PROMTOOL:-$(command -v promtool || true)}"
|
||||
if [[ -z "$promtool" ]]; then
|
||||
echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)"
|
||||
tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
|
||||
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \
|
||||
| tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool"
|
||||
promtool="$tmp/promtool"
|
||||
fi
|
||||
# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would
|
||||
# break there, and it did during development.
|
||||
promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")"
|
||||
echo "==> $("$promtool" --version 2>&1 | head -1)"
|
||||
|
||||
# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved.
|
||||
sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \
|
||||
| python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \
|
||||
> "$tmp/alerts.rules.yaml"
|
||||
cp "$here/alerts.promtool-test.yaml" "$tmp/"
|
||||
|
||||
# 2. Syntax gate first — a rules file that does not parse would make every unit test
|
||||
# "pass" vacuously by finding no rules at all.
|
||||
echo "==> promtool check rules"
|
||||
"$promtool" check rules "$tmp/alerts.rules.yaml"
|
||||
|
||||
# 3. The unit tests.
|
||||
echo "==> promtool test rules"
|
||||
cd "$tmp"
|
||||
"$promtool" test rules alerts.promtool-test.yaml
|
||||
Reference in new issue
Block a user