fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
ebce6dfabb
commit
7057bbb711
10 files changed
+787
-3
No files matched your search
@@ -0,0 +1,357 @@
|
||||
# promtool rule-unit tests for the platform-health.backups group (platform/monitoring/alerts.yaml).
|
||||
# Run with: platform/monitoring/tests/run.sh (or `make -C ci prom-rules`)
|
||||
#
|
||||
# Every rule is pinned in BOTH directions, because this repo has shipped both failure modes to
|
||||
# prod: a permanently-firing alert (CNPG, 2026-09-12) and a structurally inert one
|
||||
# (VeleroNoRecentBackup, which could not fire for a schedule that had never backed up — prod ran
|
||||
# with NO volume backups and nothing alerted).
|
||||
#
|
||||
# CLOCK CONVENTION: promtool's test clock starts at the unix epoch, so `time()` == the eval_time in
|
||||
# seconds. "This schedule has existed for 48h" is therefore expressed as a creationTimestamp of
|
||||
# -172800 (48h BEFORE the epoch) rather than by running the test for two days. Same for
|
||||
# last-successful-backup timestamps. Sample interval is 1m so nothing is ever stale inside
|
||||
# Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. TODAY'S HEALTHY PROD STATE -> the whole group must be SILENT.
|
||||
# This is the "not permanently firing" half of the proof, and it is also the test that
|
||||
# catches a broken join: kube_velero_schedule_created is >36h old for both schedules, so
|
||||
# VeleroScheduleNeverBackedUp's left-hand side is NON-EMPTY here and only the
|
||||
# `unless on (schedule)` suppresses it. Mis-spell that label and this test goes red.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: healthy prod state (both schedules succeeded recently) - everything quiet
|
||||
input_series:
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
|
||||
values: '0+0x180'
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
|
||||
values: '0+0x180'
|
||||
# Velero also exports an un-scheduled (ad-hoc backup) series. Present on purpose: every rule
|
||||
# must ignore it, or an ad-hoc backup would satisfy the schedule alerts.
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero"}'
|
||||
values: '0+0x180'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoRecentBackup
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoVolumeDataBackup
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroBackupFailure
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroBackupPartialFailure
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The anti-masking counter really does count the volume-data schedule (a `1`, not the
|
||||
# `or vector(0)` fallback) — so the rule is quiet for the right reason.
|
||||
- expr: |
|
||||
count(
|
||||
(kube_velero_schedule_protects_volume_data == 1)
|
||||
and on (schedule)
|
||||
((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
|
||||
)
|
||||
or vector(0)
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. THE PROD DEFECT, RECONSTRUCTED (2026-09-12): `velero-daily` paused with no successful
|
||||
# backup ever, while `velero-daily-resources-only` runs fine every 03:00 and masks it.
|
||||
# Note what VeleroNoRecentBackup does here: NOTHING. That assertion is the regression test
|
||||
# for the original bug — it documents that the old rule alone can never see this state.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: velero-daily paused and NEVER backed up, resources-only healthy (the prod defect)
|
||||
input_series:
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
# NO velero_backup_last_successful_timestamp for velero-daily: it never succeeded, so Velero
|
||||
# never created the series. That absence IS the bug.
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
|
||||
values: '0+0x180'
|
||||
alert_rule_test:
|
||||
# The old rule is blind to this state. Asserted, not assumed.
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoRecentBackup
|
||||
exp_alerts: []
|
||||
# `for: 1h` is real: not yet firing at 30m.
|
||||
- eval_time: 30m
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kube-state-metrics
|
||||
namespace: velero
|
||||
schedule: velero-daily
|
||||
customresource_group: velero.io
|
||||
customresource_kind: Schedule
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Velero schedule has NEVER produced a successful backup: velero-daily"
|
||||
description: "Schedule velero-daily has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
# `for: 30m` is real: not yet firing at 20m.
|
||||
- eval_time: 20m
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kube-state-metrics
|
||||
namespace: velero
|
||||
schedule: velero-daily
|
||||
customresource_group: velero.io
|
||||
customresource_kind: Schedule
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Velero schedule is PAUSED: velero-daily"
|
||||
description: "Schedule velero-daily has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoVolumeDataBackup
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "NO Velero schedule has backed up volume DATA in 36h"
|
||||
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
promql_expr_test:
|
||||
# The `or vector(0)` fallback is load-bearing: without it `count(<empty>) == 0` returns no
|
||||
# series and the rule would be inert exactly when it matters.
|
||||
- expr: |
|
||||
count(
|
||||
(kube_velero_schedule_protects_volume_data == 1)
|
||||
and on (schedule)
|
||||
((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
|
||||
)
|
||||
or vector(0)
|
||||
eval_time: 2h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 0
|
||||
# And the old expression genuinely produces nothing for a never-successful schedule.
|
||||
- expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
|
||||
eval_time: 2h
|
||||
exp_samples: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. NEVER BACKED UP BUT **NOT** PAUSED — a wedged schedule controller, a cron that never
|
||||
# comes due, a storage location that rejects every backup before one ever succeeds.
|
||||
# Proves VeleroScheduleNeverBackedUp does not secretly depend on `paused`.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: schedule enabled but has never succeeded - still alerts, paused rule stays quiet
|
||||
input_series:
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kube-state-metrics
|
||||
namespace: velero
|
||||
schedule: velero-daily
|
||||
customresource_group: velero.io
|
||||
customresource_kind: Schedule
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Velero schedule has NEVER produced a successful backup: velero-daily"
|
||||
description: "Schedule velero-daily has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
- eval_time: 2h
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. GRACE PERIOD — a Schedule added 12h ago whose first run is not due yet must NOT alert.
|
||||
# This is the "not permanently firing" guard for the new rule: without the
|
||||
# kube_velero_schedule_created term it would page on every newly added schedule.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: freshly created schedule inside its 36h grace window - no alert
|
||||
input_series:
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-43200+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
# An established, healthy volume-data schedule alongside it, so the anti-masking rule is
|
||||
# quiet and this test isolates the grace period.
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-172800+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
|
||||
values: '0+0x180'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoVolumeDataBackup
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 5. BACKED UP ONCE, THEN STOPPED (48h stale) — the case the ORIGINAL rule was written for.
|
||||
# Kept so the old rule is proven live too, and so the new rule is proven NOT to
|
||||
# double-page: the success series exists, so `unless` suppresses VeleroScheduleNeverBackedUp.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: volume-data schedule last succeeded 48h ago - stale alerts fire, never-backed-up does not
|
||||
input_series:
|
||||
- series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '-864000+0x180'
|
||||
- series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '0+0x180'
|
||||
- series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
|
||||
values: '1+0x180'
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
|
||||
values: '-172800+0x180'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoRecentBackup
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: velero
|
||||
namespace: velero
|
||||
schedule: velero-daily
|
||||
exp_annotations:
|
||||
summary: "No successful Velero backup in 24h: velero-daily"
|
||||
description: "Schedule velero-daily has not produced a successful backup in over 24h (missed schedule)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoVolumeDataBackup
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "NO Velero schedule has backed up volume DATA in 36h"
|
||||
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
|
||||
# ===========================================================================================
|
||||
# 6. THE EXPORTER ITSELF DISAPPEARS — kube-state-metrics stops publishing the Schedule CRs
|
||||
# (removed values, broken RBAC, KSM down) while Velero's own metrics still look fine.
|
||||
# The two rules that need the exporter go quiet, which is unavoidable; VeleroNoVolumeDataBackup
|
||||
# FIRES instead. Asserted deliberately: absence of the signal must be the alarm, so this
|
||||
# family can never regress to "silent because the input vanished" a second time.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: kube-state-metrics schedule metrics absent - absence itself alerts
|
||||
input_series:
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
|
||||
values: '0+0x180'
|
||||
- series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
|
||||
values: '0+0x180'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroNoVolumeDataBackup
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "NO Velero schedule has backed up volume DATA in 36h"
|
||||
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
||||
- eval_time: 2h
|
||||
alertname: VeleroScheduleNeverBackedUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: VeleroSchedulePaused
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 7. The two pre-existing counter rules, so the whole group is covered both ways rather than
|
||||
# just the parts that were touched.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: backup failures increment - failure rules fire
|
||||
input_series:
|
||||
- series: 'velero_backup_failure_total{job="velero",namespace="velero",schedule="velero-daily"}'
|
||||
values: '0+0x100 1+0x80'
|
||||
- series: 'velero_backup_partial_failure_total{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
|
||||
values: '0+0x100 1+0x80'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: VeleroBackupFailure
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: velero
|
||||
namespace: velero
|
||||
schedule: velero-daily
|
||||
exp_annotations:
|
||||
summary: "Velero backup failed: schedule velero-daily"
|
||||
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
|
||||
- eval_time: 2h
|
||||
alertname: VeleroBackupPartialFailure
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: velero
|
||||
namespace: velero
|
||||
schedule: velero-daily-resources-only
|
||||
exp_annotations:
|
||||
summary: "Velero backup partially failed"
|
||||
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Flatten this repo's PrometheusRule CRs into a plain Prometheus rule file.
|
||||
|
||||
`promtool test rules` consumes a rule file (`{groups: [...]}`), not a Kubernetes
|
||||
PrometheusRule object, so the committed manifests cannot be fed to it directly. This
|
||||
script is the (deliberately dumb) adapter: it walks the given YAML files, picks out every
|
||||
`kind: PrometheusRule` document, and concatenates their `spec.groups` into one rule file.
|
||||
|
||||
Why an adapter instead of a second copy of the rules: a rule-unit test is only worth
|
||||
anything if it tests the EXACT expression that ships. Copying expressions into a fixture
|
||||
would let the fixture stay green while the deployed rule rots -- which is the class of bug
|
||||
this harness exists to catch (a permanently-firing CNPG alert and a structurally inert
|
||||
Velero alert both reached prod in 2026-09).
|
||||
|
||||
Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in
|
||||
platform/monitoring uses one, and leaving them alone keeps annotation text identical to
|
||||
the file under test. If a placeholder ever appears inside an `expr`, that is refused
|
||||
loudly rather than silently producing an untestable rule file.
|
||||
|
||||
Group names are made unique per source file so two files may legally reuse a name.
|
||||
|
||||
Usage:
|
||||
extract_rules.py --out generated.rules.yaml alerts.yaml [more.yaml ...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover - reported by the runner as a SKIP, never a pass
|
||||
print(
|
||||
"extract_rules.py: PyYAML is required (it ships transitively with the pinned "
|
||||
"yamllint used by ci/lint.sh: pip install 'yamllint==1.35.1')",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise SystemExit(4)
|
||||
|
||||
PLACEHOLDER = re.compile(r"\$\{[A-Za-z_][A-Za-z0-9_]*\}")
|
||||
|
||||
|
||||
def extract(paths: list[Path]) -> dict:
|
||||
groups: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
for path in paths:
|
||||
docs = [d for d in yaml.safe_load_all(path.read_text()) if d]
|
||||
rules_docs = [d for d in docs if d.get("kind") == "PrometheusRule"]
|
||||
if not rules_docs:
|
||||
raise SystemExit(f"{path}: contains no PrometheusRule documents")
|
||||
for doc in rules_docs:
|
||||
for group in doc.get("spec", {}).get("groups", []) or []:
|
||||
for rule in group.get("rules", []) or []:
|
||||
expr = str(rule.get("expr", ""))
|
||||
if PLACEHOLDER.search(expr):
|
||||
raise SystemExit(
|
||||
f"{path}: rule {rule.get('alert') or rule.get('record')!r} has a "
|
||||
f"postBuild placeholder inside its expr; it cannot be unit-tested "
|
||||
f"as written"
|
||||
)
|
||||
# Namespace the group by source file so names cannot collide.
|
||||
group = dict(group)
|
||||
name = f"{path.stem}::{group['name']}"
|
||||
if name in seen:
|
||||
raise SystemExit(f"duplicate rule group {name!r}")
|
||||
seen.add(name)
|
||||
group["name"] = name
|
||||
groups.append(group)
|
||||
if not groups:
|
||||
raise SystemExit("no rule groups found")
|
||||
return {"groups": groups}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--out", required=True, type=Path, help="rule file to write")
|
||||
ap.add_argument("manifests", nargs="+", type=Path, help="PrometheusRule YAML files")
|
||||
args = ap.parse_args()
|
||||
|
||||
out = extract(args.manifests)
|
||||
args.out.write_text(yaml.safe_dump(out, sort_keys=False, default_flow_style=False))
|
||||
n = sum(len(g.get("rules", []) or []) for g in out["groups"])
|
||||
print(
|
||||
f"extract_rules.py: {len(out['groups'])} group(s), {n} rule(s) -> {args.out}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+92
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env bash
|
||||
# platform/monitoring/tests/run.sh — Prometheus RULE-UNIT-TEST harness (no cluster).
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# This repo has now shipped both halves of the alerting failure space to prod:
|
||||
# * a PERMANENTLY-FIRING rule (the CNPG alert fixed 2026-09-12), and
|
||||
# * a STRUCTURALLY INERT rule (VeleroNoRecentBackup: `time() - <absent series>` yields no
|
||||
# series, so a schedule that had NEVER backed up could not raise it — prod ran with no
|
||||
# volume backups at all and nothing alerted).
|
||||
# Neither is visible by reading the expression, by `kubectl apply`, or by looking at a healthy
|
||||
# live cluster: an inert rule and a correct rule are indistinguishable when everything is fine.
|
||||
# `promtool test rules` is the tool that separates them, because it lets the BAD state be
|
||||
# constructed on purpose and asserted on, offline and in CI. Every rule here is therefore
|
||||
# pinned twice: quiet on the good state, firing on the bad one.
|
||||
#
|
||||
# The rules under test are the COMMITTED manifests (extract_rules.py flattens the
|
||||
# PrometheusRule CRs into a rule file) — never a copy, so a fixture cannot drift green while
|
||||
# the deployed rule rots.
|
||||
#
|
||||
# Usage: ./run.sh # run every *.test.yaml here
|
||||
# ./run.sh foo.test.yaml
|
||||
#
|
||||
# Exit codes follow ci/lib.sh: 0 pass, 1 fail, 3 SKIPPED (toolchain unavailable — never green).
|
||||
# promtool is taken from $PATH if present, otherwise run from the pinned Prometheus image via
|
||||
# podman/docker. The image tag matches the version prod runs, so "passes here" means "passes
|
||||
# with the engine that will evaluate it".
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
MONITORING_DIR="$(cd "${HERE}/.." && pwd)"
|
||||
PROM_IMAGE="${PROM_IMAGE:-quay.io/prometheus/prometheus:v3.14.0}"
|
||||
|
||||
MANIFESTS=(
|
||||
"${MONITORING_DIR}/alerts.yaml"
|
||||
"${MONITORING_DIR}/pvc-capacity-alerts.yaml"
|
||||
)
|
||||
|
||||
if [ "$#" -gt 0 ]; then
|
||||
TESTS=("$@")
|
||||
else
|
||||
mapfile -t TESTS < <(find "${HERE}" -maxdepth 1 -name '*.test.yaml' | sort)
|
||||
fi
|
||||
if [ "${#TESTS[@]}" -eq 0 ]; then
|
||||
echo "run.sh: no *.test.yaml found in ${HERE}" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------- toolchain resolution
|
||||
RUNNER=""
|
||||
if command -v promtool >/dev/null 2>&1; then
|
||||
RUNNER="native"
|
||||
elif command -v podman >/dev/null 2>&1; then
|
||||
RUNNER="podman"
|
||||
elif command -v docker >/dev/null 2>&1; then
|
||||
RUNNER="docker"
|
||||
else
|
||||
echo "run.sh: SKIP — no promtool on PATH and no podman/docker to run ${PROM_IMAGE}" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# The test files reference the generated rule file by bare name, so everything must sit in one
|
||||
# directory (that is also the only directory the container needs mounted).
|
||||
rc=0
|
||||
python3 "${HERE}/extract_rules.py" --out "${WORK}/generated.rules.yaml" "${MANIFESTS[@]}" || rc=$?
|
||||
if [ "$rc" -eq 4 ]; then
|
||||
echo "run.sh: SKIP — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
|
||||
exit 3
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
exit "$rc"
|
||||
fi
|
||||
|
||||
names=()
|
||||
for t in "${TESTS[@]}"; do
|
||||
cp "$t" "${WORK}/"
|
||||
names+=("$(basename "$t")")
|
||||
done
|
||||
chmod -R a+rX "${WORK}"
|
||||
|
||||
echo "run.sh: promtool test rules (${RUNNER}) ${names[*]}" >&2
|
||||
case "$RUNNER" in
|
||||
native)
|
||||
( cd "${WORK}" && promtool test rules "${names[@]}" ) ;;
|
||||
podman)
|
||||
podman run --rm --user 0 -v "${WORK}:/rules:z" -w /rules \
|
||||
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
||||
docker)
|
||||
docker run --rm --user 0 -v "${WORK}:/rules" -w /rules \
|
||||
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
||||
esac
|
||||
Reference in new issue
Block a user