jakman85andClaude Opus 5
7057bbb711
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
93 lines
3.5 KiB
Bash
Executable File
93 lines
3.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# platform/monitoring/tests/run.sh — Prometheus RULE-UNIT-TEST harness (no cluster).
|
|
#
|
|
# WHY THIS EXISTS
|
|
# This repo has now shipped both halves of the alerting failure space to prod:
|
|
# * a PERMANENTLY-FIRING rule (the CNPG alert fixed 2026-09-12), and
|
|
# * a STRUCTURALLY INERT rule (VeleroNoRecentBackup: `time() - <absent series>` yields no
|
|
# series, so a schedule that had NEVER backed up could not raise it — prod ran with no
|
|
# volume backups at all and nothing alerted).
|
|
# Neither is visible by reading the expression, by `kubectl apply`, or by looking at a healthy
|
|
# live cluster: an inert rule and a correct rule are indistinguishable when everything is fine.
|
|
# `promtool test rules` is the tool that separates them, because it lets the BAD state be
|
|
# constructed on purpose and asserted on, offline and in CI. Every rule here is therefore
|
|
# pinned twice: quiet on the good state, firing on the bad one.
|
|
#
|
|
# The rules under test are the COMMITTED manifests (extract_rules.py flattens the
|
|
# PrometheusRule CRs into a rule file) — never a copy, so a fixture cannot drift green while
|
|
# the deployed rule rots.
|
|
#
|
|
# Usage: ./run.sh # run every *.test.yaml here
|
|
# ./run.sh foo.test.yaml
|
|
#
|
|
# Exit codes follow ci/lib.sh: 0 pass, 1 fail, 3 SKIPPED (toolchain unavailable — never green).
|
|
# promtool is taken from $PATH if present, otherwise run from the pinned Prometheus image via
|
|
# podman/docker. The image tag matches the version prod runs, so "passes here" means "passes
|
|
# with the engine that will evaluate it".
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
MONITORING_DIR="$(cd "${HERE}/.." && pwd)"
|
|
PROM_IMAGE="${PROM_IMAGE:-quay.io/prometheus/prometheus:v3.14.0}"
|
|
|
|
MANIFESTS=(
|
|
"${MONITORING_DIR}/alerts.yaml"
|
|
"${MONITORING_DIR}/pvc-capacity-alerts.yaml"
|
|
)
|
|
|
|
if [ "$#" -gt 0 ]; then
|
|
TESTS=("$@")
|
|
else
|
|
mapfile -t TESTS < <(find "${HERE}" -maxdepth 1 -name '*.test.yaml' | sort)
|
|
fi
|
|
if [ "${#TESTS[@]}" -eq 0 ]; then
|
|
echo "run.sh: no *.test.yaml found in ${HERE}" >&2
|
|
exit 3
|
|
fi
|
|
|
|
# ---------------------------------------------------------------- toolchain resolution
|
|
RUNNER=""
|
|
if command -v promtool >/dev/null 2>&1; then
|
|
RUNNER="native"
|
|
elif command -v podman >/dev/null 2>&1; then
|
|
RUNNER="podman"
|
|
elif command -v docker >/dev/null 2>&1; then
|
|
RUNNER="docker"
|
|
else
|
|
echo "run.sh: SKIP — no promtool on PATH and no podman/docker to run ${PROM_IMAGE}" >&2
|
|
exit 3
|
|
fi
|
|
|
|
WORK="$(mktemp -d)"
|
|
trap 'rm -rf "${WORK}"' EXIT
|
|
|
|
# The test files reference the generated rule file by bare name, so everything must sit in one
|
|
# directory (that is also the only directory the container needs mounted).
|
|
rc=0
|
|
python3 "${HERE}/extract_rules.py" --out "${WORK}/generated.rules.yaml" "${MANIFESTS[@]}" || rc=$?
|
|
if [ "$rc" -eq 4 ]; then
|
|
echo "run.sh: SKIP — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
|
|
exit 3
|
|
elif [ "$rc" -ne 0 ]; then
|
|
exit "$rc"
|
|
fi
|
|
|
|
names=()
|
|
for t in "${TESTS[@]}"; do
|
|
cp "$t" "${WORK}/"
|
|
names+=("$(basename "$t")")
|
|
done
|
|
chmod -R a+rX "${WORK}"
|
|
|
|
echo "run.sh: promtool test rules (${RUNNER}) ${names[*]}" >&2
|
|
case "$RUNNER" in
|
|
native)
|
|
( cd "${WORK}" && promtool test rules "${names[@]}" ) ;;
|
|
podman)
|
|
podman run --rm --user 0 -v "${WORK}:/rules:z" -w /rules \
|
|
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
|
docker)
|
|
docker run --rm --user 0 -v "${WORK}:/rules" -w /rules \
|
|
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
|
esac
|