jakman85andClaude Opus 5
7057bbb711
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
64 lines
2.4 KiB
Makefile
64 lines
2.4 KiB
Makefile
# ci/Makefile — portable, single-command local equivalent of the Gitea Actions
|
|
# merge gate (R-TEST-02: the gates are runnable TODAY, without Gitea).
|
|
#
|
|
# The LIVE Gitea Actions workflow (.gitea/workflows/gate.yml) invokes `make -C ci gate`
|
|
# directly, so `make -C ci gate` locally IS the CI merge gate — byte-for-byte. Run from
|
|
# anywhere:
|
|
# make -C ci gate
|
|
#
|
|
# Every stage is a thin wrapper over a ci/*.sh script; see ci/README.md.
|
|
SHELL := /usr/bin/env bash
|
|
CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
|
|
|
.DEFAULT_GOAL := help
|
|
|
|
.PHONY: help spec-immutability lint schema l0 render-golden l1 prom-rules conformance gate all touched versions tools test
|
|
|
|
help: ## Show this help
|
|
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
|
|
|
spec-immutability: ## R-TEST-06: fail if the spec repo is modified or drifted from ci/SPEC_PIN
|
|
@"$(CI_DIR)spec-immutability.sh"
|
|
|
|
lint: ## yaml/lint (no cluster; live now)
|
|
@"$(CI_DIR)lint.sh"
|
|
|
|
schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no cluster; live now)
|
|
@"$(CI_DIR)l0.sh"
|
|
|
|
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
|
|
@"$(CI_DIR)render-golden.sh"
|
|
|
|
l1: ## L1 rendering checks (no cluster; live now)
|
|
@"$(CI_DIR)l1.sh"
|
|
|
|
prom-rules: ## promtool rule-unit tests for platform/monitoring alerts (no cluster; live now)
|
|
@"$(CI_DIR)prom-rules.sh"
|
|
|
|
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
|
@"$(CI_DIR)conformance.sh"
|
|
|
|
gate: ## Full merge gate: all stages + R-TEST-03 decision (the single command)
|
|
@"$(CI_DIR)gate.sh"
|
|
|
|
all: gate ## Alias for `gate`
|
|
|
|
touched: ## Print the touched-requirement subset for the current branch vs main
|
|
@"$(CI_DIR)touched-requirements.sh"
|
|
|
|
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
|
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
|
echo "== $$t"; bash "$$t" || rc=1; \
|
|
done; \
|
|
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
|
exit $$rc
|
|
|
|
versions: ## Print pinned tool versions (R-TEST-02)
|
|
@cat "$(CI_DIR)versions.env"
|
|
|
|
tools: ## Report which required tools are installed locally
|
|
@source "$(CI_DIR)versions.env"; \
|
|
for t in yamllint kyverno chainsaw kubectl yq git; do \
|
|
if command -v $$t >/dev/null 2>&1; then printf " \033[32mok\033[0m %s\n" "$$t"; \
|
|
else printf " \033[33mmiss\033[0m %s\n" "$$t"; fi; done
|