0 Fork 0

Files

T

jakman85andClaude Opus 5 7057bbb711 fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused

VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-12 23:45:15 -05:00

7.7 KiB

Raw Blame History

`ci/` — portable merge gate (runnable today, without Gitea)

Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).

The same merge gate the Gitea Actions workflow runs (.gitea/workflows/), as plain scripts + a Makefile so the gates are runnable now, before any Gitea host or cluster exists. CI and a laptop call the identical scripts, so make -C ci gate == the CI merge gate.

Quick start

make -C ci gate        # the single command: run every stage + the R-TEST-03 decision
make -C ci help        # list targets
make -C ci tools       # which pinned tools are installed locally
make -C ci touched     # print the touched-requirement subset for this branch vs main

Individual stages:

make -C ci lint        # yaml/lint                          (no cluster; live now)
make -C ci l0          # schema + Kyverno + L0 static conformance (no cluster; live now)
make -C ci l1          # L1 rendering checks                 (no cluster; live now)
make -C ci prom-rules  # promtool rule-unit tests on alerts  (no cluster; live now)
make -C ci conformance # L2/L3 in-cluster Chainsaw Job       (SKIPS unless CLUSTER_AVAILABLE=true)
make -C ci test        # unit tests for these scripts themselves (ci/tests/*.test.sh)

The l0 stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate (kyverno test), and conformance/run.sh --layer l0 — the whole-repo L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC, spec-immutability enforcement). See conformance/tests/README.md.

Exit-code contract

Every stage script uses: 0 = pass · 1 = fail (blocks merge) · 3 = SKIPPED (cluster-gated with no cluster, or an unwired seam / missing tool). A 3 is never counted as green — that is how the gate stays honest in Phase 0 (no fake green).

The touched-requirement subset can also be undeterminable (exit 3)

touched-requirements.sh is the input to the R-TEST-03 decision. It resolves a diff base in this order, verifying each candidate exists before using it:

BASE_REF (explicit)  >  GITHUB_BASE_REF (Actions PR runs)  >  origin/main  >  main

If no candidate resolves, if there is no merge base, if the diff command fails, or if HEAD is not ahead of the base (you are sitting on main, or the branch is already merged), it prints a loud banner, writes nothing to stdout, and exits 3 — plus a machine-readable reason code (on-base / no-base / diff-failed) to $TOUCHED_REASON_FILE when that variable names a path.

Why this is not cosmetic. Every one of those conditions used to degrade silently to an empty touched set and exit 0. An empty set means conformance/run.sh is invoked with no --requirements, which means its coverage --require gate never runs — so on 2026-08-31 the local gate printed MERGE GATE: PASS having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED on real coverage debt. There is no silent narrowing left: the HEAD~1 last resort and the two-dot diff fallback are both gone. Locked down by ci/tests/touched-requirements-base.test.sh.

gate.sh turns exit 3 into an UNDECIDED merge decision, never a pass:

reason	headline	exit
`on-base`	`MERGE GATE: NOT APPLICABLE — no branch under gate`	0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green")
`no-base`, `diff-failed`	`MERGE GATE: BLOCKED`	1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes)

A stage that needs the subset (l0.sh, l1.sh) still runs its FULL layer as a superset sanity net in that case — nothing goes untested — and says plainly that the scoped check did not happen.

Reproducibility (R-TEST-02)

Version-pinned: all tool versions live in versions.env — the single source of truth shared by the local scripts and the CI installer. Bumping a version is a normal feature-branch → main change.
Single command, environment-agnostic: make -C ci gate on any machine with the pinned tools.
Deterministic: no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.

Files

ci/
  Makefile                 targets: lint · l0 · l1 · prom-rules · conformance · gate · touched · test · tools · versions
  versions.env             PINNED tool versions (R-TEST-02) — single source of truth
  lib.sh                   shared helpers: logging, tool checks, cluster detection, artifact paths
  yamllint.yaml            lint config for the yaml/lint stage
  lint.sh                  Stage: yaml/lint
  l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
  l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
  prom-rules.sh            Stage: `promtool test rules` over platform/monitoring's PrometheusRules
                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
                           other stage can see — permanently-firing and structurally INERT rules.
  conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
  touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                           exits 3 LOUDLY rather than guessing when the diff base is unreliable
  gate.sh                  run all stages + the merge decision (the single command)
  tests/*.test.sh          unit tests for these scripts (`make -C ci test`), run against
                           throwaway git repos via the REPO_ROOT override — no network, no cluster
  README.md                this file

Integration seams — entrypoints owned by OTHER agents

This gate is an orchestrator. It does not implement schema validation, Kyverno policies, or the conformance runner — those are other workstreams' deliverables. Each seam is env-overridable so the owning agent / PM can reconcile the real entrypoint without editing these scripts. Defaults below are this workstream's assumptions — flagged for the PM to verify at integration.

Seam	Owner / task	Env override	Assumed default	Assumed interface
ServiceDefinition schema validator	WS-GITOPS · T-0.3 (`schemas/`)	`SCHEMA_VALIDATE_CMD`	`schemas/validate.sh`	`validate.sh catalog/*.yaml` → exit 0/1; `--version` supported
Kyverno L0 policies + tests	WS-TEST · T-0.5 (`platform/kyverno/`)	`KYVERNO_POLICY_DIR`	`platform/kyverno`	contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>`
Conformance runner (Chainsaw)	WS-TEST · T-0.4 (`conformance/`)	`CONFORMANCE_CMD`	`conformance/run.sh`	`run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/`
Catalog (ServiceDefinitions)	WS-APPS	`CATALOG_DIR`	`catalog`	`*.yaml` ServiceDefinition files

Artifact locations (git-ignored): JUnit → results/junit/*.xml; coverage → reports/coverage/**. Override with JUNIT_DIR / COVERAGE_DIR if the runner writes elsewhere.

A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass — so during parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment its owner delivers a runnable entrypoint.

Cluster gating

CLUSTER_AVAILABLE=true (env locally, vars.CLUSTER_AVAILABLE in Gitea) makes the L2/L3 stage run and be required; anything else makes it SKIP (Phase 0). REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns any unverified cluster-needing touched requirement into a hard block.

7.7 KiB Raw Blame History

ci/ — portable merge gate (runnable today, without Gitea)