jakman85andClaude Opus 5
7057bbb711
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
157 lines
6.8 KiB
Bash
Executable File
157 lines
6.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/gate.sh — the merge-gate decision. Runs every stage in order and applies the
|
|
# R-TEST-03 rule: a merge is allowed only when the touched-requirement subset is
|
|
# GREEN. Honestly distinguishes three outcomes per stage:
|
|
# pass (0) green
|
|
# fail (1) BLOCKS the merge
|
|
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
|
|
# NOT counted as green (HARD RULE 4: don't fake green)
|
|
#
|
|
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
|
|
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
|
|
# available, the gate reports those requirements UNVERIFIED and — unless
|
|
# REQUIRE_CLUSTER_TESTS=true — does not hard-block Phase-0 bootstrap work, while
|
|
# making the gap loud. Set REQUIRE_CLUSTER_TESTS=true (Phase 1+) to make any
|
|
# skipped, cluster-needing touched requirement a hard block.
|
|
set -uo pipefail # NOTE: no -e; we want to run all stages and aggregate.
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
|
|
|
|
: "${REQUIRE_CLUSTER_TESTS:=false}"
|
|
|
|
declare -A OUTCOME
|
|
run_stage() {
|
|
local name="$1"; shift
|
|
log "==== stage: ${name} ===="
|
|
"$@"
|
|
local rc=$?
|
|
case "$rc" in
|
|
0) OUTCOME[$name]="pass"; ok "stage ${name}: pass" ;;
|
|
3) OUTCOME[$name]="skip"; skip "stage ${name}: SKIPPED (not run)" ;;
|
|
*) OUTCOME[$name]="fail"; err "stage ${name}: FAIL" ;;
|
|
esac
|
|
}
|
|
|
|
summary "## Merge gate (R-TEST-03)"
|
|
summary ""
|
|
|
|
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
|
|
run_stage lint "${CI_DIR}/lint.sh"
|
|
run_stage l0 "${CI_DIR}/l0.sh"
|
|
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
|
run_stage l1 "${CI_DIR}/l1.sh"
|
|
run_stage prom-rules "${CI_DIR}/prom-rules.sh"
|
|
run_stage conformance "${CI_DIR}/conformance.sh"
|
|
|
|
echo >&2
|
|
log "==== gate decision ===="
|
|
BLOCK=0
|
|
|
|
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
|
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
|
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
|
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
|
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
|
# as an UNDECIDED merge decision, never as a pass.
|
|
_reason_file="$(mktemp)"
|
|
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
|
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
|
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
|
TOUCHED_RC=$?
|
|
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
|
rm -f "$_reason_file"
|
|
TOUCHED=()
|
|
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
|
UNDECIDED=0
|
|
if [ "$TOUCHED_RC" -eq 3 ]; then
|
|
UNDECIDED=1
|
|
echo >&2
|
|
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
|
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
|
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
|
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
|
log "touched requirements: ${TOUCHED[*]}"
|
|
fi
|
|
|
|
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
|
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
|
case "${OUTCOME[spec-immutability]}" in
|
|
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
|
|
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
|
|
pass) ok "spec-immutability: green" ;;
|
|
esac
|
|
|
|
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
|
|
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
|
|
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
|
|
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
|
|
for s in lint l0 render-golden l1 prom-rules; do
|
|
case "${OUTCOME[$s]}" in
|
|
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
|
|
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
|
|
pass) ok "${s}: green" ;;
|
|
esac
|
|
done
|
|
|
|
# L2/L3 conformance: cluster-gated.
|
|
case "${OUTCOME[conformance]}" in
|
|
fail)
|
|
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
|
|
pass)
|
|
ok "conformance (L2/L3): green" ;;
|
|
skip)
|
|
if cluster_available; then
|
|
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
|
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
|
else
|
|
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
|
reqs=("${TOUCHED[@]}")
|
|
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
|
warn " (no cluster — Phase 0). These are NOT certified green."
|
|
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
|
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
|
|
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
|
|
BLOCK=1
|
|
fi
|
|
else
|
|
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
|
|
fi
|
|
fi ;;
|
|
esac
|
|
|
|
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
|
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
|
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
|
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
|
# PASS, so nobody reads it as "my branch is green".
|
|
# anything -> the environment cannot support a merge decision (no base ref, broken
|
|
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
|
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
|
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
|
BLOCK=1
|
|
fi
|
|
|
|
echo >&2
|
|
if [ "$BLOCK" -ne 0 ]; then
|
|
err "MERGE GATE: BLOCKED"
|
|
summary ""
|
|
summary "**Result: BLOCKED** — see failures above."
|
|
exit 1
|
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
|
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
|
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
|
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
|
warn " (or set BASE_REF) and run it again."
|
|
summary ""
|
|
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
|
exit 0
|
|
else
|
|
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
|
summary ""
|
|
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
|
exit 0
|
|
fi
|