jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
551 lines
30 KiB
Bash
Executable File
551 lines
30 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — alert-rules-unit-tested
|
|
# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
|
|
# VPN)." — spec docs/10-observability-and-alerting.md)
|
|
# R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
|
|
# — spec docs/14-testing-and-conformance.md)
|
|
#
|
|
# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
|
|
# --------------------------------------------------------------
|
|
# This repo has shipped BOTH halves of the alerting failure space to prod:
|
|
# * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
|
|
# it), and
|
|
# * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
|
|
# yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
|
|
# with no volume backups at all and nothing alerted (2026-09-12).
|
|
# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
|
|
# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
|
|
# rules` is the only tool that separates them, because the BAD state can be constructed on
|
|
# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
|
|
# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
|
|
# that existed acquired an engine off the network mid-run if one was missing. Coverage that
|
|
# looks present and is never enforced is the failure mode this check ends.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
|
|
# COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
|
|
# and then run over every test file in the repo,
|
|
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
|
|
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
|
|
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
|
|
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
|
|
# CI runner installs it for the l0 stage;
|
|
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
|
|
# evidence only if the engine matches the one prod evaluates rules with;
|
|
# (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
|
|
# GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
|
|
# 2026-09-12 and both are removed; this is the regression guard;
|
|
# (4) there is exactly ONE harness. Two were written independently within an hour of
|
|
# each other; the one nothing runs is the one that rots;
|
|
# (5) every rule-unit test file in the repo PASSES, run through that harness (which
|
|
# `promtool check rules` first, so a non-parsing rules file cannot make unit tests
|
|
# pass vacuously by finding no rules at all);
|
|
# (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
|
|
# through the SAME harness twice — intact (must PASS) and with one expectation
|
|
# mutated (must FAIL) — so a harness that has silently stopped evaluating anything
|
|
# is caught on every run, not only when someone thinks to try it;
|
|
# AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
|
|
# below, printed with counts on every run, and the register can only SHRINK.
|
|
#
|
|
# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
|
|
# ---------------------------------------------------------------------------------------
|
|
# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
|
|
# * FAIL on any missing test — correct in principle, but it makes the merge gate red for
|
|
# every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
|
|
# disabled gate protects nothing.
|
|
# * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
|
|
# tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
|
|
# enforcement.
|
|
# * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
|
|
# counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
|
|
# file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
|
|
# writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
|
|
# register cannot rot into fiction.
|
|
# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
|
|
# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
|
|
# register drifts, or a test that exists stops passing.
|
|
# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
|
|
# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
|
|
# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
|
|
# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
|
|
# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
|
|
# evaluating the rules with an engine prod does not run.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps):
|
|
# - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
|
|
# series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
|
|
# (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
|
|
# - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
|
|
# test says.
|
|
# - It does not prove a rule file is RECONCILED into a cluster. An alert file no
|
|
# kustomization includes is a separate "looks applied and isn't" failure, not audited here.
|
|
# - Coverage of an individual alert is detected by its NAME appearing in a test file that
|
|
# covers its manifest. A test file that merely mentions a name without asserting anything
|
|
# counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
|
|
# every author wrote a meaningful assertion. Review still owns test quality.
|
|
# - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
|
|
# against the repo's pin, which was established by a live read recorded in
|
|
# infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
|
|
|
|
# Honest SKIP (never green) if the thing under test is not present at all. Because this test
|
|
# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
|
|
# PrometheusRule-less repo is a vanished tree, not a pass.
|
|
if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
|
|
"${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
|
|
echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
|
|
exit 3
|
|
fi
|
|
|
|
# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
|
|
# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
|
|
# is reported as a gap and never as a pass.
|
|
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
|
|
if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
|
|
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
|
|
else
|
|
PROMTOOL_BIN=""
|
|
fi
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
|
|
"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
|
|
rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
from collections import defaultdict
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
HARNESS = Path(sys.argv[2])
|
|
PROMTOOL = sys.argv[3]
|
|
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
|
|
|
FAILURES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
# =============================================================================
|
|
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
|
|
#
|
|
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
|
|
# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
|
|
# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
|
|
# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
|
|
# FAILS as stale.
|
|
# =============================================================================
|
|
UNTESTED_FILES = {
|
|
"apps/wave-0/gitea.yaml",
|
|
"apps/wave-0/jackett.yaml",
|
|
"apps/wave-0/pihole.yaml",
|
|
"apps/wave-0/plex.yaml",
|
|
"apps/wave-0/qbittorrent.yaml",
|
|
"apps/wave-0/vm-demo.yaml",
|
|
"apps/wave-1/keycloak.yaml",
|
|
"apps/wave-1/nextcloud.yaml",
|
|
"apps/wave-1/radarr.yaml",
|
|
"apps/wave-1/semaphore.yaml",
|
|
"apps/wave-1/sonarr.yaml",
|
|
"apps/wave-2/seerr.yaml",
|
|
"platform/kubevirt/alerts.yaml",
|
|
}
|
|
|
|
# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
|
|
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
|
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
|
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
|
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
|
# rules file by the suite convention but has no assertions of its own yet.
|
|
UNTESTED_ALERTS = {
|
|
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
|
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
|
("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
|
|
("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
|
|
("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
|
|
("platform/monitoring/alerts.yaml", "NodeExporterDown"),
|
|
("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
|
|
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
|
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
|
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
|
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
|
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
|
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
|
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
|
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
|
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
|
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
|
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
|
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
|
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
|
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
|
}
|
|
|
|
# --------------------------------------------------------------- (1) the pins, in lockstep
|
|
def env_pin(rel: str, key: str) -> str | None:
|
|
f = ROOT / rel
|
|
if not f.is_file():
|
|
return None
|
|
m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
|
|
return m.group(1) if m else None
|
|
|
|
|
|
DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
|
|
CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
|
|
check(DEV_PIN is not None,
|
|
"infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
|
|
"baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
|
|
"have, or by nothing at all (which is where this repo was on 2026-09-12)")
|
|
check(CI_PIN is not None,
|
|
"ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
|
|
"rule engine, so CI and the devcontainer can silently diverge")
|
|
check(DEV_PIN is None or DEV_PIN == CI_PIN,
|
|
f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
|
|
f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
|
|
f"two files stay identical, so a laptop and CI run the same engine")
|
|
|
|
CF = ROOT / "infra/devcontainer/Containerfile"
|
|
cf_text = CF.read_text() if CF.is_file() else ""
|
|
check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
|
|
"infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
|
|
"the image build does not consume it")
|
|
check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
|
|
"infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
|
|
"release tarball (promtool ships inside it)")
|
|
check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
|
|
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
|
|
"that provenance line is how an operator tells a stale image from a current one")
|
|
|
|
DEVSH = ROOT / "infra/devcontainer/dev.sh"
|
|
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
|
|
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
|
|
"empty and the build would fetch a bogus URL")
|
|
|
|
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
|
if INSTALL.is_file():
|
|
itext = INSTALL.read_text()
|
|
check("PROMTOOL_VERSION" in itext
|
|
and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
|
|
".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
|
|
"on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
|
|
"would be enforced nowhere, which is the exact gap it exists to close")
|
|
|
|
# --------------------------------------------- (3)(4) one harness, and it fetches no engine
|
|
check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
|
|
f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
|
|
f"nothing would run the alert unit tests")
|
|
|
|
# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
|
|
# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
|
|
# and the one nothing runs is the one that rots out of sync with the rules.
|
|
harnesses: list[str] = []
|
|
fetchers: list[str] = []
|
|
FETCH_RE = re.compile(
|
|
r"(curl|wget)[^\n]*prometheus/prometheus/releases" # GitHub release download
|
|
r"|(podman|docker)\s+run[^\n]*prometheus", # container-image pull
|
|
re.I)
|
|
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
|
|
# ci/touched-requirements.sh changed_diff() rules 2-3):
|
|
# * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
|
|
# scanning itself would make both guards permanently red — the self-referential trap.
|
|
# * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
|
|
# reviewed seam (an image build, or the runner's tool-install step), from a version in
|
|
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
|
# version, the network and the machine are decided silently at test time, and a missing
|
|
# toolchain becomes a green run instead of an exit 3.
|
|
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
|
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
|
for p in sorted(ROOT.rglob("*.sh")):
|
|
rel = str(p.relative_to(ROOT))
|
|
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
|
continue
|
|
try:
|
|
text = p.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
|
|
if "promtool" in body and re.search(r"test\s+rules", body):
|
|
harnesses.append(rel)
|
|
if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
|
|
fetchers.append(rel)
|
|
|
|
check(harnesses == [str(HARNESS.relative_to(ROOT))],
|
|
f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
|
|
f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
|
|
f"one the gate does not run is the one that rots. Fold it in instead")
|
|
check(not fetchers,
|
|
f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
|
|
f"prometheus container image). promtool must come from the pinned toolchain "
|
|
f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
|
|
f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
|
|
f"toolchain has no promtool' into a green run — which is how it stayed missing")
|
|
|
|
# ------------------------------------------------------------------ discover the alert files
|
|
# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
|
|
# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
|
|
# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
|
|
# apps/, platform/, platform-config/ or clusters/.
|
|
def tracked_yaml() -> list[Path]:
|
|
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
|
capture_output=True, text=True)
|
|
if out.returncode == 0 and out.stdout:
|
|
return [ROOT / p for p in out.stdout.split("\0") if p]
|
|
# No usable index (the gate container mounts a worktree whose .git points outside the
|
|
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
|
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
|
|
|
|
|
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
|
ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
|
|
|
|
alert_files: dict[str, list[str]] = {}
|
|
for p in tracked_yaml():
|
|
rel = str(p.relative_to(ROOT))
|
|
if rel.startswith("conformance/") or not p.is_file():
|
|
continue
|
|
try:
|
|
text = p.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
if KIND_RE.search(text):
|
|
alert_files[rel] = ALERT_RE.findall(text)
|
|
|
|
check(len(alert_files) > 0,
|
|
"no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
|
|
"this check would silently audit nothing")
|
|
|
|
# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
|
|
# Two implementations of "which rules does this test cover" would eventually disagree, and the
|
|
# disagreement shows up as a file the gate believes is tested and the harness never touches.
|
|
listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
|
|
cwd=str(ROOT))
|
|
check(listing.returncode == 0,
|
|
f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
|
|
f"{(listing.stderr or listing.stdout).strip()[-600:]}")
|
|
covering: dict[str, list[str]] = defaultdict(list) # manifest -> [test files]
|
|
for line in listing.stdout.splitlines():
|
|
if "\t" in line:
|
|
tfile, manifest = line.split("\t", 1)
|
|
covering[manifest.strip()].append(tfile.strip())
|
|
|
|
for manifest in sorted(covering):
|
|
check(manifest in alert_files,
|
|
f"the harness reports {manifest} as under test, but it is not one of the "
|
|
f"PrometheusRule files found in the repo — the mapping and the tree disagree")
|
|
|
|
# --------------------------------------------------------- (2) the engine is the pinned one
|
|
version_line = ""
|
|
if PROMTOOL:
|
|
vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
|
|
out = (vp.stdout + vp.stderr).strip()
|
|
version_line = out.splitlines()[0] if out else ""
|
|
m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
|
|
found = m.group(1) if m else None
|
|
check(found is not None,
|
|
f"could not parse a version from `promtool --version`: {version_line!r}")
|
|
check(found is None or DEV_PIN is None or found == DEV_PIN,
|
|
f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
|
|
f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
|
|
f"one version and red under another is worse than no test. Run the gate in the pinned "
|
|
f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
|
|
f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
|
|
f"SUCCESS")
|
|
|
|
# ------------------------------------------------- (5) every rule-unit test file must pass
|
|
if PROMTOOL:
|
|
env = dict(os.environ, PROMTOOL=PROMTOOL)
|
|
run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
|
|
cwd=str(ROOT), env=env)
|
|
tailed = "\n ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
|
|
check(run.returncode == 0,
|
|
f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
|
|
f"rules do not behave as their own unit tests say:\n {tailed}")
|
|
|
|
# ------------------------------------------- (6) the harness is proven non-vacuous
|
|
# A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
|
|
# discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
|
|
# this, a harness that stopped evaluating anything (an empty extraction, a rule file written
|
|
# under the wrong name) would report green forever — which is the disease being treated.
|
|
work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
|
|
try:
|
|
(Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: selftest
|
|
spec:
|
|
groups:
|
|
- name: selftest
|
|
rules:
|
|
- alert: SelfTestCanary
|
|
expr: selftest_metric > 0
|
|
for: 5m
|
|
annotations:
|
|
runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
|
|
""")
|
|
test_yaml = """rule_files:
|
|
- generated.rules.yaml
|
|
evaluation_interval: 1m
|
|
tests:
|
|
- interval: 1m
|
|
name: canary fires only after `for` elapses
|
|
input_series:
|
|
- series: 'selftest_metric'
|
|
values: '1x10'
|
|
promql_expr_test:
|
|
- expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
|
eval_time: EVAL_TIME
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
|
value: 1
|
|
"""
|
|
synth_test = Path(work) / "synth.promtool-test.yaml"
|
|
synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
|
|
good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
|
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
|
check(good.returncode == 0,
|
|
"self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
|
|
"broken and every green result above is meaningless:\n "
|
|
+ "\n ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
|
|
# Mutation: assert firing at 2m, which `for: 5m` forbids.
|
|
synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
|
|
bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
|
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
|
check(bad.returncode != 0,
|
|
"self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
|
|
"the harness is not actually evaluating anything, so every green result above is "
|
|
"vacuous")
|
|
finally:
|
|
shutil.rmtree(work, ignore_errors=True)
|
|
|
|
# --------------------------------------------------------- (7) the gap inventory + ratchet
|
|
INVENTORY: list[str] = []
|
|
GAP_FILES: list[str] = []
|
|
alerts_total = covered_total = 0
|
|
|
|
for rel in sorted(alert_files):
|
|
names = alert_files[rel]
|
|
alerts_total += len(names)
|
|
check(len(names) > 0,
|
|
f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
|
|
f"parser or the file shape changed, and any unit test over it would pass vacuously")
|
|
tests = covering.get(rel, [])
|
|
if not tests:
|
|
GAP_FILES.append(rel)
|
|
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
|
|
check(rel in UNTESTED_FILES,
|
|
f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
|
|
f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
|
|
f"permanently-firing critical and a structurally inert alert, and neither was "
|
|
f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
|
|
f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
|
|
f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
|
|
f"visible instead of silent")
|
|
continue
|
|
|
|
check(rel not in UNTESTED_FILES,
|
|
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
|
f"shrink-only: delete the line so the next missing test is caught")
|
|
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
|
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
|
covered_total += len(covered)
|
|
for n in names:
|
|
if n in covered:
|
|
check((rel, n) not in UNTESTED_ALERTS,
|
|
f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
|
|
f"remove the stale entry (the register is shrink-only)")
|
|
else:
|
|
check((rel, n) in UNTESTED_ALERTS,
|
|
f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
|
|
f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
|
|
f"quietly grows untested rules")
|
|
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
|
|
f"{','.join(tests)}")
|
|
|
|
# stale register entries
|
|
for rel in sorted(UNTESTED_FILES - set(alert_files)):
|
|
check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
|
|
f"more — a stale exemption hides the next real gap; delete the line")
|
|
for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
|
|
check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
|
|
f"repo any more — delete those lines")
|
|
|
|
# --------------------------------------------------------------------------------- report
|
|
print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
|
|
for line in INVENTORY:
|
|
print(line)
|
|
print(f"\nharness: {HARNESS.relative_to(ROOT)} promtool: {version_line or 'NOT FOUND'}")
|
|
print(f"pinned: PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
|
|
f"v{DEV_PIN}")
|
|
print(f"alert files: {len(alert_files)} covered by a test file: "
|
|
f"{len(alert_files) - len(GAP_FILES)} with NO test file: {len(GAP_FILES)}")
|
|
print(f"alert rules: {alerts_total} unit-tested: {covered_total} "
|
|
f"NOT unit-tested: {alerts_total - covered_total}")
|
|
|
|
if GAP_FILES:
|
|
print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
|
|
print("rule nothing has ever evaluated, NOT a rule known to work):")
|
|
for g in GAP_FILES:
|
|
print(f" - {g} ({len(alert_files[g])} alert(s))")
|
|
if UNTESTED_ALERTS:
|
|
print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
|
|
for f, n in sorted(UNTESTED_ALERTS):
|
|
print(f" - {f} {n}")
|
|
|
|
print()
|
|
if not PROMTOOL and not FAILURES:
|
|
print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
|
|
"parsed and NO unit test was evaluated.", file=sys.stderr)
|
|
print(f" promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
|
|
f"ci/versions.env.", file=sys.stderr)
|
|
print(" Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
|
|
"make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
|
|
print(f" The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
|
|
f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
|
|
print(" This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
|
|
"green.", file=sys.stderr)
|
|
raise SystemExit(3)
|
|
|
|
if not FAILURES:
|
|
print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
|
|
f"({covered_total}/{alerts_total} alert rules unit-tested, "
|
|
f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
|
|
raise SystemExit(0)
|
|
print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
|
|
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" FAIL {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
PY
|