feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
4865734c5f
commit
78d0f1b0f6
18 files changed
+950
-187
No files matched your search
@@ -18,7 +18,7 @@
|
||||
| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
|
||||
| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
|
||||
| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
|
||||
| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh`, or `make -C ci prom-rules`) |
|
||||
| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh` — the repo-wide harness, it also runs other components' companion `*.promtool-test.yaml`; enforced by the L0 check `alert-rules-unit-tested`) |
|
||||
| `kustomization.yaml` | builds all of the above |
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# promtool rule-unit tests for the platform-health.backups group (platform/monitoring/alerts.yaml).
|
||||
# Run with: platform/monitoring/tests/run.sh (or `make -C ci prom-rules`)
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# Every rule is pinned in BOTH directions, because this repo has shipped both failure modes to
|
||||
# prod: a permanently-firing alert (CNPG, 2026-09-12) and a structurally inert one
|
||||
|
||||
@@ -12,10 +12,19 @@ would let the fixture stay green while the deployed rule rots -- which is the cl
|
||||
this harness exists to catch (a permanently-firing CNPG alert and a structurally inert
|
||||
Velero alert both reached prod in 2026-09).
|
||||
|
||||
Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in
|
||||
platform/monitoring uses one, and leaving them alone keeps annotation text identical to
|
||||
the file under test. If a placeholder ever appears inside an `expr`, that is refused
|
||||
loudly rather than silently producing an untestable rule file.
|
||||
Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in any
|
||||
tested manifest uses one, and leaving them alone keeps annotation text identical to the
|
||||
file under test. If a placeholder ever appears inside an `expr`, that is refused loudly
|
||||
rather than silently producing an untestable rule file.
|
||||
|
||||
The cnpg-only script this replaced (platform/cnpg/run-alert-tests.sh, folded in 2026-09-12)
|
||||
SUBSTITUTED those placeholders instead, on the reasoning that "leaving a raw ${...} in a
|
||||
rendered rules file is exactly the kind of thing that gets copied somewhere it DOES
|
||||
matter". That hazard is real but it is addressed by WHERE the file goes, not by rewriting
|
||||
it: run.sh writes every generated rule file into a temp dir it deletes, so no rendered copy
|
||||
ever exists next to its source to be picked up. Verbatim is then the better default —
|
||||
one transformation fewer between the committed rule and the rule under test, and an
|
||||
`exp_alerts` assertion on annotation text still matches the file as shipped.
|
||||
|
||||
Group names are made unique per source file so two files may legally reuse a name.
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# platform/monitoring/tests/run.sh — Prometheus RULE-UNIT-TEST harness (no cluster).
|
||||
# platform/monitoring/tests/run.sh — the repo's Prometheus RULE-UNIT-TEST harness (no cluster).
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# This repo has now shipped both halves of the alerting failure space to prod:
|
||||
@@ -17,76 +17,228 @@
|
||||
# PrometheusRule CRs into a rule file) — never a copy, so a fixture cannot drift green while
|
||||
# the deployed rule rots.
|
||||
#
|
||||
# Usage: ./run.sh # run every *.test.yaml here
|
||||
# ./run.sh foo.test.yaml
|
||||
# ONE HARNESS, REPO-WIDE (consolidated 2026-09-12). This started as the monitoring-only
|
||||
# runner and there was a second, cnpg-only script (`platform/cnpg/run-alert-tests.sh`) doing
|
||||
# the same three steps by hand. Two harnesses means two behaviours, and the one nothing ran
|
||||
# rots — so the cnpg script is gone and this one covers EVERY PrometheusRule in the repo that
|
||||
# has unit tests. It still lives under platform/monitoring/ because observability owns alert
|
||||
# testing; the tests themselves live next to the rules they test.
|
||||
#
|
||||
# TWO TEST-FILE CONVENTIONS, both discovered automatically:
|
||||
# 1. COMPANION — `<manifest>.promtool-test.yaml` beside `<manifest>.yaml`. The manifest under
|
||||
# test is that sibling. This is how a platform component ships tests with its own alerts
|
||||
# (platform/cnpg/alerts.promtool-test.yaml).
|
||||
# 2. SUITE — `<dir>/tests/*.test.yaml`. The manifests under test are every PrometheusRule
|
||||
# file in `<dir>` (platform/monitoring/tests/backups.test.yaml covers alerts.yaml +
|
||||
# pvc-capacity-alerts.yaml, so one suite can assert across rule groups).
|
||||
#
|
||||
# WHAT THE ADAPTER DOES, AND WHY (inherited from the cnpg script's header, which was right):
|
||||
# * A PrometheusRule is a Kubernetes CR; promtool wants a bare rules file with `groups:` at
|
||||
# the top level. So `.spec.groups` is extracted (extract_rules.py), never re-typed.
|
||||
# * The extracted file is written under whatever name the test file's own `rule_files:`
|
||||
# declares, so a test file needs no knowledge of this harness.
|
||||
# * `promtool check rules` runs FIRST. A rules file that does not parse would make every
|
||||
# unit test "pass" vacuously by finding no rules at all.
|
||||
# * Everything generated goes to a temp dir. A generated file sitting next to its source is
|
||||
# how a stale copy becomes the thing under test — and it is also why leaving Flux's
|
||||
# `${BASE_DOMAIN}` placeholders verbatim (see extract_rules.py) is safe here.
|
||||
#
|
||||
# Usage: ./run.sh # every test file in the repo (both conventions)
|
||||
# ./run.sh platform/cnpg/alerts.promtool-test.yaml # one file
|
||||
# ./run.sh --list # print `<test-file>\t<manifest>` pairs, run nothing
|
||||
#
|
||||
# `--list` exists so the L0 check can ask THIS script which manifests are under test instead of
|
||||
# re-deriving the conventions itself. Two implementations of "which rules does this test cover"
|
||||
# would eventually disagree, and the disagreement would show up as a file the gate believes is
|
||||
# tested and the harness never touches. It needs no promtool, so it works on any host.
|
||||
#
|
||||
# Exit codes follow ci/lib.sh: 0 pass, 1 fail, 3 SKIPPED (toolchain unavailable — never green).
|
||||
# promtool is taken from $PATH if present, otherwise run from the pinned Prometheus image via
|
||||
# podman/docker. The image tag matches the version prod runs, so "passes here" means "passes
|
||||
# with the engine that will evaluate it".
|
||||
#
|
||||
# PROMTOOL COMES FROM THE PINNED TOOLCHAIN, AND NOWHERE ELSE. `PROMTOOL_VERSION` in
|
||||
# infra/devcontainer/versions.env (baked into the image) and ci/versions.env (installed on the
|
||||
# CI runner), pinned to the Prometheus version prod actually runs — promtool IS the rule
|
||||
# engine, so a test green under one version and red under another is worse than no test.
|
||||
# The two run-time acquisition paths this harness and the cnpg script used to have — a
|
||||
# `curl | tar -xz` of a GitHub release, and a `podman run` of quay.io/prometheus/prometheus —
|
||||
# are both REMOVED. Not because a pinned download is unusual here (the Containerfile downloads
|
||||
# every tool it bakes) but because those decided the version, the network and the machine
|
||||
# SILENTLY, at test time, on whatever host ran the tests instead of once at reviewed
|
||||
# image-build time; and each one quietly turned "the toolchain has no promtool" into a green
|
||||
# run, which is how the toolchain stayed without one. Missing promtool is now exit 3, never a
|
||||
# pass. Override with PROMTOOL=/path/to/promtool.
|
||||
#
|
||||
# ENFORCEMENT. This script is the developer entrypoint. The merge gate runs it through
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested, which additionally reports every
|
||||
# PrometheusRule in the repo that has NO unit tests at all.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
MONITORING_DIR="$(cd "${HERE}/.." && pwd)"
|
||||
PROM_IMAGE="${PROM_IMAGE:-quay.io/prometheus/prometheus:v3.14.0}"
|
||||
ROOT="$(cd "${HERE}/../../.." && pwd)"
|
||||
|
||||
MANIFESTS=(
|
||||
"${MONITORING_DIR}/alerts.yaml"
|
||||
"${MONITORING_DIR}/pvc-capacity-alerts.yaml"
|
||||
)
|
||||
|
||||
if [ "$#" -gt 0 ]; then
|
||||
TESTS=("$@")
|
||||
else
|
||||
mapfile -t TESTS < <(find "${HERE}" -maxdepth 1 -name '*.test.yaml' | sort)
|
||||
fi
|
||||
if [ "${#TESTS[@]}" -eq 0 ]; then
|
||||
echo "run.sh: no *.test.yaml found in ${HERE}" >&2
|
||||
exit 3
|
||||
fi
|
||||
LIST_ONLY=0
|
||||
if [ "${1:-}" = "--list" ]; then LIST_ONLY=1; shift; fi
|
||||
|
||||
# ---------------------------------------------------------------- toolchain resolution
|
||||
RUNNER=""
|
||||
if command -v promtool >/dev/null 2>&1; then
|
||||
RUNNER="native"
|
||||
elif command -v podman >/dev/null 2>&1; then
|
||||
RUNNER="podman"
|
||||
elif command -v docker >/dev/null 2>&1; then
|
||||
RUNNER="docker"
|
||||
# Skipped for --list: that mode runs no rules, and the L0 check must be able to resolve the
|
||||
# test->manifest mapping (and report the gaps) even on a host with no promtool.
|
||||
PROMTOOL_BIN=""
|
||||
if [ "$LIST_ONLY" = 0 ]; then
|
||||
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
|
||||
if [ -z "${PROMTOOL_BIN}" ]; then
|
||||
pinned="$(sed -n 's/^PROMTOOL_VERSION="\{0,1\}\([^"]*\)"\{0,1\}.*/\1/p' \
|
||||
"${ROOT}/ci/versions.env" 2>/dev/null | head -1)"
|
||||
{
|
||||
echo "run.sh: SKIP (exit 3, NOT a pass) — promtool is not on PATH, and this harness no"
|
||||
echo " longer fetches one (no GitHub release download, no container-image pull)."
|
||||
echo " promtool is pinned at v${pinned:-<see ci/versions.env>} — the Prometheus version prod runs:"
|
||||
echo " * infra/devcontainer/versions.env PROMTOOL_VERSION (baked into the image)"
|
||||
echo " * ci/versions.env PROMTOOL_VERSION (installed on the CI runner)"
|
||||
echo " Run it in the pinned toolbox:"
|
||||
echo " ./infra/devcontainer/dev.sh build # if the image predates the promtool pin"
|
||||
echo " ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh'"
|
||||
echo " Or point at a local binary of the same version: PROMTOOL=/path/to/promtool ./run.sh"
|
||||
} >&2
|
||||
exit 3
|
||||
fi
|
||||
# A PROMTOOL= that does not resolve is an OPERATOR MISTAKE, not a missing toolchain: exit 2,
|
||||
# so a typo'd override cannot masquerade as the honest "no promtool here" skip above.
|
||||
if [ ! -x "${PROMTOOL_BIN}" ]; then
|
||||
echo "run.sh: PROMTOOL='${PROMTOOL_BIN}' is not an executable file — fix the override" >&2
|
||||
exit 2
|
||||
fi
|
||||
# Absolutised: each test runs with cwd inside a temp dir, where a relative PROMTOOL= breaks
|
||||
# (it did, during development).
|
||||
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------- which test files to run
|
||||
# NOT `git ls-files`: the gate runs inside a container that mounts only the worktree, whose
|
||||
# `.git` file points outside the mount, so git cannot read the index there — and with
|
||||
# `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
|
||||
# (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
|
||||
# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
|
||||
discover() {
|
||||
# Companion convention, anywhere in the tree.
|
||||
find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
|
||||
# Suite convention: this directory's own *.test.yaml files.
|
||||
find "${HERE}" -maxdepth 1 -name '*.test.yaml' -print 2>/dev/null
|
||||
}
|
||||
|
||||
TESTS=()
|
||||
if [ "$#" -gt 0 ]; then
|
||||
for t in "$@"; do
|
||||
# Accept absolute, repo-relative, or cwd-relative paths.
|
||||
if [ -f "$t" ]; then TESTS+=("$(cd "$(dirname "$t")" && pwd)/$(basename "$t")")
|
||||
elif [ -f "${ROOT}/$t" ]; then TESTS+=("${ROOT}/$t")
|
||||
else echo "run.sh: no such test file: $t" >&2; exit 2
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo "run.sh: SKIP — no promtool on PATH and no podman/docker to run ${PROM_IMAGE}" >&2
|
||||
while IFS= read -r abs; do
|
||||
[ -n "$abs" ] && TESTS+=("$abs")
|
||||
done < <(discover | sort -u)
|
||||
fi
|
||||
|
||||
if [ "${#TESTS[@]}" -eq 0 ]; then
|
||||
echo "run.sh: SKIP (exit 3) — no *.promtool-test.yaml or tests/*.test.yaml found in ${ROOT}" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
# ---------------------------------------------------------------- manifest resolution
|
||||
# Which committed PrometheusRule files a given test file tests. The conventions are the
|
||||
# CONTRACT (see the header); anything else is refused rather than guessed at, because a test
|
||||
# silently run against the wrong (or an empty) rule set is the vacuous pass this harness exists
|
||||
# to prevent.
|
||||
manifests_for() {
|
||||
local test_abs="$1" base dir
|
||||
base="$(basename "$test_abs")"
|
||||
dir="$(dirname "$test_abs")"
|
||||
case "$base" in
|
||||
*.promtool-test.yaml)
|
||||
local sib="${dir}/${base%.promtool-test.yaml}.yaml"
|
||||
if [ ! -f "$sib" ]; then
|
||||
echo "run.sh: ${test_abs#"${ROOT}/"} has no companion manifest (${sib#"${ROOT}/"})" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$sib" ;;
|
||||
*.test.yaml)
|
||||
if [ "$(basename "$dir")" != tests ]; then
|
||||
echo "run.sh: ${test_abs#"${ROOT}/"}: a *.test.yaml must live in a tests/ directory so its" >&2
|
||||
echo " manifests are the PrometheusRules of the component above it" >&2
|
||||
return 1
|
||||
fi
|
||||
local comp; comp="$(dirname "$dir")"
|
||||
local found=0 f
|
||||
while IFS= read -r f; do
|
||||
found=1; printf '%s\n' "$f"
|
||||
done < <(grep -rl --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
|
||||
"$comp" --exclude-dir=tests 2>/dev/null | sort)
|
||||
if [ "$found" = 0 ]; then
|
||||
echo "run.sh: ${test_abs#"${ROOT}/"}: no PrometheusRule manifest in ${comp#"${ROOT}/"}" >&2
|
||||
return 1
|
||||
fi ;;
|
||||
*)
|
||||
echo "run.sh: ${test_abs#"${ROOT}/"}: unrecognised test-file name; use" >&2
|
||||
echo " <manifest>.promtool-test.yaml (companion) or tests/<name>.test.yaml (suite)" >&2
|
||||
return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# The test files reference the generated rule file by bare name, so everything must sit in one
|
||||
# directory (that is also the only directory the container needs mounted).
|
||||
# The rule-file name(s) a test file expects, from its own `rule_files:` block — so the
|
||||
# extracted rules land under the name the test already references and no test file needs to
|
||||
# know this harness exists.
|
||||
rule_file_names() {
|
||||
sed -n '/^rule_files:[[:space:]]*$/,/^[^[:space:]-]/p' "$1" \
|
||||
| sed -n 's/^[[:space:]]*-[[:space:]]*//p' | sed 's/[[:space:]]*$//' | grep . || true
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------------------- --list
|
||||
if [ "$LIST_ONLY" = 1 ]; then
|
||||
lrc=0
|
||||
for test_abs in "${TESTS[@]}"; do
|
||||
mapfile -t mans < <(manifests_for "$test_abs") || { lrc=1; continue; }
|
||||
for m in "${mans[@]}"; do
|
||||
printf '%s\t%s\n' "${test_abs#"${ROOT}/"}" "${m#"${ROOT}/"}"
|
||||
done
|
||||
done
|
||||
exit "$lrc"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------- run
|
||||
rc=0
|
||||
python3 "${HERE}/extract_rules.py" --out "${WORK}/generated.rules.yaml" "${MANIFESTS[@]}" || rc=$?
|
||||
if [ "$rc" -eq 4 ]; then
|
||||
echo "run.sh: SKIP — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
|
||||
exit 3
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
exit "$rc"
|
||||
fi
|
||||
for test_abs in "${TESTS[@]}"; do
|
||||
rel="${test_abs#"${ROOT}/"}"
|
||||
mapfile -t mans < <(manifests_for "$test_abs") || { rc=1; continue; }
|
||||
mapfile -t rule_names < <(rule_file_names "$test_abs")
|
||||
if [ "${#rule_names[@]}" -eq 0 ]; then
|
||||
echo "run.sh: ${rel}: no \`rule_files:\` entries — promtool would test nothing" >&2
|
||||
rc=1; continue
|
||||
fi
|
||||
|
||||
names=()
|
||||
for t in "${TESTS[@]}"; do
|
||||
cp "$t" "${WORK}/"
|
||||
names+=("$(basename "$t")")
|
||||
work="$(mktemp -d)"
|
||||
# The test file references its rule file by bare name, so everything sits in one directory.
|
||||
gen="${work}/${rule_names[0]}"
|
||||
ex_rc=0
|
||||
python3 "${HERE}/extract_rules.py" --out "$gen" "${mans[@]}" || ex_rc=$?
|
||||
if [ "$ex_rc" -eq 4 ]; then
|
||||
echo "run.sh: SKIP (exit 3) — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
|
||||
rm -rf "$work"; exit 3
|
||||
elif [ "$ex_rc" -ne 0 ]; then
|
||||
rm -rf "$work"; rc=1; continue
|
||||
fi
|
||||
# A test file may legally name the same rule file more than once, or several names.
|
||||
for n in "${rule_names[@]:1}"; do cp "$gen" "${work}/${n}"; done
|
||||
cp "$test_abs" "${work}/"
|
||||
|
||||
# Syntax gate first: rules that do not parse would make every unit test pass vacuously.
|
||||
echo "run.sh: promtool check rules ${rel} (manifests: ${mans[*]#"${ROOT}/"})" >&2
|
||||
( cd "$work" && "$PROMTOOL_BIN" check rules "${rule_names[0]}" ) || rc=1
|
||||
echo "run.sh: promtool test rules ${rel}" >&2
|
||||
( cd "$work" && "$PROMTOOL_BIN" test rules "$(basename "$test_abs")" ) || rc=1
|
||||
rm -rf "$work"
|
||||
done
|
||||
chmod -R a+rX "${WORK}"
|
||||
|
||||
echo "run.sh: promtool test rules (${RUNNER}) ${names[*]}" >&2
|
||||
case "$RUNNER" in
|
||||
native)
|
||||
( cd "${WORK}" && promtool test rules "${names[@]}" ) ;;
|
||||
podman)
|
||||
podman run --rm --user 0 -v "${WORK}:/rules:z" -w /rules \
|
||||
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
||||
docker)
|
||||
docker run --rm --user 0 -v "${WORK}:/rules" -w /rules \
|
||||
--entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
|
||||
esac
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
echo "run.sh: OK — ${#TESTS[@]} rule-unit test file(s) passed under $("$PROMTOOL_BIN" --version 2>&1 | head -1)" >&2
|
||||
fi
|
||||
exit "$rc"
|
||||
Reference in new issue
Block a user