jakman85andClaude Opus 5
cf7d97e546
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
546 lines
31 KiB
Bash
Executable File
546 lines
31 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — alert-rules-unit-tested
|
|
# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
|
|
# VPN)." — spec docs/10-observability-and-alerting.md)
|
|
# R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
|
|
# — spec docs/14-testing-and-conformance.md)
|
|
#
|
|
# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
|
|
# --------------------------------------------------------------
|
|
# This repo has shipped BOTH halves of the alerting failure space to prod:
|
|
# * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
|
|
# it), and
|
|
# * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
|
|
# yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
|
|
# with no volume backups at all and nothing alerted (2026-09-12).
|
|
# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
|
|
# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
|
|
# rules` is the only tool that separates them, because the BAD state can be constructed on
|
|
# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
|
|
# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
|
|
# that existed acquired an engine off the network mid-run if one was missing. Coverage that
|
|
# looks present and is never enforced is the failure mode this check ends.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
|
|
# COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
|
|
# and then run over every test file in the repo,
|
|
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
|
|
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
|
|
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
|
|
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
|
|
# CI runner installs it for the l0 stage;
|
|
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
|
|
# evidence only if the engine matches the one prod evaluates rules with;
|
|
# (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
|
|
# GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
|
|
# 2026-09-12 and both are removed; this is the regression guard;
|
|
# (4) there is exactly ONE harness. Two were written independently within an hour of
|
|
# each other; the one nothing runs is the one that rots;
|
|
# (5) every rule-unit test file in the repo PASSES, run through that harness (which
|
|
# `promtool check rules` first, so a non-parsing rules file cannot make unit tests
|
|
# pass vacuously by finding no rules at all);
|
|
# (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
|
|
# through the SAME harness twice — intact (must PASS) and with one expectation
|
|
# mutated (must FAIL) — so a harness that has silently stopped evaluating anything
|
|
# is caught on every run, not only when someone thinks to try it;
|
|
# AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
|
|
# below, printed with counts on every run, and the register can only SHRINK.
|
|
#
|
|
# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
|
|
# ---------------------------------------------------------------------------------------
|
|
# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
|
|
# * FAIL on any missing test — correct in principle, but it makes the merge gate red for
|
|
# every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
|
|
# disabled gate protects nothing.
|
|
# * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
|
|
# tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
|
|
# enforcement.
|
|
# * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
|
|
# counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
|
|
# file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
|
|
# writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
|
|
# register cannot rot into fiction.
|
|
# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
|
|
# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
|
|
# register drifts, or a test that exists stops passing.
|
|
# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
|
|
# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
|
|
# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
|
|
# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
|
|
# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
|
|
# evaluating the rules with an engine prod does not run.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps):
|
|
# - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
|
|
# series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
|
|
# (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
|
|
# - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
|
|
# test says.
|
|
# - It does not prove a rule file is RECONCILED into a cluster. An alert file no
|
|
# kustomization includes is a separate "looks applied and isn't" failure, not audited here.
|
|
# - Coverage of an individual alert is detected by its NAME appearing in a test file that
|
|
# covers its manifest. A test file that merely mentions a name without asserting anything
|
|
# counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
|
|
# every author wrote a meaningful assertion. Review still owns test quality.
|
|
# - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
|
|
# against the repo's pin, which was established by a live read recorded in
|
|
# infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
|
|
|
|
# Honest SKIP (never green) if the thing under test is not present at all. Because this test
|
|
# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
|
|
# PrometheusRule-less repo is a vanished tree, not a pass.
|
|
if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
|
|
"${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
|
|
echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
|
|
exit 3
|
|
fi
|
|
|
|
# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
|
|
# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
|
|
# is reported as a gap and never as a pass.
|
|
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
|
|
if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
|
|
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
|
|
else
|
|
PROMTOOL_BIN=""
|
|
fi
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
|
|
"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
|
|
rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
from collections import defaultdict
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
HARNESS = Path(sys.argv[2])
|
|
PROMTOOL = sys.argv[3]
|
|
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
|
|
|
FAILURES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
# =============================================================================
|
|
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
|
|
#
|
|
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
|
|
# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
|
|
#
|
|
# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
|
|
# generated from one ServiceDefinition template, and the register's description of them as one
|
|
# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
|
|
# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
|
|
# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
|
|
# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
|
|
# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
|
|
# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
|
|
# now the only entry left and is where the next tests belong.
|
|
# =============================================================================
|
|
UNTESTED_FILES = {
|
|
"platform/kubevirt/alerts.yaml",
|
|
}
|
|
|
|
# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
|
|
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
|
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
|
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
|
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
|
# rules file by the suite convention but has no assertions of its own yet.
|
|
UNTESTED_ALERTS = {
|
|
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
|
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
|
("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
|
|
("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
|
|
("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
|
|
("platform/monitoring/alerts.yaml", "NodeExporterDown"),
|
|
("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
|
|
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
|
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
|
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
|
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
|
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
|
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
|
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
|
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
|
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
|
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
|
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
|
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
|
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
|
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
|
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
|
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
|
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
|
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
|
}
|
|
|
|
# --------------------------------------------------------------- (1) the pins, in lockstep
|
|
def env_pin(rel: str, key: str) -> str | None:
|
|
f = ROOT / rel
|
|
if not f.is_file():
|
|
return None
|
|
m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
|
|
return m.group(1) if m else None
|
|
|
|
|
|
DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
|
|
CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
|
|
check(DEV_PIN is not None,
|
|
"infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
|
|
"baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
|
|
"have, or by nothing at all (which is where this repo was on 2026-09-12)")
|
|
check(CI_PIN is not None,
|
|
"ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
|
|
"rule engine, so CI and the devcontainer can silently diverge")
|
|
check(DEV_PIN is None or DEV_PIN == CI_PIN,
|
|
f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
|
|
f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
|
|
f"two files stay identical, so a laptop and CI run the same engine")
|
|
|
|
CF = ROOT / "infra/devcontainer/Containerfile"
|
|
cf_text = CF.read_text() if CF.is_file() else ""
|
|
check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
|
|
"infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
|
|
"the image build does not consume it")
|
|
check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
|
|
"infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
|
|
"release tarball (promtool ships inside it)")
|
|
check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
|
|
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
|
|
"that provenance line is how an operator tells a stale image from a current one")
|
|
|
|
DEVSH = ROOT / "infra/devcontainer/dev.sh"
|
|
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
|
|
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
|
|
"empty and the build would fetch a bogus URL")
|
|
|
|
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
|
if INSTALL.is_file():
|
|
itext = INSTALL.read_text()
|
|
check("PROMTOOL_VERSION" in itext
|
|
and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
|
|
".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
|
|
"on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
|
|
"would be enforced nowhere, which is the exact gap it exists to close")
|
|
|
|
# --------------------------------------------- (3)(4) one harness, and it fetches no engine
|
|
check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
|
|
f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
|
|
f"nothing would run the alert unit tests")
|
|
|
|
# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
|
|
# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
|
|
# and the one nothing runs is the one that rots out of sync with the rules.
|
|
harnesses: list[str] = []
|
|
fetchers: list[str] = []
|
|
FETCH_RE = re.compile(
|
|
r"(curl|wget)[^\n]*prometheus/prometheus/releases" # GitHub release download
|
|
r"|(podman|docker)\s+run[^\n]*prometheus", # container-image pull
|
|
re.I)
|
|
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
|
|
# ci/touched-requirements.sh changed_diff() rules 2-3):
|
|
# * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
|
|
# scanning itself would make both guards permanently red — the self-referential trap.
|
|
# * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
|
|
# reviewed seam (an image build, or the runner's tool-install step), from a version in
|
|
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
|
# version, the network and the machine are decided silently at test time, and a missing
|
|
# toolchain becomes a green run instead of an exit 3.
|
|
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
|
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
|
for p in sorted(ROOT.rglob("*.sh")):
|
|
rel = str(p.relative_to(ROOT))
|
|
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
|
continue
|
|
try:
|
|
text = p.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
|
|
if "promtool" in body and re.search(r"test\s+rules", body):
|
|
harnesses.append(rel)
|
|
if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
|
|
fetchers.append(rel)
|
|
|
|
check(harnesses == [str(HARNESS.relative_to(ROOT))],
|
|
f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
|
|
f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
|
|
f"one the gate does not run is the one that rots. Fold it in instead")
|
|
check(not fetchers,
|
|
f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
|
|
f"prometheus container image). promtool must come from the pinned toolchain "
|
|
f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
|
|
f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
|
|
f"toolchain has no promtool' into a green run — which is how it stayed missing")
|
|
|
|
# ------------------------------------------------------------------ discover the alert files
|
|
# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
|
|
# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
|
|
# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
|
|
# apps/, platform/, platform-config/ or clusters/.
|
|
def tracked_yaml() -> list[Path]:
|
|
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
|
capture_output=True, text=True)
|
|
if out.returncode == 0 and out.stdout:
|
|
return [ROOT / p for p in out.stdout.split("\0") if p]
|
|
# No usable index (the gate container mounts a worktree whose .git points outside the
|
|
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
|
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
|
|
|
|
|
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
|
ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
|
|
|
|
alert_files: dict[str, list[str]] = {}
|
|
for p in tracked_yaml():
|
|
rel = str(p.relative_to(ROOT))
|
|
if rel.startswith("conformance/") or not p.is_file():
|
|
continue
|
|
try:
|
|
text = p.read_text()
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
if KIND_RE.search(text):
|
|
alert_files[rel] = ALERT_RE.findall(text)
|
|
|
|
check(len(alert_files) > 0,
|
|
"no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
|
|
"this check would silently audit nothing")
|
|
|
|
# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
|
|
# Two implementations of "which rules does this test cover" would eventually disagree, and the
|
|
# disagreement shows up as a file the gate believes is tested and the harness never touches.
|
|
listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
|
|
cwd=str(ROOT))
|
|
check(listing.returncode == 0,
|
|
f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
|
|
f"{(listing.stderr or listing.stdout).strip()[-600:]}")
|
|
covering: dict[str, list[str]] = defaultdict(list) # manifest -> [test files]
|
|
for line in listing.stdout.splitlines():
|
|
if "\t" in line:
|
|
tfile, manifest = line.split("\t", 1)
|
|
covering[manifest.strip()].append(tfile.strip())
|
|
|
|
for manifest in sorted(covering):
|
|
check(manifest in alert_files,
|
|
f"the harness reports {manifest} as under test, but it is not one of the "
|
|
f"PrometheusRule files found in the repo — the mapping and the tree disagree")
|
|
|
|
# --------------------------------------------------------- (2) the engine is the pinned one
|
|
version_line = ""
|
|
if PROMTOOL:
|
|
vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
|
|
out = (vp.stdout + vp.stderr).strip()
|
|
version_line = out.splitlines()[0] if out else ""
|
|
m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
|
|
found = m.group(1) if m else None
|
|
check(found is not None,
|
|
f"could not parse a version from `promtool --version`: {version_line!r}")
|
|
check(found is None or DEV_PIN is None or found == DEV_PIN,
|
|
f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
|
|
f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
|
|
f"one version and red under another is worse than no test. Run the gate in the pinned "
|
|
f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
|
|
f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
|
|
f"SUCCESS")
|
|
|
|
# ------------------------------------------------- (5) every rule-unit test file must pass
|
|
if PROMTOOL:
|
|
env = dict(os.environ, PROMTOOL=PROMTOOL)
|
|
run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
|
|
cwd=str(ROOT), env=env)
|
|
tailed = "\n ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
|
|
check(run.returncode == 0,
|
|
f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
|
|
f"rules do not behave as their own unit tests say:\n {tailed}")
|
|
|
|
# ------------------------------------------- (6) the harness is proven non-vacuous
|
|
# A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
|
|
# discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
|
|
# this, a harness that stopped evaluating anything (an empty extraction, a rule file written
|
|
# under the wrong name) would report green forever — which is the disease being treated.
|
|
work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
|
|
try:
|
|
(Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: selftest
|
|
spec:
|
|
groups:
|
|
- name: selftest
|
|
rules:
|
|
- alert: SelfTestCanary
|
|
expr: selftest_metric > 0
|
|
for: 5m
|
|
annotations:
|
|
runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
|
|
""")
|
|
test_yaml = """rule_files:
|
|
- generated.rules.yaml
|
|
evaluation_interval: 1m
|
|
tests:
|
|
- interval: 1m
|
|
name: canary fires only after `for` elapses
|
|
input_series:
|
|
- series: 'selftest_metric'
|
|
values: '1x10'
|
|
promql_expr_test:
|
|
- expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
|
eval_time: EVAL_TIME
|
|
exp_samples:
|
|
- labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
|
value: 1
|
|
"""
|
|
synth_test = Path(work) / "synth.promtool-test.yaml"
|
|
synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
|
|
good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
|
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
|
check(good.returncode == 0,
|
|
"self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
|
|
"broken and every green result above is meaningless:\n "
|
|
+ "\n ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
|
|
# Mutation: assert firing at 2m, which `for: 5m` forbids.
|
|
synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
|
|
bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
|
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
|
check(bad.returncode != 0,
|
|
"self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
|
|
"the harness is not actually evaluating anything, so every green result above is "
|
|
"vacuous")
|
|
finally:
|
|
shutil.rmtree(work, ignore_errors=True)
|
|
|
|
# --------------------------------------------------------- (7) the gap inventory + ratchet
|
|
INVENTORY: list[str] = []
|
|
GAP_FILES: list[str] = []
|
|
alerts_total = covered_total = 0
|
|
|
|
for rel in sorted(alert_files):
|
|
names = alert_files[rel]
|
|
alerts_total += len(names)
|
|
check(len(names) > 0,
|
|
f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
|
|
f"parser or the file shape changed, and any unit test over it would pass vacuously")
|
|
tests = covering.get(rel, [])
|
|
if not tests:
|
|
GAP_FILES.append(rel)
|
|
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
|
|
check(rel in UNTESTED_FILES,
|
|
f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
|
|
f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
|
|
f"permanently-firing critical and a structurally inert alert, and neither was "
|
|
f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
|
|
f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
|
|
f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
|
|
f"visible instead of silent")
|
|
continue
|
|
|
|
check(rel not in UNTESTED_FILES,
|
|
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
|
f"shrink-only: delete the line so the next missing test is caught")
|
|
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
|
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
|
covered_total += len(covered)
|
|
for n in names:
|
|
if n in covered:
|
|
check((rel, n) not in UNTESTED_ALERTS,
|
|
f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
|
|
f"remove the stale entry (the register is shrink-only)")
|
|
else:
|
|
check((rel, n) in UNTESTED_ALERTS,
|
|
f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
|
|
f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
|
|
f"quietly grows untested rules")
|
|
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
|
|
f"{','.join(tests)}")
|
|
|
|
# stale register entries
|
|
for rel in sorted(UNTESTED_FILES - set(alert_files)):
|
|
check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
|
|
f"more — a stale exemption hides the next real gap; delete the line")
|
|
for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
|
|
check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
|
|
f"repo any more — delete those lines")
|
|
|
|
# --------------------------------------------------------------------------------- report
|
|
print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
|
|
for line in INVENTORY:
|
|
print(line)
|
|
print(f"\nharness: {HARNESS.relative_to(ROOT)} promtool: {version_line or 'NOT FOUND'}")
|
|
print(f"pinned: PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
|
|
f"v{DEV_PIN}")
|
|
print(f"alert files: {len(alert_files)} covered by a test file: "
|
|
f"{len(alert_files) - len(GAP_FILES)} with NO test file: {len(GAP_FILES)}")
|
|
print(f"alert rules: {alerts_total} unit-tested: {covered_total} "
|
|
f"NOT unit-tested: {alerts_total - covered_total}")
|
|
|
|
if GAP_FILES:
|
|
print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
|
|
print("rule nothing has ever evaluated, NOT a rule known to work):")
|
|
for g in GAP_FILES:
|
|
print(f" - {g} ({len(alert_files[g])} alert(s))")
|
|
if UNTESTED_ALERTS:
|
|
print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
|
|
for f, n in sorted(UNTESTED_ALERTS):
|
|
print(f" - {f} {n}")
|
|
|
|
print()
|
|
if not PROMTOOL and not FAILURES:
|
|
print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
|
|
"parsed and NO unit test was evaluated.", file=sys.stderr)
|
|
print(f" promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
|
|
f"ci/versions.env.", file=sys.stderr)
|
|
print(" Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
|
|
"make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
|
|
print(f" The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
|
|
f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
|
|
print(" This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
|
|
"green.", file=sys.stderr)
|
|
raise SystemExit(3)
|
|
|
|
if not FAILURES:
|
|
print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
|
|
f"({covered_total}/{alerts_total} alert rules unit-tested, "
|
|
f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
|
|
raise SystemExit(0)
|
|
print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
|
|
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" FAIL {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
PY
|