fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors

Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.

THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.

FIXED IN THE GENERATOR, not in twelve rendered files:
  * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
    service's own workload object —
    `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
    `kube_statefulset_status_replicas_ready` for a StatefulSet, and
    `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
    KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
    exporter and no new target. The `or absent(...)` half is load-bearing: the signal
    DISAPPEARING now fires the alert instead of silencing it.
  * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
    ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
    scrape together: the named Service port, the monitor endpoint, and the
    monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
    dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
    kube-state-metrics liveness.
  * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
    it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
    would have made its `absent()` half fire FOREVER — the other failure mode this repo has
    shipped.
  * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
    must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
    must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.

PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
  * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
    the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
    keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
    keycloak-db StatefulSet ready in the same namespace).
  * mutation-checked: six mutations, each turning its suite red — reverting plex to
    `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
    capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
    inverting plex's `== 0` to `!= 0`. Control run green.
  * live on prod with `promtool query instant`: all ten deployed app Deployments export
    `kube_deployment_status_replicas_available`,
    `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
    `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
    SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
    twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
    — its rule is not applied either.

DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.

GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.

Also fixed, found on the way:
  * platform/monitoring/tests/run.sh discovered the SUITE convention only inside
    platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
    was never run and still looked like coverage. Now discovered tree-wide; a misplaced
    *.test.yaml fails loudly instead of vanishing.
  * KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
    apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
    gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
  * renderer-contract.md claimed the default-deny baseline carried a standing monitoring
    scrape allow. It never did; the claim is corrected and the allow is now real, per declared
    metrics endpoint and scoped to that one port.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 17:11:30 -05:00
1 parent 72f0ba4e1f
commit cf7d97e546
29 files changed
+1632 -428

No files matched your search

@@ -0,0 +1,105 @@
# =============================================================================
# L0 STATIC — every ServiceMonitor endpoint names a scrape target that RESOLVES
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE DEFECT THIS CLOSES, AND WHY ITS SIBLING COULD NOT.
# alert-metrics-have-scrape-targets audits scrape wiring FROM THE ALERT SIDE: it walks alert rules
# and asks whether something scrapes the metric each one reads. Its assertion (4) is the `up{job=X}`
# axis, and it is reachable ONLY through an alert that selects that job, only where the Service and
# the ServiceMonitor sit in the SAME FILE. Its own header names the hole. This test is the other
# direction: it walks the MONITORS, and asks of each one whether the target it names can exist —
# with no alert rule required anywhere.
#
# That difference is not academic. On 2026-09-13 all twelve apps/* ServiceMonitors named
# `endpoints[].port: metrics` and NO app Service defined a port of that name, so each discovered
# ZERO targets (verified on prod: `count by (job) (up)` listed 24 jobs, none of the twelve; both
# `up{job="plex"}` and `up{job="gitea"}` returned nothing). That was caught only because each app
# also shipped an `up{job=...}` alert. THE FIX TO THOSE ALERTS DELETED THE ONLY THING THAT MADE THE
# BROKEN MONITORS VISIBLE — they no longer read `up{job=...}` at all — so a monitor that discovers
# nothing would from that moment have been invisible again, and the twelve register entries that
# recorded the gap correctly became stale and were removed. A monitor scraping nothing is a defect
# on its own terms: it is dead configuration that reads as live coverage, and every dashboard,
# recording rule or future alert built on its series inherits the emptiness.
#
# GIVEN every ServiceMonitor and PodMonitor this repo authors,
# WHEN each endpoint's scrape target is resolved against every Service this repo authors,
# THEN (1) every endpoint names a target AT ALL (`port` or `targetPort`) — an endpoint naming
# neither scrapes nothing and is accepted silently by the operator;
# (2) every endpoint that names a `port` names a port that at least one Service its selector
# matches ACTUALLY DEFINES — the exact disagreement that made twelve monitors inert.
# Resolution is by SELECTOR ACROSS THE WHOLE TREE, not by same-file adjacency, so moving
# a Service into its own file cannot defeat it;
# (3) a monitor whose selector matches no Service in this repo is UNAUDITABLE (its Service is
# created by a Helm chart or an operator, so the port name lives outside the tree) and
# must be DECLARED in KNOWN_EXTERNAL_MONITORS, naming who creates the Service. That
# register is shrink-only: a monitor that later gains an in-repo Service, or that is
# deleted, FAILS as stale, so the unauditable set cannot quietly grow;
# (4) THE RESOLVER ITSELF IS PROVEN, on synthetic fixtures inside the check — one agreeing
# pair it must accept, one disagreeing pair it must reject. This is not decoration: after
# the fix there are zero in-repo monitors left for (2) to judge, and a guard whose only
# evidence is "nothing failed" is indistinguishable from a guard that stopped working.
# The fixtures are the exact shape catalog/render.py emits from
# `observability.metricsEndpoint`;
# (5) the GENERATOR cannot reintroduce the class: catalog/render.py must not hardcode an
# endpoint port name — it must take it from the ServiceDefinition, the same declaration
# that renders the Service port. A hardcoded literal is how twelve services acquired an
# identical wrong port name in one edit;
# AND (6) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
#
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps apps/* to NOTHING, so a change that
# adds a ServiceMonitor under apps/ attributes no requirement and would filter this check out of
# exactly the change that breaks it. Same reasoning as its sibling.
#
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
# the sibling check use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# NOT proven (honest gaps; this is static):
# - That a resolved target is UP, answers, or emits the series an alert wants. The sibling check
# owns the metric-name axis and admits it cannot prove the SERIES NAME either; both are live
# reads and belong to a future L1/L3 check against /api/v1/label/__name__/values.
# - That a NetworkPolicy permits the scrape. The default-deny baseline drops it unless an allow is
# rendered, which is now part of what `observability.metricsEndpoint` emits
# (r_monitoring_ingress_policy) — but this check does not verify the allow, only the ports.
# - For a monitor in KNOWN_EXTERNAL_MONITORS, nothing about the chart's port names. The register
# records WHO owns them so the claim is reviewable, which is all a static check can offer.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: servicemonitor-ports-resolve
labels:
conformance.home.k3s/R-OBS-01: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-01"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceMonitor/PodMonitor and every
# Service in the tree (apps/wave-*, apps/vms, platform/keycloak, platform/kubevirt,
# platform/oauth2-proxy, platform/monitoring, platform/crafty, platform/homepage,
# runbooks/serving). No single requirement path-maps to that set, and ci/touched-requirements.sh
# maps apps/* to no requirement at all — so a requirement-scoped selection would skip this
# check on precisely the changes that can break it.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every ServiceMonitor/PodMonitor endpoint this repo authors names a scrape target that
resolves — a port some matching Service actually defines — the monitors whose Services come
from outside the tree are enumerated in a shrink-only register, and the resolver that decides
this is itself proven on fixtures, so a monitor can never again be authored, reviewed and
merged against a port nothing serves.
spec:
steps:
- name: resolve-every-monitor-endpoint-against-every-service-this-repo-authors
description: >-
Inventory every ServiceMonitor/PodMonitor and Service in the tree, resolve each endpoint's
port against the Services its selector matches, ratchet the register of monitors whose
Service is chart-owned, prove the resolver on synthetic agreeing/disagreeing fixtures, and
assert the renderer does not hardcode an endpoint port name.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,416 @@
#!/usr/bin/env bash
# =============================================================================
# L0 STATIC CHECK — every ServiceMonitor/PodMonitor endpoint names a scrape target that RESOLVES.
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
#
# alert-metrics-have-scrape-targets audits scrape wiring from the ALERT side, and its `up{job=X}`
# axis is reachable only through an alert that selects that job, only for a Service and a
# ServiceMonitor in the SAME FILE. That is how it caught the twelve inert apps/* monitors on
# 2026-09-13 (`endpoints[].port: metrics`, and no app Service defined such a port; on prod
# `count by (job) (up)` listed 24 jobs and none of the twelve appeared). But FIXING those alerts
# removed their `up{job=...}` selectors, which removed the only path by which the broken monitors
# were visible at all. This check walks the MONITORS instead, needs no alert rule, and resolves by
# SELECTOR across the whole tree — so a monitor that discovers nothing fails on its own terms.
#
# ASSERTIONS
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
# (1) every endpoint names a scrape target at all — `port` or `targetPort`;
# (2) every endpoint naming a `port` names a port at least one Service its selector matches
# actually DEFINES;
# (3) a monitor whose selector matches no in-repo Service is unauditable (chart/operator-owned
# Service) and must be declared in KNOWN_EXTERNAL_MONITORS — shrink-only, stale entries FAIL;
# (4) the RESOLVER is proven on synthetic fixtures (one agreeing pair, one disagreeing pair),
# because after the fix there are zero in-repo monitors for (2) to judge and a guard whose
# only evidence is "nothing failed" cannot be told apart from one that stopped working;
# (5) catalog/render.py does not HARDCODE an endpoint port name — it derives it from the
# ServiceDefinition, the same declaration that renders the Service port. A hardcoded literal
# is how twelve services acquired one identical wrong port name in a single edit;
# (6) the check is LOAD-BEARING: whole-repo scope declared here, honoured by ci/l0.sh's
# --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written ci/l0.sh
# stage (whole-repo-invariants-wired fails the gate if one appears).
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC):
# - Nothing about liveness: that a resolved target is UP, answers, or emits the series a rule
# wants. The sibling check owns the metric-name axis and admits it cannot prove the series NAME
# either. Both need a live read of /api/v1/label/__name__/values (a future L1/L3 check).
# - Nothing about NetworkPolicy. An app namespace's default-deny baseline is `ingress: []`, so a
# scrape is dropped unless an allow is rendered; that allow is now part of what
# `observability.metricsEndpoint` emits (catalog/render.py r_monitoring_ingress_policy), but this
# check verifies ports, not policy.
# - Nothing about a chart's port names, for a monitor in KNOWN_EXTERNAL_MONITORS. The register
# records WHO owns them so the claim is reviewable; that is all a static check can offer.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
#
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns into
# rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the repo
# contains no ServiceMonitor or PodMonitor at all — genuinely nothing to audit.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
if ! grep -rlE '^[[:space:]]*kind:[[:space:]]*(Service|Pod)Monitor[[:space:]]*$' \
"${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
echo "check SKIPPED: no ServiceMonitor/PodMonitor found under platform/ or apps/ — nothing to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every ServiceMonitor/PodMonitor endpoint's scrape target resolves.
PyYAML (the same dependency as catalog/render.py and the sibling check) rather than line-based
parsing: a monitor's selector, namespaceSelector and endpoint list are nested structures, and a
line-based reader of them is precisely the approximate parser that silently stops matching — the
failure class this check exists to prevent.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================================================
# REGISTER — KNOWN_EXTERNAL_MONITORS. Shrink-only.
#
# A monitor whose selector matches NO Service in this repo cannot be audited here: the Service is
# created by a Helm chart or an operator, so the port name lives outside the tree. That is a real
# limit, not a pass — so each one is DECLARED, with WHO creates the Service it selects, and the
# register can only shrink. If such a monitor later gains an in-repo Service its entry goes stale
# and FAILS; if it is deleted, likewise. The same ratchet reasoning as the two registers in
# alert-metrics-have-scrape-targets: failing outright would make the gate red for chart-owned
# monitors that are perfectly correct, and a gate that is red for correct code gets disabled.
#
# Keyed by "<relpath>:<monitor name>".
# =============================================================================================
KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
"platform/keycloak/servicemonitor.yaml:keycloak":
"selects the operator-created `keycloak-service` (labels app=keycloak) — the Keycloak "
"operator owns that Service. Scrapes by targetPort 9000 (the Quarkus management "
"interface), so it names no Service port at all and assertion (2) has nothing to resolve.",
"platform/kubevirt/servicemonitor.yaml:kubevirt":
"selects the KubeVirt operator's own `kubevirt-prometheus-metrics` Service (label "
"prometheus.kubevirt.io=true), created by the KubeVirt CR, not by this repo. Its `metrics` "
"port is defined there. Live on prod: job kubevirt-prometheus-metrics, 11 targets up.",
"platform/oauth2-proxy/servicemonitor.yaml:oauth2-proxy":
"selects the oauth2-proxy Helm chart's Service (label app=oauth2-proxy); the chart defines "
"its `metrics` port. Live on prod: job oauth2-proxy, 2 targets up.",
"platform/monitoring/flux-podmonitor.yaml:flux-system":
"a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
"Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
"manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
}
MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
def is_test_fixture(p: Path) -> bool:
"""conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
The l1-rendering goldens are expected-output copies of rendered apps; the l2/l3 fixtures are
throwaway objects a behavioural test applies itself. Auditing either would be auditing a copy —
the same "reference, not declaration" distinction ci/touched-requirements.sh draws for fixture
paths, and the same exclusion the sibling check makes for goldens. The templates they come from
ARE audited, through apps/ and catalog/.
"""
parts = p.parts
return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
# ============================================================ (0) INVENTORY
# services: list of (relpath, namespace, name, labels, {named ports})
# monitors: list of (relpath, kind, namespace, name, selector, namespaceSelector, [endpoints])
services: list[tuple] = []
monitors: list[tuple] = []
unparseable: list[str] = []
for p in sorted(ROOT.rglob("*.yaml")):
if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
continue
txt = read(p)
if not txt:
continue
if not re.search(r"^\s*kind:\s*(Service|ServiceMonitor|PodMonitor)\s*$", txt, re.M):
continue
rel = str(p.relative_to(ROOT))
try:
docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
except yaml.YAMLError as exc:
unparseable.append(f"{rel} ({exc})")
continue
for d in docs:
kind = d.get("kind")
md = d.get("metadata") or {}
spec = d.get("spec") or {}
if kind == "Service":
names = {str(port["name"]) for port in (spec.get("ports") or [])
if isinstance(port, dict) and port.get("name")}
services.append((rel, md.get("namespace"), md.get("name"),
dict(md.get("labels") or {}), names))
elif kind in MONITOR_KINDS:
eps = spec.get("endpoints") or spec.get("podMetricsEndpoints") or []
monitors.append((rel, kind, md.get("namespace"), md.get("name"),
dict((spec.get("selector") or {}).get("matchLabels") or {}),
dict(spec.get("namespaceSelector") or {}),
[e for e in eps if isinstance(e, dict)]))
for u in unparseable:
check(False, f"{u} is not parseable YAML — every monitor and Service in it went unaudited")
check(len(monitors) > 0,
"no ServiceMonitor/PodMonitor parsed out of the tree, yet the grep gate above found one. The "
"parse contract broke and this check would pass vacuously while every scrape target went "
"unaudited")
check(len(services) >= 10,
f"only {len(services)} Service(s) parsed from the tree. This repo authors more than that "
f"(every apps/wave-* ServiceDefinition renders one), so the inventory has stopped being "
f"populated — assertion (2) would then resolve nothing and pass for the wrong reason")
# ============================================================ THE RESOLVER
def monitor_namespaces(mon_ns, ns_selector) -> set | None:
"""Namespaces a monitor selects. None means ANY (`any: true`).
An empty namespaceSelector defaults to the monitor's own namespace — the operator's documented
behaviour, and the case every repo-authored monitor uses.
"""
if ns_selector.get("any") is True:
return None
names = ns_selector.get("matchNames")
if names:
return {str(n) for n in names}
return {mon_ns} if mon_ns else set()
def matching_services(selector: dict, namespaces: set | None) -> list[tuple]:
"""Services whose labels are a SUPERSET of the selector and whose namespace is in range.
An EMPTY selector matches every Service in range (Kubernetes label-selector semantics). A
PodMonitor selector is compared against Service labels too — deliberately, so the resolver has
one code path; when that yields no match the monitor lands in the unauditable register with the
reason recorded, which is the honest outcome for a pod-side target this repo does not author.
"""
out = []
for svc in services:
_, svc_ns, _, labels, _ = svc
if namespaces is not None and svc_ns not in namespaces:
continue
if all(labels.get(k) == v for k, v in selector.items()):
out.append(svc)
return out
def resolve(mon: tuple) -> tuple[list[str], list[tuple]]:
"""(failures, matched services) for one monitor. The unit assertions (1) and (2) live here so
the SELF-TEST at the bottom exercises the same code path the repo audit does."""
rel, kind, ns, name, selector, ns_sel, eps = mon
site = f"{rel}:{name}"
problems: list[str] = []
matched = matching_services(selector, monitor_namespaces(ns, ns_sel))
for i, ep in enumerate(eps):
if not ep.get("port") and not ep.get("targetPort"):
problems.append(
f"{site} endpoint[{i}] names NEITHER `port` NOR `targetPort`. The operator accepts "
f"that silently and the monitor scrapes nothing")
if not eps:
problems.append(
f"{site} has NO endpoints at all — it can never produce a target, and nothing about "
f"the object looks wrong")
named = [(i, str(ep["port"])) for i, ep in enumerate(eps) if ep.get("port")]
if named and matched:
defined: set = set()
for svc in matched:
defined |= svc[4]
for i, port in named:
if port not in defined:
where = ", ".join(f"{s[0]}:{s[2]}" for s in matched)
problems.append(
f"{site} endpoint[{i}] scrapes port {port!r}, which NO Service its selector "
f"matches defines. Matched Service(s): {where}; the port names they define are "
f"{sorted(defined) or ['(none)']}. The monitor therefore discovers ZERO "
f"targets: no `up{{job=\"{name}\"}}` series exists, anything built on its "
f"series is empty, and PrometheusTargetsMissing cannot see it either because a "
f"target that was never discovered has no up series to be 0. Give the Service a "
f"named port the workload actually serves metrics on (for a ServiceDefinition: "
f"`observability.metricsEndpoint`, which renders the Service port, this "
f"endpoint and the monitoring ingress allow together), or scrape by targetPort")
return problems, matched
# ============================================================ (1)(2)(3) THE REPO AUDIT
auditable = 0
for mon in monitors:
rel, kind, ns, name, selector, ns_sel, eps = mon
site = f"{rel}:{name}"
problems, matched = resolve(mon)
for msg in problems:
check(False, msg)
if not problems:
# An endpoint-shape assertion passed for this monitor; count it so the report is honest
# about how much of the run was real work.
check(True, f"{site}: every endpoint names a scrape target")
if matched:
auditable += 1
check(site not in KNOWN_EXTERNAL_MONITORS,
f"{site} is listed in KNOWN_EXTERNAL_MONITORS but its selector NOW matches "
f"{len(matched)} Service(s) this repo authors — the register is shrink-only: delete "
f"the entry so the monitor is audited for real")
else:
check(site in KNOWN_EXTERNAL_MONITORS,
f"{site} selects {selector or '(everything)'} in namespace(s) "
f"{sorted(monitor_namespaces(ns, ns_sel) or ['<any>'])} and NO Service in this repo "
f"matches, so its endpoint ports cannot be audited here. Either the selector is wrong "
f"(a monitor matching nothing discovers nothing), or the Service is chart/operator "
f"owned — in which case DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it")
for site in sorted(KNOWN_EXTERNAL_MONITORS):
check(any(f"{m[0]}:{m[3]}" == site for m in monitors),
f"KNOWN_EXTERNAL_MONITORS declares {site}, which is not a monitor in this repo any more "
f"— a stale exemption hides the next real gap; delete the entry")
# ============================================================ (4) THE RESOLVER IS PROVEN
# After the twelve apps/* monitors were removed there is NO in-repo monitor left for assertion (2)
# to judge (every survivor is chart-owned, and keycloak's scrapes by targetPort). A guard in that
# state passes whether it works or not. So the resolver is run against two synthetic monitors over
# two synthetic Services, in the exact shape catalog/render.py emits from
# `observability.metricsEndpoint`, and BOTH outcomes are asserted.
_real_services = services
services = [
("<selftest>", "fixture", "agreeing", {"app.kubernetes.io/name": "agreeing"},
{"port-8080", "metrics"}),
("<selftest>", "fixture", "disagreeing", {"app.kubernetes.io/name": "disagreeing"},
{"port-8080"}),
]
good = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
bad = ("<selftest>", "ServiceMonitor", "fixture", "disagreeing",
{"app.kubernetes.io/name": "disagreeing"}, {"matchNames": ["fixture"]},
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
empty = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
[{"path": "/metrics"}])
good_problems, good_matched = resolve(good)
bad_problems, bad_matched = resolve(bad)
empty_problems, _ = resolve(empty)
services = _real_services
check(good_matched and not good_problems,
f"SELF-TEST: the resolver rejected a CORRECT monitor (Service defines `metrics`, endpoint "
f"scrapes `metrics`): {good_problems or 'selector matched no fixture Service'}. The resolver "
f"is broken in the direction that makes the gate red for correct code")
check(bad_matched and len(bad_problems) == 1 and "discovers ZERO targets" in bad_problems[0],
"SELF-TEST: the resolver ACCEPTED a monitor whose endpoint names a port its Service does not "
"define — the exact defect that left twelve apps/* monitors discovering nothing on prod. "
"Assertion (2) is inert; every pass above proves nothing")
check(len(empty_problems) == 1 and "NEITHER" in empty_problems[0],
"SELF-TEST: the resolver ACCEPTED an endpoint naming neither `port` nor `targetPort`, so "
"assertion (1) is inert")
# ============================================================ (5) THE GENERATOR CANNOT REGRESS
RENDER = read(ROOT / "catalog" / "render.py")
check(RENDER != "", "catalog/render.py is unreadable — the generator half cannot be audited")
sm_fn = RENDER.split("def r_servicemonitor")[-1].split("\ndef ")[0] if RENDER else ""
check("def r_servicemonitor" in RENDER,
"catalog/render.py no longer defines r_servicemonitor — this assertion is auditing nothing")
check(not re.search(r'"port":\s*"[a-z][a-z0-9-]*"', sm_fn),
"catalog/render.py's r_servicemonitor HARDCODES a literal endpoint port name. That is the "
"original defect's mechanism: one edit gave twelve services the same wrong port name "
"(`metrics`), which no rendered Service defined, and every `up`-based SLI built on them was "
"structurally inert on prod. The port name must come from the ServiceDefinition's "
"`observability.metricsEndpoint`, the same declaration that renders the Service port")
check("metricsEndpoint" in sm_fn,
"catalog/render.py's r_servicemonitor no longer reads `metricsEndpoint`, so it is back to "
"rendering a monitor from an intent (`metrics: true`) that cannot say where metrics are "
"served — for most of this catalog, nowhere")
SCHEMA = read(ROOT / "schemas" / "servicedefinition.schema.json")
check("metricsEndpoint" in SCHEMA,
"schemas/servicedefinition.schema.json does not define observability.metricsEndpoint, so the "
"field the renderer depends on is not part of the contract and `additionalProperties: false` "
"would reject any ServiceDefinition that declares one")
# ============================================================ (6) LOAD-BEARING
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "servicemonitor-ports-resolve"
/ "chainsaw-test.yaml")
check(re.search(r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo", SELF, re.M) is not None,
"servicemonitor-ports-resolve/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch R-OBS-01. ci/touched-requirements.sh maps apps/* to NOTHING, so "
"adding a ServiceMonitor under apps/ would be filtered out of exactly the change that "
"reintroduces the defect")
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration")
check("servicemonitor-ports-resolve" not in L0,
"ci/l0.sh names this check in a hand-written stage. Whole-repo invariants are selected BY "
"DECLARING the scope; a per-test stage is the pattern tests/l0-static/"
"whole-repo-invariants-wired fails the gate for, because the next invariant's author has to "
"REMEMBER to add one and the omission is silent")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
print(f"Monitor inventory: {len(monitors)} ServiceMonitor/PodMonitor(s), {len(services)} "
f"Service(s) authored in this repo")
for rel, kind, ns, name, selector, ns_sel, eps in monitors:
tgt = ", ".join(
str(e.get("port") or f"targetPort {e.get('targetPort')}" or "<none>") for e in eps
) or "<no endpoints>"
mark = "EXTERNAL" if f"{rel}:{name}" in KNOWN_EXTERNAL_MONITORS else "resolved"
print(f" {mark:<8} {kind:<14} {rel}:{name} -> {tgt}")
print(f"\n resolvable against an in-repo Service: {auditable}/{len(monitors)}")
print(f" DECLARED GAP (shrink-only): KNOWN_EXTERNAL_MONITORS: {len(KNOWN_EXTERNAL_MONITORS)} "
f"monitor(s) whose Service is created by a chart/operator outside this tree")
print(" RESOLVER SELF-TEST: agreeing pair accepted, disagreeing pair rejected, "
"target-less endpoint rejected")
print()
if not FAILURES:
print(f"check OK: servicemonitor-ports-resolve — {CHECKS} assertions (R-OBS-01)")
raise SystemExit(0)
print(f"CHECK FAIL (servicemonitor-ports-resolve): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY