fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
72f0ba4e1f
commit
cf7d97e546
29 files changed
+1632
-428
No files matched your search
@@ -36,18 +36,27 @@
|
||||
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
||||
# check above can never see this failure: where the repo authors a Service and a
|
||||
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
|
||||
# port `metrics`; no app Service defines one), so they are registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
|
||||
# named port `metrics`; no app Service defined one) and were registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
|
||||
# THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
|
||||
# kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
|
||||
# ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
|
||||
# moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
|
||||
# alert rule to reach a monitor, resolves by selector across the whole tree instead of by
|
||||
# same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
|
||||
# lack of subjects;
|
||||
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
||||
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
||||
#
|
||||
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
|
||||
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
|
||||
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
|
||||
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
|
||||
# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
|
||||
# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
|
||||
# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
|
||||
# signal that already existed. The register is empty now, and that is the outcome it was for.)
|
||||
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
||||
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
||||
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
||||
|
||||
@@ -333,38 +333,36 @@ KNOWN_UNWIRED_METRICS: dict[str, str] = {
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
|
||||
# names an endpoint port that the Service it selects does not define, so the monitor discovers
|
||||
# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
|
||||
# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
|
||||
# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
|
||||
# has no up series to be 0.
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
|
||||
#
|
||||
# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
|
||||
# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
|
||||
# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
|
||||
# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
|
||||
# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
|
||||
# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
|
||||
# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
|
||||
# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
|
||||
# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
|
||||
# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
|
||||
# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
|
||||
# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
|
||||
# it either — a target that was never discovered has no up series to be 0.
|
||||
#
|
||||
# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
|
||||
# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
|
||||
# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
|
||||
# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
|
||||
# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
|
||||
# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
|
||||
# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
|
||||
# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
|
||||
# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
|
||||
# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
|
||||
# twelfth (vm-demo) is not deployed at all.
|
||||
#
|
||||
# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
|
||||
# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
|
||||
# apps/ — which is why the monitor-side axis moved into its own check,
|
||||
# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
|
||||
# Service defines fails, with no alert rule required, resolving by selector across the whole tree
|
||||
# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
|
||||
# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
|
||||
# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {
|
||||
"gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
|
||||
"jackett": "apps/wave-0/jackett.yaml — no exporter",
|
||||
"pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
|
||||
"plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
|
||||
"qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
|
||||
"vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
|
||||
"keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
|
||||
"keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
|
||||
"neither, so it is a rename away from being live rather than an exporter away",
|
||||
"nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
|
||||
"radarr": "apps/wave-1/radarr.yaml — needs exportarr",
|
||||
"semaphore": "apps/wave-1/semaphore.yaml — no exporter",
|
||||
"sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
|
||||
"seerr": "apps/wave-2/seerr.yaml — no exporter",
|
||||
}
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) PARSE EVERY ALERT RULE
|
||||
|
||||
@@ -147,25 +147,20 @@ def check(cond, msg):
|
||||
# =============================================================================
|
||||
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
|
||||
#
|
||||
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
|
||||
# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
|
||||
# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
|
||||
# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
|
||||
# FAILS as stale.
|
||||
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
|
||||
# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
|
||||
#
|
||||
# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
|
||||
# generated from one ServiceDefinition template, and the register's description of them as one
|
||||
# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
|
||||
# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
|
||||
# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
|
||||
# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
|
||||
# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
|
||||
# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
|
||||
# now the only entry left and is where the next tests belong.
|
||||
# =============================================================================
|
||||
UNTESTED_FILES = {
|
||||
"apps/wave-0/gitea.yaml",
|
||||
"apps/wave-0/jackett.yaml",
|
||||
"apps/wave-0/pihole.yaml",
|
||||
"apps/wave-0/plex.yaml",
|
||||
"apps/wave-0/qbittorrent.yaml",
|
||||
"apps/wave-0/vm-demo.yaml",
|
||||
"apps/wave-1/keycloak.yaml",
|
||||
"apps/wave-1/nextcloud.yaml",
|
||||
"apps/wave-1/radarr.yaml",
|
||||
"apps/wave-1/semaphore.yaml",
|
||||
"apps/wave-1/sonarr.yaml",
|
||||
"apps/wave-2/seerr.yaml",
|
||||
"platform/kubevirt/alerts.yaml",
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every ServiceMonitor endpoint names a scrape target that RESOLVES
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES, AND WHY ITS SIBLING COULD NOT.
|
||||
# alert-metrics-have-scrape-targets audits scrape wiring FROM THE ALERT SIDE: it walks alert rules
|
||||
# and asks whether something scrapes the metric each one reads. Its assertion (4) is the `up{job=X}`
|
||||
# axis, and it is reachable ONLY through an alert that selects that job, only where the Service and
|
||||
# the ServiceMonitor sit in the SAME FILE. Its own header names the hole. This test is the other
|
||||
# direction: it walks the MONITORS, and asks of each one whether the target it names can exist —
|
||||
# with no alert rule required anywhere.
|
||||
#
|
||||
# That difference is not academic. On 2026-09-13 all twelve apps/* ServiceMonitors named
|
||||
# `endpoints[].port: metrics` and NO app Service defined a port of that name, so each discovered
|
||||
# ZERO targets (verified on prod: `count by (job) (up)` listed 24 jobs, none of the twelve; both
|
||||
# `up{job="plex"}` and `up{job="gitea"}` returned nothing). That was caught only because each app
|
||||
# also shipped an `up{job=...}` alert. THE FIX TO THOSE ALERTS DELETED THE ONLY THING THAT MADE THE
|
||||
# BROKEN MONITORS VISIBLE — they no longer read `up{job=...}` at all — so a monitor that discovers
|
||||
# nothing would from that moment have been invisible again, and the twelve register entries that
|
||||
# recorded the gap correctly became stale and were removed. A monitor scraping nothing is a defect
|
||||
# on its own terms: it is dead configuration that reads as live coverage, and every dashboard,
|
||||
# recording rule or future alert built on its series inherits the emptiness.
|
||||
#
|
||||
# GIVEN every ServiceMonitor and PodMonitor this repo authors,
|
||||
# WHEN each endpoint's scrape target is resolved against every Service this repo authors,
|
||||
# THEN (1) every endpoint names a target AT ALL (`port` or `targetPort`) — an endpoint naming
|
||||
# neither scrapes nothing and is accepted silently by the operator;
|
||||
# (2) every endpoint that names a `port` names a port that at least one Service its selector
|
||||
# matches ACTUALLY DEFINES — the exact disagreement that made twelve monitors inert.
|
||||
# Resolution is by SELECTOR ACROSS THE WHOLE TREE, not by same-file adjacency, so moving
|
||||
# a Service into its own file cannot defeat it;
|
||||
# (3) a monitor whose selector matches no Service in this repo is UNAUDITABLE (its Service is
|
||||
# created by a Helm chart or an operator, so the port name lives outside the tree) and
|
||||
# must be DECLARED in KNOWN_EXTERNAL_MONITORS, naming who creates the Service. That
|
||||
# register is shrink-only: a monitor that later gains an in-repo Service, or that is
|
||||
# deleted, FAILS as stale, so the unauditable set cannot quietly grow;
|
||||
# (4) THE RESOLVER ITSELF IS PROVEN, on synthetic fixtures inside the check — one agreeing
|
||||
# pair it must accept, one disagreeing pair it must reject. This is not decoration: after
|
||||
# the fix there are zero in-repo monitors left for (2) to judge, and a guard whose only
|
||||
# evidence is "nothing failed" is indistinguishable from a guard that stopped working.
|
||||
# The fixtures are the exact shape catalog/render.py emits from
|
||||
# `observability.metricsEndpoint`;
|
||||
# (5) the GENERATOR cannot reintroduce the class: catalog/render.py must not hardcode an
|
||||
# endpoint port name — it must take it from the ServiceDefinition, the same declaration
|
||||
# that renders the Service port. A hardcoded literal is how twelve services acquired an
|
||||
# identical wrong port name in one edit;
|
||||
# AND (6) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps apps/* to NOTHING, so a change that
|
||||
# adds a ServiceMonitor under apps/ attributes no requirement and would filter this check out of
|
||||
# exactly the change that breaks it. Same reasoning as its sibling.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling check use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static):
|
||||
# - That a resolved target is UP, answers, or emits the series an alert wants. The sibling check
|
||||
# owns the metric-name axis and admits it cannot prove the SERIES NAME either; both are live
|
||||
# reads and belong to a future L1/L3 check against /api/v1/label/__name__/values.
|
||||
# - That a NetworkPolicy permits the scrape. The default-deny baseline drops it unless an allow is
|
||||
# rendered, which is now part of what `observability.metricsEndpoint` emits
|
||||
# (r_monitoring_ingress_policy) — but this check does not verify the allow, only the ports.
|
||||
# - For a monitor in KNOWN_EXTERNAL_MONITORS, nothing about the chart's port names. The register
|
||||
# records WHO owns them so the claim is reviewable, which is all a static check can offer.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: servicemonitor-ports-resolve
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceMonitor/PodMonitor and every
|
||||
# Service in the tree (apps/wave-*, apps/vms, platform/keycloak, platform/kubevirt,
|
||||
# platform/oauth2-proxy, platform/monitoring, platform/crafty, platform/homepage,
|
||||
# runbooks/serving). No single requirement path-maps to that set, and ci/touched-requirements.sh
|
||||
# maps apps/* to no requirement at all — so a requirement-scoped selection would skip this
|
||||
# check on precisely the changes that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ServiceMonitor/PodMonitor endpoint this repo authors names a scrape target that
|
||||
resolves — a port some matching Service actually defines — the monitors whose Services come
|
||||
from outside the tree are enumerated in a shrink-only register, and the resolver that decides
|
||||
this is itself proven on fixtures, so a monitor can never again be authored, reviewed and
|
||||
merged against a port nothing serves.
|
||||
spec:
|
||||
steps:
|
||||
- name: resolve-every-monitor-endpoint-against-every-service-this-repo-authors
|
||||
description: >-
|
||||
Inventory every ServiceMonitor/PodMonitor and Service in the tree, resolve each endpoint's
|
||||
port against the Services its selector matches, ratchet the register of monitors whose
|
||||
Service is chart-owned, prove the resolver on synthetic agreeing/disagreeing fixtures, and
|
||||
assert the renderer does not hardcode an endpoint port name.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,416 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — every ServiceMonitor/PodMonitor endpoint names a scrape target that RESOLVES.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# alert-metrics-have-scrape-targets audits scrape wiring from the ALERT side, and its `up{job=X}`
|
||||
# axis is reachable only through an alert that selects that job, only for a Service and a
|
||||
# ServiceMonitor in the SAME FILE. That is how it caught the twelve inert apps/* monitors on
|
||||
# 2026-09-13 (`endpoints[].port: metrics`, and no app Service defined such a port; on prod
|
||||
# `count by (job) (up)` listed 24 jobs and none of the twelve appeared). But FIXING those alerts
|
||||
# removed their `up{job=...}` selectors, which removed the only path by which the broken monitors
|
||||
# were visible at all. This check walks the MONITORS instead, needs no alert rule, and resolves by
|
||||
# SELECTOR across the whole tree — so a monitor that discovers nothing fails on its own terms.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every endpoint names a scrape target at all — `port` or `targetPort`;
|
||||
# (2) every endpoint naming a `port` names a port at least one Service its selector matches
|
||||
# actually DEFINES;
|
||||
# (3) a monitor whose selector matches no in-repo Service is unauditable (chart/operator-owned
|
||||
# Service) and must be declared in KNOWN_EXTERNAL_MONITORS — shrink-only, stale entries FAIL;
|
||||
# (4) the RESOLVER is proven on synthetic fixtures (one agreeing pair, one disagreeing pair),
|
||||
# because after the fix there are zero in-repo monitors for (2) to judge and a guard whose
|
||||
# only evidence is "nothing failed" cannot be told apart from one that stopped working;
|
||||
# (5) catalog/render.py does not HARDCODE an endpoint port name — it derives it from the
|
||||
# ServiceDefinition, the same declaration that renders the Service port. A hardcoded literal
|
||||
# is how twelve services acquired one identical wrong port name in a single edit;
|
||||
# (6) the check is LOAD-BEARING: whole-repo scope declared here, honoured by ci/l0.sh's
|
||||
# --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written ci/l0.sh
|
||||
# stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC):
|
||||
# - Nothing about liveness: that a resolved target is UP, answers, or emits the series a rule
|
||||
# wants. The sibling check owns the metric-name axis and admits it cannot prove the series NAME
|
||||
# either. Both need a live read of /api/v1/label/__name__/values (a future L1/L3 check).
|
||||
# - Nothing about NetworkPolicy. An app namespace's default-deny baseline is `ingress: []`, so a
|
||||
# scrape is dropped unless an allow is rendered; that allow is now part of what
|
||||
# `observability.metricsEndpoint` emits (catalog/render.py r_monitoring_ingress_policy), but this
|
||||
# check verifies ports, not policy.
|
||||
# - Nothing about a chart's port names, for a monitor in KNOWN_EXTERNAL_MONITORS. The register
|
||||
# records WHO owns them so the claim is reviewable; that is all a static check can offer.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns into
|
||||
# rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the repo
|
||||
# contains no ServiceMonitor or PodMonitor at all — genuinely nothing to audit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
|
||||
if ! grep -rlE '^[[:space:]]*kind:[[:space:]]*(Service|Pod)Monitor[[:space:]]*$' \
|
||||
"${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
|
||||
echo "check SKIPPED: no ServiceMonitor/PodMonitor found under platform/ or apps/ — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: every ServiceMonitor/PodMonitor endpoint's scrape target resolves.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling check) rather than line-based
|
||||
parsing: a monitor's selector, namespaceSelector and endpoint list are nested structures, and a
|
||||
line-based reader of them is precisely the approximate parser that silently stops matching — the
|
||||
failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_EXTERNAL_MONITORS. Shrink-only.
|
||||
#
|
||||
# A monitor whose selector matches NO Service in this repo cannot be audited here: the Service is
|
||||
# created by a Helm chart or an operator, so the port name lives outside the tree. That is a real
|
||||
# limit, not a pass — so each one is DECLARED, with WHO creates the Service it selects, and the
|
||||
# register can only shrink. If such a monitor later gains an in-repo Service its entry goes stale
|
||||
# and FAILS; if it is deleted, likewise. The same ratchet reasoning as the two registers in
|
||||
# alert-metrics-have-scrape-targets: failing outright would make the gate red for chart-owned
|
||||
# monitors that are perfectly correct, and a gate that is red for correct code gets disabled.
|
||||
#
|
||||
# Keyed by "<relpath>:<monitor name>".
|
||||
# =============================================================================================
|
||||
KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
|
||||
"platform/keycloak/servicemonitor.yaml:keycloak":
|
||||
"selects the operator-created `keycloak-service` (labels app=keycloak) — the Keycloak "
|
||||
"operator owns that Service. Scrapes by targetPort 9000 (the Quarkus management "
|
||||
"interface), so it names no Service port at all and assertion (2) has nothing to resolve.",
|
||||
"platform/kubevirt/servicemonitor.yaml:kubevirt":
|
||||
"selects the KubeVirt operator's own `kubevirt-prometheus-metrics` Service (label "
|
||||
"prometheus.kubevirt.io=true), created by the KubeVirt CR, not by this repo. Its `metrics` "
|
||||
"port is defined there. Live on prod: job kubevirt-prometheus-metrics, 11 targets up.",
|
||||
"platform/oauth2-proxy/servicemonitor.yaml:oauth2-proxy":
|
||||
"selects the oauth2-proxy Helm chart's Service (label app=oauth2-proxy); the chart defines "
|
||||
"its `metrics` port. Live on prod: job oauth2-proxy, 2 targets up.",
|
||||
"platform/monitoring/flux-podmonitor.yaml:flux-system":
|
||||
"a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
|
||||
"Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
|
||||
"manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
|
||||
}
|
||||
|
||||
MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
|
||||
|
||||
|
||||
def is_test_fixture(p: Path) -> bool:
|
||||
"""conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
|
||||
|
||||
The l1-rendering goldens are expected-output copies of rendered apps; the l2/l3 fixtures are
|
||||
throwaway objects a behavioural test applies itself. Auditing either would be auditing a copy —
|
||||
the same "reference, not declaration" distinction ci/touched-requirements.sh draws for fixture
|
||||
paths, and the same exclusion the sibling check makes for goldens. The templates they come from
|
||||
ARE audited, through apps/ and catalog/.
|
||||
"""
|
||||
parts = p.parts
|
||||
return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
|
||||
|
||||
|
||||
# ============================================================ (0) INVENTORY
|
||||
# services: list of (relpath, namespace, name, labels, {named ports})
|
||||
# monitors: list of (relpath, kind, namespace, name, selector, namespaceSelector, [endpoints])
|
||||
services: list[tuple] = []
|
||||
monitors: list[tuple] = []
|
||||
unparseable: list[str] = []
|
||||
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
|
||||
continue
|
||||
txt = read(p)
|
||||
if not txt:
|
||||
continue
|
||||
if not re.search(r"^\s*kind:\s*(Service|ServiceMonitor|PodMonitor)\s*$", txt, re.M):
|
||||
continue
|
||||
rel = str(p.relative_to(ROOT))
|
||||
try:
|
||||
docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
|
||||
except yaml.YAMLError as exc:
|
||||
unparseable.append(f"{rel} ({exc})")
|
||||
continue
|
||||
for d in docs:
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
spec = d.get("spec") or {}
|
||||
if kind == "Service":
|
||||
names = {str(port["name"]) for port in (spec.get("ports") or [])
|
||||
if isinstance(port, dict) and port.get("name")}
|
||||
services.append((rel, md.get("namespace"), md.get("name"),
|
||||
dict(md.get("labels") or {}), names))
|
||||
elif kind in MONITOR_KINDS:
|
||||
eps = spec.get("endpoints") or spec.get("podMetricsEndpoints") or []
|
||||
monitors.append((rel, kind, md.get("namespace"), md.get("name"),
|
||||
dict((spec.get("selector") or {}).get("matchLabels") or {}),
|
||||
dict(spec.get("namespaceSelector") or {}),
|
||||
[e for e in eps if isinstance(e, dict)]))
|
||||
|
||||
for u in unparseable:
|
||||
check(False, f"{u} is not parseable YAML — every monitor and Service in it went unaudited")
|
||||
|
||||
check(len(monitors) > 0,
|
||||
"no ServiceMonitor/PodMonitor parsed out of the tree, yet the grep gate above found one. The "
|
||||
"parse contract broke and this check would pass vacuously while every scrape target went "
|
||||
"unaudited")
|
||||
check(len(services) >= 10,
|
||||
f"only {len(services)} Service(s) parsed from the tree. This repo authors more than that "
|
||||
f"(every apps/wave-* ServiceDefinition renders one), so the inventory has stopped being "
|
||||
f"populated — assertion (2) would then resolve nothing and pass for the wrong reason")
|
||||
|
||||
|
||||
# ============================================================ THE RESOLVER
|
||||
def monitor_namespaces(mon_ns, ns_selector) -> set | None:
|
||||
"""Namespaces a monitor selects. None means ANY (`any: true`).
|
||||
|
||||
An empty namespaceSelector defaults to the monitor's own namespace — the operator's documented
|
||||
behaviour, and the case every repo-authored monitor uses.
|
||||
"""
|
||||
if ns_selector.get("any") is True:
|
||||
return None
|
||||
names = ns_selector.get("matchNames")
|
||||
if names:
|
||||
return {str(n) for n in names}
|
||||
return {mon_ns} if mon_ns else set()
|
||||
|
||||
|
||||
def matching_services(selector: dict, namespaces: set | None) -> list[tuple]:
|
||||
"""Services whose labels are a SUPERSET of the selector and whose namespace is in range.
|
||||
|
||||
An EMPTY selector matches every Service in range (Kubernetes label-selector semantics). A
|
||||
PodMonitor selector is compared against Service labels too — deliberately, so the resolver has
|
||||
one code path; when that yields no match the monitor lands in the unauditable register with the
|
||||
reason recorded, which is the honest outcome for a pod-side target this repo does not author.
|
||||
"""
|
||||
out = []
|
||||
for svc in services:
|
||||
_, svc_ns, _, labels, _ = svc
|
||||
if namespaces is not None and svc_ns not in namespaces:
|
||||
continue
|
||||
if all(labels.get(k) == v for k, v in selector.items()):
|
||||
out.append(svc)
|
||||
return out
|
||||
|
||||
|
||||
def resolve(mon: tuple) -> tuple[list[str], list[tuple]]:
|
||||
"""(failures, matched services) for one monitor. The unit assertions (1) and (2) live here so
|
||||
the SELF-TEST at the bottom exercises the same code path the repo audit does."""
|
||||
rel, kind, ns, name, selector, ns_sel, eps = mon
|
||||
site = f"{rel}:{name}"
|
||||
problems: list[str] = []
|
||||
matched = matching_services(selector, monitor_namespaces(ns, ns_sel))
|
||||
|
||||
for i, ep in enumerate(eps):
|
||||
if not ep.get("port") and not ep.get("targetPort"):
|
||||
problems.append(
|
||||
f"{site} endpoint[{i}] names NEITHER `port` NOR `targetPort`. The operator accepts "
|
||||
f"that silently and the monitor scrapes nothing")
|
||||
if not eps:
|
||||
problems.append(
|
||||
f"{site} has NO endpoints at all — it can never produce a target, and nothing about "
|
||||
f"the object looks wrong")
|
||||
|
||||
named = [(i, str(ep["port"])) for i, ep in enumerate(eps) if ep.get("port")]
|
||||
if named and matched:
|
||||
defined: set = set()
|
||||
for svc in matched:
|
||||
defined |= svc[4]
|
||||
for i, port in named:
|
||||
if port not in defined:
|
||||
where = ", ".join(f"{s[0]}:{s[2]}" for s in matched)
|
||||
problems.append(
|
||||
f"{site} endpoint[{i}] scrapes port {port!r}, which NO Service its selector "
|
||||
f"matches defines. Matched Service(s): {where}; the port names they define are "
|
||||
f"{sorted(defined) or ['(none)']}. The monitor therefore discovers ZERO "
|
||||
f"targets: no `up{{job=\"{name}\"}}` series exists, anything built on its "
|
||||
f"series is empty, and PrometheusTargetsMissing cannot see it either because a "
|
||||
f"target that was never discovered has no up series to be 0. Give the Service a "
|
||||
f"named port the workload actually serves metrics on (for a ServiceDefinition: "
|
||||
f"`observability.metricsEndpoint`, which renders the Service port, this "
|
||||
f"endpoint and the monitoring ingress allow together), or scrape by targetPort")
|
||||
return problems, matched
|
||||
|
||||
|
||||
# ============================================================ (1)(2)(3) THE REPO AUDIT
|
||||
auditable = 0
|
||||
for mon in monitors:
|
||||
rel, kind, ns, name, selector, ns_sel, eps = mon
|
||||
site = f"{rel}:{name}"
|
||||
problems, matched = resolve(mon)
|
||||
for msg in problems:
|
||||
check(False, msg)
|
||||
if not problems:
|
||||
# An endpoint-shape assertion passed for this monitor; count it so the report is honest
|
||||
# about how much of the run was real work.
|
||||
check(True, f"{site}: every endpoint names a scrape target")
|
||||
if matched:
|
||||
auditable += 1
|
||||
check(site not in KNOWN_EXTERNAL_MONITORS,
|
||||
f"{site} is listed in KNOWN_EXTERNAL_MONITORS but its selector NOW matches "
|
||||
f"{len(matched)} Service(s) this repo authors — the register is shrink-only: delete "
|
||||
f"the entry so the monitor is audited for real")
|
||||
else:
|
||||
check(site in KNOWN_EXTERNAL_MONITORS,
|
||||
f"{site} selects {selector or '(everything)'} in namespace(s) "
|
||||
f"{sorted(monitor_namespaces(ns, ns_sel) or ['<any>'])} and NO Service in this repo "
|
||||
f"matches, so its endpoint ports cannot be audited here. Either the selector is wrong "
|
||||
f"(a monitor matching nothing discovers nothing), or the Service is chart/operator "
|
||||
f"owned — in which case DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it")
|
||||
|
||||
for site in sorted(KNOWN_EXTERNAL_MONITORS):
|
||||
check(any(f"{m[0]}:{m[3]}" == site for m in monitors),
|
||||
f"KNOWN_EXTERNAL_MONITORS declares {site}, which is not a monitor in this repo any more "
|
||||
f"— a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
|
||||
# ============================================================ (4) THE RESOLVER IS PROVEN
|
||||
# After the twelve apps/* monitors were removed there is NO in-repo monitor left for assertion (2)
|
||||
# to judge (every survivor is chart-owned, and keycloak's scrapes by targetPort). A guard in that
|
||||
# state passes whether it works or not. So the resolver is run against two synthetic monitors over
|
||||
# two synthetic Services, in the exact shape catalog/render.py emits from
|
||||
# `observability.metricsEndpoint`, and BOTH outcomes are asserted.
|
||||
_real_services = services
|
||||
services = [
|
||||
("<selftest>", "fixture", "agreeing", {"app.kubernetes.io/name": "agreeing"},
|
||||
{"port-8080", "metrics"}),
|
||||
("<selftest>", "fixture", "disagreeing", {"app.kubernetes.io/name": "disagreeing"},
|
||||
{"port-8080"}),
|
||||
]
|
||||
good = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
|
||||
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
|
||||
bad = ("<selftest>", "ServiceMonitor", "fixture", "disagreeing",
|
||||
{"app.kubernetes.io/name": "disagreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
|
||||
empty = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
|
||||
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"path": "/metrics"}])
|
||||
|
||||
good_problems, good_matched = resolve(good)
|
||||
bad_problems, bad_matched = resolve(bad)
|
||||
empty_problems, _ = resolve(empty)
|
||||
services = _real_services
|
||||
|
||||
check(good_matched and not good_problems,
|
||||
f"SELF-TEST: the resolver rejected a CORRECT monitor (Service defines `metrics`, endpoint "
|
||||
f"scrapes `metrics`): {good_problems or 'selector matched no fixture Service'}. The resolver "
|
||||
f"is broken in the direction that makes the gate red for correct code")
|
||||
check(bad_matched and len(bad_problems) == 1 and "discovers ZERO targets" in bad_problems[0],
|
||||
"SELF-TEST: the resolver ACCEPTED a monitor whose endpoint names a port its Service does not "
|
||||
"define — the exact defect that left twelve apps/* monitors discovering nothing on prod. "
|
||||
"Assertion (2) is inert; every pass above proves nothing")
|
||||
check(len(empty_problems) == 1 and "NEITHER" in empty_problems[0],
|
||||
"SELF-TEST: the resolver ACCEPTED an endpoint naming neither `port` nor `targetPort`, so "
|
||||
"assertion (1) is inert")
|
||||
|
||||
|
||||
# ============================================================ (5) THE GENERATOR CANNOT REGRESS
|
||||
RENDER = read(ROOT / "catalog" / "render.py")
|
||||
check(RENDER != "", "catalog/render.py is unreadable — the generator half cannot be audited")
|
||||
sm_fn = RENDER.split("def r_servicemonitor")[-1].split("\ndef ")[0] if RENDER else ""
|
||||
check("def r_servicemonitor" in RENDER,
|
||||
"catalog/render.py no longer defines r_servicemonitor — this assertion is auditing nothing")
|
||||
check(not re.search(r'"port":\s*"[a-z][a-z0-9-]*"', sm_fn),
|
||||
"catalog/render.py's r_servicemonitor HARDCODES a literal endpoint port name. That is the "
|
||||
"original defect's mechanism: one edit gave twelve services the same wrong port name "
|
||||
"(`metrics`), which no rendered Service defined, and every `up`-based SLI built on them was "
|
||||
"structurally inert on prod. The port name must come from the ServiceDefinition's "
|
||||
"`observability.metricsEndpoint`, the same declaration that renders the Service port")
|
||||
check("metricsEndpoint" in sm_fn,
|
||||
"catalog/render.py's r_servicemonitor no longer reads `metricsEndpoint`, so it is back to "
|
||||
"rendering a monitor from an intent (`metrics: true`) that cannot say where metrics are "
|
||||
"served — for most of this catalog, nowhere")
|
||||
SCHEMA = read(ROOT / "schemas" / "servicedefinition.schema.json")
|
||||
check("metricsEndpoint" in SCHEMA,
|
||||
"schemas/servicedefinition.schema.json does not define observability.metricsEndpoint, so the "
|
||||
"field the renderer depends on is not part of the contract and `additionalProperties: false` "
|
||||
"would reject any ServiceDefinition that declares one")
|
||||
|
||||
|
||||
# ============================================================ (6) LOAD-BEARING
|
||||
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "servicemonitor-ports-resolve"
|
||||
/ "chainsaw-test.yaml")
|
||||
check(re.search(r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo", SELF, re.M) is not None,
|
||||
"servicemonitor-ports-resolve/chainsaw-test.yaml does not declare "
|
||||
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
|
||||
"change happens to touch R-OBS-01. ci/touched-requirements.sh maps apps/* to NOTHING, so "
|
||||
"adding a ServiceMonitor under apps/ would be filtered out of exactly the change that "
|
||||
"reintroduces the defect")
|
||||
L0 = read(ROOT / "ci" / "l0.sh")
|
||||
GATE = read(ROOT / "ci" / "gate.sh")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
|
||||
"purely requirement-scoped run again and the scope annotation above is inert decoration")
|
||||
check("servicemonitor-ports-resolve" not in L0,
|
||||
"ci/l0.sh names this check in a hand-written stage. Whole-repo invariants are selected BY "
|
||||
"DECLARING the scope; a per-test stage is the pattern tests/l0-static/"
|
||||
"whole-repo-invariants-wired fails the gate for, because the next invariant's author has to "
|
||||
"REMEMBER to add one and the omission is silent")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
|
||||
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
# ======================================================================== report
|
||||
print(f"Monitor inventory: {len(monitors)} ServiceMonitor/PodMonitor(s), {len(services)} "
|
||||
f"Service(s) authored in this repo")
|
||||
for rel, kind, ns, name, selector, ns_sel, eps in monitors:
|
||||
tgt = ", ".join(
|
||||
str(e.get("port") or f"targetPort {e.get('targetPort')}" or "<none>") for e in eps
|
||||
) or "<no endpoints>"
|
||||
mark = "EXTERNAL" if f"{rel}:{name}" in KNOWN_EXTERNAL_MONITORS else "resolved"
|
||||
print(f" {mark:<8} {kind:<14} {rel}:{name} -> {tgt}")
|
||||
print(f"\n resolvable against an in-repo Service: {auditable}/{len(monitors)}")
|
||||
print(f" DECLARED GAP (shrink-only): KNOWN_EXTERNAL_MONITORS: {len(KNOWN_EXTERNAL_MONITORS)} "
|
||||
f"monitor(s) whose Service is created by a chart/operator outside this tree")
|
||||
print(" RESOLVER SELF-TEST: agreeing pair accepted, disagreeing pair rejected, "
|
||||
"target-less endpoint rejected")
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: servicemonitor-ports-resolve — {CHECKS} assertions (R-OBS-01)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (servicemonitor-ports-resolve): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -212,26 +212,6 @@ spec:
|
||||
- name: paas-demo
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: paas-demo
|
||||
namespace: paas-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: paas-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- paas-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
kind: ImageRepository
|
||||
|
||||
@@ -72,26 +72,6 @@ spec:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: baseline-demo
|
||||
namespace: baseline-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: baseline-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: baseline-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- baseline-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
kind: ImageRepository
|
||||
|
||||
+2
-22
@@ -239,26 +239,6 @@ spec:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -273,14 +253,14 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: up{job="sonarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
|
||||
@@ -239,26 +239,6 @@ spec:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -273,14 +253,14 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: up{job="sonarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
|
||||
Reference in new issue
Block a user