jakman85andClaude Opus 5
cf7d97e546
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
129 lines
9.3 KiB
YAML
129 lines
9.3 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
|
|
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
|
|
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
|
|
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
|
|
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
|
|
# survives every dashboard (nobody looks for a panel that was never built).
|
|
#
|
|
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
|
|
# count(longhorn_volume_robustness) => (empty) 4 rules
|
|
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
|
|
# count(hubble_drop_total) => (empty) 1 rule
|
|
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
|
|
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
|
|
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
|
|
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
|
|
# was the same defect and was fixed separately in a79693d.
|
|
#
|
|
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
|
|
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
|
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
|
|
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
|
|
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
|
|
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
|
|
# storage alerts;
|
|
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
|
|
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
|
|
# both set, Longhorn's own NetworkPolicies drop the scrape unless
|
|
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
|
|
# kube-state-metrics' metricLabelsAllowlist names the label;
|
|
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
|
|
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
|
|
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
|
# check above can never see this failure: where the repo authors a Service and a
|
|
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
|
# actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
|
|
# named port `metrics`; no app Service defined one) and were registered in
|
|
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
|
|
# THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
|
|
# kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
|
|
# ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
|
|
# moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
|
|
# alert rule to reach a monitor, resolves by selector across the whole tree instead of by
|
|
# same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
|
|
# lack of subjects;
|
|
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
|
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
|
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
|
#
|
|
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
|
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
|
|
# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
|
|
# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
|
|
# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
|
# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
|
|
# signal that already existed. The register is empty now, and that is the outcome it was for.)
|
|
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
|
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
|
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
|
# into fiction.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
|
|
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
|
|
# run it directly, and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
|
|
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
|
|
# TARGET exists, never that the exporter emits that exact SERIES NAME.
|
|
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
|
|
# and did not exist at all on the deployed KubeVirt (corrected here to
|
|
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
|
|
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
|
|
# that diffs alert metrics against /api/v1/label/__name__/values.
|
|
#
|
|
# Also not proven: that a target is UP. This is static — it asserts the
|
|
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
|
|
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
|
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
|
|
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
|
|
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
|
|
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
|
|
# (R-OBS-02 / the L3 alerting test).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: alert-metrics-have-scrape-targets
|
|
labels:
|
|
conformance.home.k3s/R-OBS-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-OBS-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
|
|
# every scrape-enabling declaration that feeds them, which live in eight different components
|
|
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
|
|
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
|
|
# path-maps to that set.
|
|
#
|
|
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
|
|
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
|
|
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
|
|
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
|
|
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
|
|
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
|
|
# UNWIRED, never green) rather than silently green.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
|
|
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
|
|
monitor can actually produce, and the two gaps that remain are enumerated in registers that
|
|
can only shrink — so an alert can never again be authored, reviewed and merged against a
|
|
metric nothing scrapes.
|
|
spec:
|
|
steps:
|
|
- name: audit-every-alert-metric-against-the-configured-scrape-targets
|
|
description: >-
|
|
Parse every metric name out of every alerting rule, resolve each against the declared
|
|
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
|
|
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
|
|
agreement, and ratchet both registers of declared gaps.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|