jakman85andClaude Opus 5
a92fa11701
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
120 lines
8.4 KiB
YAML
120 lines
8.4 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
|
|
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
|
|
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
|
|
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
|
|
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
|
|
# survives every dashboard (nobody looks for a panel that was never built).
|
|
#
|
|
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
|
|
# count(longhorn_volume_robustness) => (empty) 4 rules
|
|
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
|
|
# count(hubble_drop_total) => (empty) 1 rule
|
|
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
|
|
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
|
|
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
|
|
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
|
|
# was the same defect and was fixed separately in a79693d.
|
|
#
|
|
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
|
|
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
|
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
|
|
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
|
|
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
|
|
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
|
|
# storage alerts;
|
|
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
|
|
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
|
|
# both set, Longhorn's own NetworkPolicies drop the scrape unless
|
|
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
|
|
# kube-state-metrics' metricLabelsAllowlist names the label;
|
|
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
|
|
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
|
|
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
|
# check above can never see this failure: where the repo authors a Service and a
|
|
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
|
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
|
|
# port `metrics`; no app Service defines one), so they are registered in
|
|
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
|
|
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
|
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
|
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
|
#
|
|
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
|
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
|
|
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
|
|
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
|
|
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
|
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
|
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
|
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
|
# into fiction.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
|
|
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
|
|
# run it directly, and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
|
|
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
|
|
# TARGET exists, never that the exporter emits that exact SERIES NAME.
|
|
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
|
|
# and did not exist at all on the deployed KubeVirt (corrected here to
|
|
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
|
|
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
|
|
# that diffs alert metrics against /api/v1/label/__name__/values.
|
|
#
|
|
# Also not proven: that a target is UP. This is static — it asserts the
|
|
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
|
|
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
|
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
|
|
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
|
|
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
|
|
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
|
|
# (R-OBS-02 / the L3 alerting test).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: alert-metrics-have-scrape-targets
|
|
labels:
|
|
conformance.home.k3s/R-OBS-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-OBS-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
|
|
# every scrape-enabling declaration that feeds them, which live in eight different components
|
|
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
|
|
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
|
|
# path-maps to that set.
|
|
#
|
|
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
|
|
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
|
|
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
|
|
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
|
|
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
|
|
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
|
|
# UNWIRED, never green) rather than silently green.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
|
|
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
|
|
monitor can actually produce, and the two gaps that remain are enumerated in registers that
|
|
can only shrink — so an alert can never again be authored, reviewed and merged against a
|
|
metric nothing scrapes.
|
|
spec:
|
|
steps:
|
|
- name: audit-every-alert-metric-against-the-configured-scrape-targets
|
|
description: >-
|
|
Parse every metric name out of every alerting rule, resolve each against the declared
|
|
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
|
|
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
|
|
agreement, and ratchet both registers of declared gaps.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|