merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert
Neither family was scraped at all, so every Longhorn alert (degraded, faulted, capacity) and every cert-manager alert (expiry, issuance) was structurally incapable of firing: storage health unmonitored on a cluster whose node just had a failing drive, TLS expiry unmonitored with the live wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with longhorn-backend up=3, certmanager_* 104 series across 3 targets. Adds an L0 guard (118 assertions) that crosses from each alert rule to the scrape config and re-proves each enabling declaration against the working tree, including the negatives — cert-manager's monitor mutual-exclusion, the Longhorn metrics NetworkPolicy, and kube_node_labels needing metricLabelsAllowlist. Proven able to fail five ways. Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an explicit parity contract with the Ansible template that a HelmRelease-only change would silently break. Registered as a declared gap instead. Two findings this surfaced, neither fixed here and neither absorbed silently: twelve apps/* <app>-down SLI rules discover zero targets because their ServiceMonitors name port 'metrics' and no app Service defines that port; and kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen kube_* rules intermittently inert.
This commit is contained in:
jakman85 committed
commit
2f579f103b
10 files changed
+1069
-1
No files matched your search
@@ -0,0 +1,119 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
|
||||
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
|
||||
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
|
||||
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
|
||||
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
|
||||
# survives every dashboard (nobody looks for a panel that was never built).
|
||||
#
|
||||
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
|
||||
# count(longhorn_volume_robustness) => (empty) 4 rules
|
||||
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
|
||||
# count(hubble_drop_total) => (empty) 1 rule
|
||||
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
|
||||
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
|
||||
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
|
||||
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
|
||||
# was the same defect and was fixed separately in a79693d.
|
||||
#
|
||||
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
|
||||
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
||||
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
|
||||
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
|
||||
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
|
||||
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
|
||||
# storage alerts;
|
||||
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
|
||||
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
|
||||
# both set, Longhorn's own NetworkPolicies drop the scrape unless
|
||||
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
|
||||
# kube-state-metrics' metricLabelsAllowlist names the label;
|
||||
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
|
||||
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
|
||||
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
||||
# check above can never see this failure: where the repo authors a Service and a
|
||||
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
|
||||
# port `metrics`; no app Service defines one), so they are registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
|
||||
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
||||
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
||||
#
|
||||
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
|
||||
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
|
||||
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
|
||||
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
||||
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
||||
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
||||
# into fiction.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
|
||||
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
|
||||
# run it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
|
||||
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
|
||||
# TARGET exists, never that the exporter emits that exact SERIES NAME.
|
||||
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
|
||||
# and did not exist at all on the deployed KubeVirt (corrected here to
|
||||
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
|
||||
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
|
||||
# that diffs alert metrics against /api/v1/label/__name__/values.
|
||||
#
|
||||
# Also not proven: that a target is UP. This is static — it asserts the
|
||||
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
|
||||
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
||||
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
|
||||
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
|
||||
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
|
||||
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
|
||||
# (R-OBS-02 / the L3 alerting test).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alert-metrics-have-scrape-targets
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
|
||||
# every scrape-enabling declaration that feeds them, which live in eight different components
|
||||
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
|
||||
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
|
||||
# path-maps to that set.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
|
||||
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
|
||||
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
|
||||
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
|
||||
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
|
||||
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
|
||||
# UNWIRED, never green) rather than silently green.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
|
||||
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
|
||||
monitor can actually produce, and the two gaps that remain are enumerated in registers that
|
||||
can only shrink — so an alert can never again be authored, reviewed and merged against a
|
||||
metric nothing scrapes.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-alert-metric-against-the-configured-scrape-targets
|
||||
description: >-
|
||||
Parse every metric name out of every alerting rule, resolve each against the declared
|
||||
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
|
||||
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
|
||||
agreement, and ratchet both registers of declared gaps.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,702 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
|
||||
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
|
||||
# VPN)." — spec docs/10-observability-and-alerting.md)
|
||||
#
|
||||
# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
|
||||
# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
|
||||
# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
|
||||
# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
|
||||
# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
|
||||
# nobody looks for a panel that was never built.
|
||||
#
|
||||
# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
|
||||
# count(longhorn_volume_robustness) => (empty) LonghornVolumeDegraded/Faulted/
|
||||
# CapacityHigh, LonghornNodeStorageHigh
|
||||
# count({__name__=~"certmanager.*"}) => (empty) CertificateExpiringSoon/
|
||||
# ExpiryImminent/NotReady
|
||||
# count(hubble_drop_total) => (empty) HubbleDeniedFlowSpike
|
||||
# count by (job) (up) => 20 jobs, none Longhorn, cert-manager or Hubble
|
||||
# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
|
||||
# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
|
||||
# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
|
||||
# separately in a79693d.
|
||||
#
|
||||
# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
|
||||
# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
|
||||
# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
|
||||
# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
|
||||
# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
|
||||
# what the owner table below does.
|
||||
#
|
||||
# GIVEN every alerting rule in every PrometheusRule in the repo,
|
||||
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
||||
# THEN (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
|
||||
# owner's `require` regexes still match its file — so deleting the enablement from
|
||||
# platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
|
||||
# storage alerts;
|
||||
# (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
|
||||
# defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
|
||||
# servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
|
||||
# (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
|
||||
# (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
|
||||
# this — by requiring that where this repo authors a Service and a ServiceMonitor in the
|
||||
# SAME file, the monitor's endpoint port is a port that Service actually defines;
|
||||
# violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
|
||||
# AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
|
||||
# so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
|
||||
#
|
||||
# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
|
||||
# alert-rules-unit-tested faced:
|
||||
# * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
|
||||
# Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
|
||||
# none of which speak Prometheus natively. The gate would be red for every change until that
|
||||
# work lands, so it would get disabled, and a disabled gate protects nothing.
|
||||
# * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
|
||||
# * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
|
||||
# GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
|
||||
# with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
|
||||
# also FAILS, so neither register can rot into fiction.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
||||
# - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
|
||||
# values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
||||
# recorded in the commit message: with the enablements applied, longhorn_volume_robustness
|
||||
# returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
|
||||
# up==1. A live probe needs a cluster, which L0 by definition does not have.
|
||||
# - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
|
||||
# repo: the port name lives outside the tree. That is why (4) is scoped to same-file
|
||||
# Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
|
||||
# platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
|
||||
# liveness is covered by the fact that their metrics have owners under (1).
|
||||
# - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
|
||||
# metric in a scraped family whose exact name the exporter does not emit still passes. This is
|
||||
# a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
|
||||
# (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
|
||||
# series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
|
||||
# only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
|
||||
# names are the phase_transition_time histograms. Corrected in the same change to
|
||||
# `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
|
||||
# matcher selects nothing and is just as inert). Catching that class needs the exporter's own
|
||||
# metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
|
||||
# diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
|
||||
# every LABEL VALUE an alert matches on.
|
||||
# - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
|
||||
# label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
|
||||
# metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
|
||||
# be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
|
||||
# finding metrics at all.
|
||||
# - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
|
||||
# L3 alerting test own that).
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
|
||||
# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
|
||||
# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
|
||||
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
|
||||
echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
|
||||
|
||||
PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
|
||||
alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
|
||||
kind of approximate parser that silently stops matching — the exact failure class this check
|
||||
exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
|
||||
#
|
||||
# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
|
||||
# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
|
||||
# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
|
||||
# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
|
||||
# regex is re-evaluated against the working tree on every gate run.
|
||||
#
|
||||
# `require` — ALL must match the file. The enablement itself.
|
||||
# `forbid` — NONE may match. For enablements that can be defeated without being deleted.
|
||||
#
|
||||
# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
|
||||
# =============================================================================================
|
||||
KPS = "platform/monitoring/kube-prometheus-stack.yaml"
|
||||
|
||||
SCRAPE_OWNERS: list[dict] = [
|
||||
{
|
||||
"owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
|
||||
"prefixes": ("node_",),
|
||||
"file": KPS,
|
||||
# The subchart block must be present and configured here (it is where the exporter's
|
||||
# resources live). The chart default is enabled:true, so the regression to guard is a
|
||||
# DISABLE, not a missing enable.
|
||||
"require": [r"^\s*prometheus-node-exporter:\s*$"],
|
||||
"forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
|
||||
"rules that exist because of prod-1's etcd WAL starvation",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
|
||||
"prefixes": ("kubelet_",),
|
||||
"file": KPS,
|
||||
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
|
||||
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
|
||||
"pvc-capacity-alerts rules",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
|
||||
"prefixes": ("kube_velero_schedule_",),
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"customResourceState:\s*\n\s*enabled:\s*true",
|
||||
r"metricNamePrefix:\s*kube_velero_schedule",
|
||||
# The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
|
||||
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
|
||||
# and the inert-alert failure mode all over again (measured; see the block's header).
|
||||
r'apiGroups:\s*\["velero\.io"\]',
|
||||
r'resources:\s*\["schedules"\]',
|
||||
],
|
||||
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
|
||||
"why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
|
||||
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
|
||||
"Velero itself publishes nothing until a backup is attempted",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
|
||||
"prefixes": ("kube_",),
|
||||
"file": KPS,
|
||||
"require": [r"^\s*kube-state-metrics:\s*$"],
|
||||
"forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
# kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
|
||||
# custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
|
||||
# names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
|
||||
# nothing.
|
||||
"why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
|
||||
"rules",
|
||||
},
|
||||
{
|
||||
"owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
|
||||
"exact": ("kube_node_labels",),
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
"kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
|
||||
"matches nothing — a rule that is present, evaluating, and structurally inert",
|
||||
},
|
||||
{
|
||||
"owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
|
||||
"prefixes": ("cnpg_",),
|
||||
"file": "platform/keycloak/postgres-cluster.yaml",
|
||||
"require": [r"^\s*enablePodMonitor:\s*true\s*$"],
|
||||
"forbid": [],
|
||||
"why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
|
||||
"and no-recoverability-point rules that guard R-STATE-03",
|
||||
},
|
||||
{
|
||||
"owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
|
||||
"prefixes": ("gotk_",),
|
||||
"file": "platform/monitoring/flux-podmonitor.yaml",
|
||||
"require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
|
||||
"forbid": [],
|
||||
"why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
|
||||
"automated-update rules — on a platform whose entire delivery model is Flux",
|
||||
},
|
||||
{
|
||||
"owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("certmanager_",),
|
||||
"file": "platform/cert-manager/helmrelease.yaml",
|
||||
"require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
|
||||
# The chart calls `fail` when both monitor kinds are enabled, which would break the whole
|
||||
# HelmRelease rather than just the scrape. Measured against v1.21.2.
|
||||
"forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
|
||||
"why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
|
||||
"warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
|
||||
},
|
||||
{
|
||||
"owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("longhorn_",),
|
||||
"file": "platform/longhorn/helmrelease.yaml",
|
||||
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
|
||||
"forbid": [],
|
||||
"why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
|
||||
"is a disk",
|
||||
# The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
|
||||
"companion": {
|
||||
"file": "platform/longhorn/kustomization.yaml",
|
||||
"require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
|
||||
"why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
|
||||
"(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
|
||||
"networkPolicies.enabled) whose longhorn-manager policy admits only other "
|
||||
"Longhorn components in the same namespace. MEASURED on prod: with the "
|
||||
"ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
|
||||
"every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
|
||||
"a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
|
||||
"the four storage alerts are inert even though the monitor exists",
|
||||
},
|
||||
},
|
||||
{
|
||||
"owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
|
||||
"prefixes": ("kubevirt_",),
|
||||
"file": "platform/kubevirt/servicemonitor.yaml",
|
||||
"require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
|
||||
"forbid": [],
|
||||
"why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
|
||||
"the kubevirt_cdi_* names, not bare cdi_*)",
|
||||
},
|
||||
{
|
||||
"owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("velero_",),
|
||||
"file": "platform/velero/helmrelease.yaml",
|
||||
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
|
||||
"forbid": [],
|
||||
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
|
||||
"that failed to notice prod running with zero volume backups",
|
||||
},
|
||||
]
|
||||
|
||||
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
|
||||
# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
|
||||
UNIVERSAL = {"up"}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
|
||||
# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
|
||||
# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
|
||||
# or that gains an owner, FAILS as stale.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_METRICS: dict[str, str] = {
|
||||
"hubble_drop_total":
|
||||
"HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
|
||||
"visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
|
||||
"only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
|
||||
"in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
|
||||
"but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
|
||||
"OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
|
||||
"— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
|
||||
"cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
|
||||
"and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
|
||||
"commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
|
||||
"one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
|
||||
"header in platform/monitoring/alerts.yaml.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
|
||||
# names an endpoint port that the Service it selects does not define, so the monitor discovers
|
||||
# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
|
||||
# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
|
||||
# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
|
||||
# has no up series to be 0.
|
||||
#
|
||||
# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
|
||||
# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
|
||||
# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
|
||||
# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
|
||||
# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
|
||||
# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
|
||||
# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
|
||||
# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {
|
||||
"gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
|
||||
"jackett": "apps/wave-0/jackett.yaml — no exporter",
|
||||
"pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
|
||||
"plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
|
||||
"qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
|
||||
"vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
|
||||
"keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
|
||||
"keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
|
||||
"neither, so it is a rename away from being live rather than an exporter away",
|
||||
"nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
|
||||
"radarr": "apps/wave-1/radarr.yaml — needs exportarr",
|
||||
"semaphore": "apps/wave-1/semaphore.yaml — no exporter",
|
||||
"sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
|
||||
"seerr": "apps/wave-2/seerr.yaml — no exporter",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) PARSE EVERY ALERT RULE
|
||||
# =============================================================================================
|
||||
# PromQL metric-name approximation. Documented as a gap in the header: string literals,
|
||||
# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
|
||||
# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
|
||||
# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
|
||||
# approximate parser silently stops guarding.
|
||||
PROMQL_KEYWORDS = set("""
|
||||
abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
|
||||
day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
|
||||
histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
|
||||
label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
|
||||
sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
|
||||
count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
|
||||
last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
|
||||
quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
|
||||
double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
|
||||
by without on ignoring group_left group_right offset bool and or unless if atan2 start end
|
||||
inf nan NaN Inf
|
||||
""".split())
|
||||
|
||||
IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
|
||||
UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
|
||||
|
||||
|
||||
def metric_names(expr: str) -> list[str]:
|
||||
e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
|
||||
e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
|
||||
e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
|
||||
e = re.sub(r"\{[^{}]*\}", " ", e) # label matchers
|
||||
for _ in range(6): # nested aggregations
|
||||
e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
|
||||
out, seen = [], set()
|
||||
for m in IDENT_RE.finditer(e):
|
||||
n = m.group(1)
|
||||
if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
|
||||
continue
|
||||
seen.add(n)
|
||||
out.append(n)
|
||||
return out
|
||||
|
||||
|
||||
def is_golden_fixture(p: Path) -> bool:
|
||||
"""conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
|
||||
|
||||
conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
|
||||
rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
|
||||
reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
|
||||
same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
|
||||
fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
|
||||
"""
|
||||
parts = p.parts
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
"this check would silently audit nothing")
|
||||
|
||||
# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
|
||||
refs: dict[str, list[str]] = {}
|
||||
job_refs: dict[str, list[str]] = {}
|
||||
n_rules = 0
|
||||
for f in rule_files:
|
||||
rel = str(f.relative_to(ROOT))
|
||||
try:
|
||||
docs = list(yaml.safe_load_all(read(f)))
|
||||
except yaml.YAMLError as exc:
|
||||
check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
|
||||
continue
|
||||
for d in docs:
|
||||
if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
|
||||
continue
|
||||
for grp in (d.get("spec") or {}).get("groups") or []:
|
||||
for r in grp.get("rules") or []:
|
||||
if "alert" not in r:
|
||||
continue # recording rules: not this check's subject
|
||||
n_rules += 1
|
||||
site = f"{rel}:{r['alert']}"
|
||||
expr = str(r.get("expr", ""))
|
||||
for n in metric_names(expr):
|
||||
refs.setdefault(n, []).append(site)
|
||||
for j in UP_JOB_RE.findall(expr):
|
||||
job_refs.setdefault(j, []).append(site)
|
||||
|
||||
check(n_rules > 0,
|
||||
"no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
|
||||
"check would pass vacuously while every alert's metric went unaudited")
|
||||
check(len(refs) >= 20,
|
||||
f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
|
||||
f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
|
||||
f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
|
||||
f"with a real parser")
|
||||
|
||||
# =============================================================================================
|
||||
# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
|
||||
# =============================================================================================
|
||||
def resolve(metric: str) -> dict | None:
|
||||
"""Longest-prefix / exact-name match against SCRAPE_OWNERS."""
|
||||
best, best_len = None, -1
|
||||
for o in SCRAPE_OWNERS:
|
||||
if metric in o.get("exact", ()): # exact names win outright
|
||||
return o
|
||||
for pfx in o.get("prefixes", ()):
|
||||
if metric.startswith(pfx) and len(pfx) > best_len:
|
||||
best, best_len = o, len(pfx)
|
||||
return best
|
||||
|
||||
|
||||
# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
|
||||
# at a deleted file is a claim nothing backs.
|
||||
for o in SCRAPE_OWNERS:
|
||||
check((ROOT / o["file"]).is_file(),
|
||||
f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
|
||||
f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
|
||||
f"claim is unbacked and the alerts that depend on it are inert")
|
||||
|
||||
claimed: dict[str, dict] = {}
|
||||
unowned: list[str] = []
|
||||
for metric in sorted(refs):
|
||||
if metric in UNIVERSAL:
|
||||
continue
|
||||
o = resolve(metric)
|
||||
if o is None:
|
||||
if metric in KNOWN_UNWIRED_METRICS:
|
||||
continue # declared gap; ratcheted below
|
||||
unowned.append(metric)
|
||||
check(False,
|
||||
f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
|
||||
f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
|
||||
f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
|
||||
f"vector, stays silent forever, and is indistinguishable from a healthy result. "
|
||||
f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
|
||||
f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
|
||||
f"instead of silent")
|
||||
continue
|
||||
check(metric not in KNOWN_UNWIRED_METRICS,
|
||||
f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
|
||||
f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
|
||||
f"the one that shows up")
|
||||
claimed[metric] = o
|
||||
|
||||
# The owners that are actually load-bearing right now get their declarations verified.
|
||||
for o in SCRAPE_OWNERS:
|
||||
if o not in claimed.values():
|
||||
continue # nothing queries it today; nothing to guard
|
||||
body = read(ROOT / o["file"])
|
||||
for rx in o.get("require", []):
|
||||
check(re.search(rx, body, re.M) is not None,
|
||||
f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
|
||||
f"Removing it re-inerts every alert on "
|
||||
f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
|
||||
for rx in o.get("forbid", []):
|
||||
check(re.search(rx, body, re.M) is None,
|
||||
f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
|
||||
f"even though the enablement is still authored. {o['why']}")
|
||||
comp = o.get("companion")
|
||||
if comp:
|
||||
cbody = read(ROOT / comp["file"])
|
||||
check((ROOT / comp["file"]).is_file(),
|
||||
f"{comp['file']} is missing — {comp['why']}")
|
||||
for rx in comp.get("require", []):
|
||||
check(re.search(rx, cbody, re.M) is not None,
|
||||
f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
|
||||
f"{comp['why']}")
|
||||
|
||||
# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
|
||||
# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
|
||||
NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
|
||||
if any(m.startswith("longhorn_") for m in refs):
|
||||
np = read(NP)
|
||||
check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
|
||||
f"four alerts are inert")
|
||||
check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
|
||||
"platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
|
||||
check(re.search(r"app:\s*longhorn-manager", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
|
||||
"open the pods that serve /metrics")
|
||||
check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
|
||||
"still cannot reach longhorn-manager and the storage alerts stay inert")
|
||||
check(re.search(r"port:\s*9500", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
|
||||
"port, which serves /metrics)")
|
||||
# LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
|
||||
check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
|
||||
and "Egress" not in np,
|
||||
"the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
|
||||
"scrape path and nothing else (R-SEC-02)")
|
||||
# Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
|
||||
# they stay closed, and a substring search over prose would fail on the documentation of the
|
||||
# very property it is asserting.
|
||||
np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
|
||||
for closed in ("9502", "9503"):
|
||||
check(closed not in np_spec,
|
||||
f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
|
||||
f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
|
||||
f"is reach this observability path does not need (R-SEC-02)")
|
||||
|
||||
# stale KNOWN_UNWIRED_METRICS entries
|
||||
for m in sorted(KNOWN_UNWIRED_METRICS):
|
||||
check(m in refs,
|
||||
f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
|
||||
f"more — a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
# =============================================================================================
|
||||
# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
|
||||
#
|
||||
# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
|
||||
# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
|
||||
# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
|
||||
# =============================================================================================
|
||||
def yaml_docs(path: Path) -> list[dict]:
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
continue
|
||||
docs = yaml_docs(p)
|
||||
monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
|
||||
services = [d for d in docs if d.get("kind") == "Service"]
|
||||
if not monitors or not services:
|
||||
continue
|
||||
svc_ports: set[str] = set()
|
||||
for s in services:
|
||||
for port in (s.get("spec") or {}).get("ports") or []:
|
||||
if isinstance(port, dict) and port.get("name"):
|
||||
svc_ports.add(str(port["name"]))
|
||||
for m in monitors:
|
||||
ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
|
||||
if isinstance(e, dict) and e.get("port")}
|
||||
name = (m.get("metadata") or {}).get("name")
|
||||
if name:
|
||||
same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
|
||||
|
||||
for job in sorted(job_refs):
|
||||
sites = ", ".join(sorted(set(job_refs[job])))
|
||||
mon = same_file_monitors.get(job)
|
||||
if mon is None:
|
||||
# No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
|
||||
# by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
|
||||
# are), which this check cannot see — so this is NOT a failure, only unauditable.
|
||||
continue
|
||||
rel, ep_ports, svc_ports = mon
|
||||
missing = sorted(ep_ports - svc_ports)
|
||||
if missing:
|
||||
check(job in KNOWN_UNWIRED_JOBS,
|
||||
f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
|
||||
f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
|
||||
f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
|
||||
f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
|
||||
f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
|
||||
f"a target that was never discovered has no up series to be 0. Give the Service a "
|
||||
f"named port the workload actually serves metrics on (and an exporter if it has "
|
||||
f"none), or declare it in KNOWN_UNWIRED_JOBS")
|
||||
else:
|
||||
check(job not in KNOWN_UNWIRED_JOBS,
|
||||
f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
|
||||
f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
|
||||
f"delete the entry")
|
||||
|
||||
# stale KNOWN_UNWIRED_JOBS entries
|
||||
for job in sorted(KNOWN_UNWIRED_JOBS):
|
||||
check(job in job_refs,
|
||||
f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
|
||||
f"more — a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
|
||||
# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
|
||||
# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
|
||||
# HONOURED by ci/l0.sh's --include-scope union.
|
||||
# =============================================================================================
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
|
||||
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
|
||||
/ "chainsaw-test.yaml")
|
||||
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
|
||||
"alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
|
||||
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
|
||||
"change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
|
||||
"platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
|
||||
"chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
|
||||
"that re-inerts the alerts")
|
||||
L0 = read(ROOT / "ci" / "l0.sh")
|
||||
GATE = read(ROOT / "ci" / "gate.sh")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
|
||||
"purely requirement-scoped run again and the scope annotation above is inert decoration")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
|
||||
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ======================================================================== report
|
||||
owners_used = {id(o): o for o in claimed.values()}
|
||||
print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
|
||||
f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
|
||||
print(f" claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
|
||||
f" owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
|
||||
for o in SCRAPE_OWNERS:
|
||||
ms = sorted(m for m, oo in claimed.items() if oo is o)
|
||||
mark = " " if ms else "· "
|
||||
print(f" {mark}{o['file']:<46} {len(ms):>2} metric(s) {o['owner'][:64]}")
|
||||
print(f"\n DECLARED GAPS (shrink-only registers):")
|
||||
print(f" KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
|
||||
f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
|
||||
print(f" KNOWN_UNWIRED_JOBS: {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
|
||||
f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
|
||||
if unowned:
|
||||
print(f"\n UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -31,3 +31,53 @@ spec:
|
||||
# cluster/split-horizon DNS (docs/09 split-horizon note; realises the TODO that was here).
|
||||
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
||||
dns01RecursiveNameserversOnly: true
|
||||
# --- SCRAPE THE TLS ISSUER, or every certificate alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships CertificateExpiringSoon, CertificateExpiryImminent
|
||||
# (critical) and CertificateNotReady — and NONE of them could ever fire. The chart exposes the
|
||||
# controller's metrics port by default (`prometheus.enabled: true`, confirmed on prod:
|
||||
# Service/cert-manager listens on 9402) but defaults `prometheus.servicemonitor.enabled: false`,
|
||||
# and no ServiceMonitor existed in ns cert-manager. VERIFIED live on prod 2026-09-13:
|
||||
# `count({__name__=~"certmanager.*"})` returned NOTHING and `count by (job) (up)` listed no
|
||||
# cert-manager job — so certmanager_certificate_expiration_timestamp_seconds and
|
||||
# certmanager_certificate_ready_status were never in the TSDB.
|
||||
#
|
||||
# WHY THIS IS PART OF THE TLS PATH: the live wildcard `wildcard-home` (ns gateway) expires
|
||||
# 2026-12-12T02:16:57Z and is renewed by an ACME DNS-01 order that can fail quietly — a Cloudflare
|
||||
# token rotation, a DNS-01 self-check timeout (the very failure the two lines above work around),
|
||||
# a rate limit. With no scrape, the FIRST signal of a failed renewal is a browser TLS error for
|
||||
# every *.${BASE_DOMAIN} service. That is a user-discovered outage on a fully automated path,
|
||||
# which is exactly what R-TLS-01/R-OBS-04 exist to prevent.
|
||||
#
|
||||
# Chart mechanics, verified by rendering cert-manager v1.21.2 (the version this range resolves to):
|
||||
# * templates/servicemonitor.yaml renders on `prometheus.enabled AND
|
||||
# prometheus.servicemonitor.enabled`. Measured: `helm template --set
|
||||
# prometheus.servicemonitor.enabled=true` emits exactly 1 ServiceMonitor, 0 without it.
|
||||
# `prometheus.enabled` is already true by chart default, so this one line is sufficient.
|
||||
# * podmonitor and servicemonitor are MUTUALLY EXCLUSIVE — the chart calls `fail` if both are
|
||||
# set. Do not add `prometheus.podmonitor.enabled` alongside this.
|
||||
# * the rendered ServiceMonitor selects the controller/webhook/cainjector Services by
|
||||
# app.kubernetes.io/{name,instance,component} on targetPort `http-metrics`, and sets
|
||||
# `jobLabel: app.kubernetes.io/name` — so the scrape job lands as `cert-manager`.
|
||||
# * it carries the label `prometheus: default` (servicemonitor.prometheusInstance). Harmless
|
||||
# here: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues false, so Prometheus
|
||||
# selects every ServiceMonitor regardless of labels. Left at the chart default deliberately —
|
||||
# overriding it would imply a label-selecting Prometheus this cluster does not have.
|
||||
# Cost: 3 scrape targets, one certmanager_* series set — negligible.
|
||||
#
|
||||
# PROVEN END TO END ON PROD, 2026-09-13 21:0xZ, by applying the rendered ServiceMonitor by hand
|
||||
# (then removing it, so the merge's Helm upgrade owns it) — no NetworkPolicy was needed, as
|
||||
# neither ns cert-manager nor ns monitoring carries one:
|
||||
# up{namespace="cert-manager"} => 3 targets, all 1
|
||||
# (jobs cert-manager / cainjector / webhook,
|
||||
# because jobLabel is app.kubernetes.io/name)
|
||||
# count({__name__=~"certmanager.*"}) => 104 (was EMPTY)
|
||||
# (certmanager_certificate_expiration_timestamp_seconds - time())/86400
|
||||
# => 89.2 days for `wildcard-home` (ns gateway,
|
||||
# ClusterIssuer letsencrypt-prod)
|
||||
# certmanager_certificate_ready_status{condition="True"} => 1
|
||||
# So the three Certificate* rules now evaluate against a real series and are correctly QUIET:
|
||||
# the wildcard is healthy and 89 days out, which is exactly what CertificateExpiringSoon's 14-day
|
||||
# threshold is meant to catch and could not have.
|
||||
prometheus:
|
||||
servicemonitor:
|
||||
enabled: true
|
||||
@@ -4,3 +4,8 @@
|
||||
# dns01RecursiveNameservers / -Only # TODO(WS-SEC): may be needed with split-horizon DNS
|
||||
# ClusterIssuers (letsencrypt-staging/-prod) live in clusterissuers.yaml, not here.
|
||||
# Cloudflare API token: SOPS-encrypted secret `cloudflare-api-token` (WS-SEC, OQ-4) — name only.
|
||||
# prometheus.servicemonitor.enabled: true # scrape the controller's 9402 metrics port; without it
|
||||
# # the certmanager_* family is absent from the TSDB and
|
||||
# # the three Certificate* alert rules in
|
||||
# # platform/monitoring/alerts.yaml are INERT. Mutually
|
||||
# # exclusive with prometheus.podmonitor.enabled.
|
||||
@@ -22,8 +22,22 @@ spec:
|
||||
rules:
|
||||
# Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
|
||||
# sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
|
||||
#
|
||||
# METRIC CORRECTED 2026-09-13 (was `kubevirt_vmi_phase_count{phase="Failed"}`). That series
|
||||
# DOES NOT EXIST on the deployed KubeVirt, so this rule was structurally INERT — unlike the
|
||||
# Longhorn/cert-manager/Flux gaps in the same audit, NOT for want of a scrape target: the
|
||||
# `kubevirt-prometheus-metrics` job is up and its siblings below (virt_api_ready_status,
|
||||
# virt_controller_ready_status, cdi_import_pods_high_restart) all resolve. KubeVirt simply
|
||||
# retired the phase-count gauge in favour of an info-style series carrying `phase` as a label.
|
||||
# VERIFIED live on prod 2026-09-13:
|
||||
# count(kubevirt_vmi_phase_count) => (empty)
|
||||
# count(kubevirt_vmi_info) => 1
|
||||
# kubevirt_vmi_info{...,name="unifi",namespace="unifi",phase="running",...} => 1
|
||||
# Note `phase="running"` — KubeVirt LOWERCASES the phase in this label, so the matcher is
|
||||
# "failed", not "Failed". A capital-F matcher here would silently select nothing and leave the
|
||||
# rule just as inert as the metric name did; that is the whole point of this class of defect.
|
||||
- alert: KubeVirtVMLaunchFailing
|
||||
expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
|
||||
expr: sum by (namespace) (kubevirt_vmi_info{phase="failed"}) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
|
||||
@@ -102,6 +102,54 @@ spec:
|
||||
# Reclaim replica space of deleted/failed volumes promptly in dev.
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
|
||||
# LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
|
||||
# and NONE of them could ever fire, because the chart defaults
|
||||
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
||||
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
||||
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
||||
# So the entire longhorn_* family was absent from the TSDB and four alert rules were decoration.
|
||||
#
|
||||
# WHY THIS IS PART OF THE STORAGE PATH, NOT AN OBSERVABILITY NICE-TO-HAVE: this cluster's
|
||||
# recurring failure mode is a disk. prod-1's failing SSD starved etcd's WAL fsync and wedged the
|
||||
# node (DECISIONS-LOG; the prod-1 drive replacement), and a degraded/faulted replica is precisely
|
||||
# the first symptom Longhorn publishes. R-STATE-02 promises 3-way replication survives a node
|
||||
# loss; with no scrape, a volume can sit at 2 replicas — or faulted — indefinitely, unalerted.
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml's header already flagged this exact gap as "a
|
||||
# separate defect (a missing Longhorn ServiceMonitor/metrics config)". This closes it.
|
||||
#
|
||||
# Chart mechanics, verified by rendering longhorn 1.12.1 (the version this range resolves to):
|
||||
# * templates/servicemonitor.yaml is gated on `{{- if .Values.metrics.serviceMonitor.enabled }}`
|
||||
# and NOTHING else — no `.Capabilities` probe. Measured: `helm template --set
|
||||
# metrics.serviceMonitor.enabled=true` emits exactly 1 ServiceMonitor, and 0 without the flag.
|
||||
# So this line is both necessary and sufficient; unlike velero's, it needs no API capability.
|
||||
# * the rendered ServiceMonitor selects `app: longhorn-manager` in the release namespace on the
|
||||
# port named `manager`. CONFIRMED against prod: Service/longhorn-backend carries
|
||||
# `app: longhorn-manager` and a port `manager` (9500) — the selector resolves to real
|
||||
# endpoints, it is not a dangling monitor (which is how the apps/ SLI monitors fail: they name
|
||||
# a `metrics` port their Service does not have, so they discover zero targets and their
|
||||
# `up{job=...}` alerts are inert in the very same way).
|
||||
# * no `additionalLabels` needed: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues
|
||||
# false, so Prometheus selects every ServiceMonitor regardless of labels.
|
||||
# Cost: one scrape target per manager pod (3), the longhorn_* families — low hundreds of series
|
||||
# on this cluster's volume count.
|
||||
#
|
||||
# PROVEN END TO END ON PROD, 2026-09-13 21:0xZ. The rendered ServiceMonitor was applied to prod
|
||||
# by hand (then removed again, so the merge's Helm upgrade creates and OWNS it rather than
|
||||
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
||||
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
||||
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
||||
# count(longhorn_volume_robustness) => 44
|
||||
# count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached)
|
||||
# and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
|
||||
# them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
|
||||
# volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
|
||||
# freed blocks. Reported, NOT silenced by moving the threshold).
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
|
||||
# Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
|
||||
# oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.
|
||||
|
||||
|
||||
@@ -5,4 +5,10 @@ resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml
|
||||
# Opens ns monitoring -> longhorn-manager:9500 so Prometheus can actually scrape the metrics the
|
||||
# HelmRelease's `metrics.serviceMonitor.enabled` exposes. Longhorn 1.12.1's own (effectively
|
||||
# unconditional) NetworkPolicies drop that path, so WITHOUT this the ServiceMonitor discovers 3
|
||||
# targets that all sit at up==0 and the Longhorn alerts stay inert. See the file header for the
|
||||
# measured proof. (2026-09-13, from the inert-alert audit.)
|
||||
- networkpolicy-metrics.yaml
|
||||
- storageclasses.yaml
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
# Allow Prometheus to reach longhorn-manager's metrics port (R-OBS-04, R-STATE-02).
|
||||
#
|
||||
# WITHOUT THIS FILE, `metrics.serviceMonitor.enabled: true` IN helmrelease.yaml IS NOT ENOUGH and the
|
||||
# four Longhorn alert rules in platform/monitoring/alerts.yaml STAY INERT. This was measured, not
|
||||
# predicted — prod, 2026-09-13, with the ServiceMonitor in place:
|
||||
#
|
||||
# count by (job) (up) => {job="longhorn-backend"} => 3 (targets DISCOVERED)
|
||||
# up{job="longhorn-backend"} => 0, 0, 0 (every scrape FAILING)
|
||||
# count(longhorn_volume_robustness) => (empty) (still nothing in the TSDB)
|
||||
# /api/v1/targets lastError => 'Get "http://10.0.0.169:9500/metrics": context deadline
|
||||
# exceeded' (10.0007s — a DROP, not a refusal)
|
||||
#
|
||||
# The endpoint itself is healthy and fast; only the path from ns monitoring is blocked. Proven by
|
||||
# curling it from inside the manager pod:
|
||||
# http://<podIP>:9500/metrics -> 200 in 0.031s (the manager binds the POD IP, not localhost —
|
||||
# http://localhost:9500/metrics -> 000 which is why a localhost probe misleads here)
|
||||
#
|
||||
# WHY IT IS BLOCKED: longhorn 1.12.1 ships SIX NetworkPolicies of its own, and the one selecting
|
||||
# `app: longhorn-manager` is Ingress-only with a from-list of nothing but other Longhorn components
|
||||
# (longhorn-manager, longhorn-ui, longhorn-csi-plugin, the recurring-job/job-task pods,
|
||||
# longhorn-driver-deployer) — all bare podSelectors, i.e. same-namespace. Prometheus lives in ns
|
||||
# monitoring, so it matches no clause and its packets are dropped.
|
||||
#
|
||||
# THE CHART OFFERS NO KNOB FOR THIS, and the obvious-looking one is a trap. `networkPolicies.enabled`
|
||||
# (default false) does NOT gate these templates — verified by rendering 1.12.1: `helm template` emits
|
||||
# the same 6 NetworkPolicies with the flag on AND off, because templates/network-policies/* are
|
||||
# guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) instead. So the policies are
|
||||
# effectively unconditional, and nothing in the chart adds a Prometheus allowance when
|
||||
# metrics.serviceMonitor.enabled is set.
|
||||
#
|
||||
# WHY NOT `networkPolicies.restrictInternalTraffic: false`: that is the one-line fix and it is the
|
||||
# wrong one. It deletes Longhorn's entire internal-traffic restriction across all six components to
|
||||
# open one port to one namespace — a straight R-SEC-02 (least network reach) regression, trading a
|
||||
# security control for an observability one when the two are not in conflict.
|
||||
#
|
||||
# THIS IS ADDITIVE AND NARROW. Kubernetes NetworkPolicies are OR'd (Cilium enforces the UNION — see
|
||||
# platform/headlamp/networkpolicy.yaml's header), so this grants exactly one new path on top of
|
||||
# Longhorn's floor: ns monitoring -> app=longhorn-manager, TCP 9500 only. Longhorn's own policy is
|
||||
# left untouched, every other component stays restricted, and nothing else in longhorn-system becomes
|
||||
# reachable.
|
||||
#
|
||||
# TIER: a core networking.k8s.io kind with no CRD dependency, so it belongs to the `platform`
|
||||
# Kustomization alongside the HelmRelease (referenced from platform/longhorn/kustomization.yaml) —
|
||||
# NOT platform-config, which exists for CRs of platform-installed CRDs.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: longhorn-manager-allow-prometheus-scrape
|
||||
namespace: longhorn-system
|
||||
labels:
|
||||
app.kubernetes.io/name: longhorn
|
||||
# Marks this as an observability path, not a data path — so a future reviewer trimming Longhorn
|
||||
# network reach can see at a glance what breaks (the storage alerts) if they delete it.
|
||||
homelab.keithjackson.dev/purpose: metrics-scrape
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: longhorn-manager
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
# The whole monitoring namespace rather than the Prometheus pod labels: the operator owns
|
||||
# those labels and rewrites them on every kube-prometheus-stack bump (they carry the chart's
|
||||
# instance name and a controller-revision hash). A pod-label selector here would silently
|
||||
# stop matching on an upgrade and put the storage alerts back to inert — the exact regression
|
||||
# class this whole change exists to close. The namespace holds only the monitoring stack.
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
# longhorn-manager's container port `manager` (Service longhorn-backend:9500), which serves
|
||||
# /metrics. NOT 9502/9503 (admission webhook / recovery backend) — those stay closed.
|
||||
port: 9500
|
||||
@@ -22,3 +22,10 @@
|
||||
# defaultDataLocality: disabled # keep replicas spread for durability
|
||||
# orphanAutoDeletion: true # reclaim orphaned replica space promptly
|
||||
# backupTarget: <off-cluster> # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
|
||||
#
|
||||
# metrics:
|
||||
# serviceMonitor:
|
||||
# enabled: true # scrape longhorn-manager (port `manager`, 9500);
|
||||
# # without it the whole longhorn_* family is absent
|
||||
# # from the TSDB and the four Longhorn alert rules
|
||||
# # in platform/monitoring/alerts.yaml are INERT
|
||||
@@ -333,6 +333,47 @@ spec:
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
||||
# health is covered by the Cilium platform alerts.
|
||||
# *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
|
||||
# *** ServiceMonitor: a monitor alone cannot make this rule fire.
|
||||
# Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
|
||||
# (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
|
||||
# stale entry there FAILS the gate, so this cannot quietly become fiction.
|
||||
#
|
||||
# VERIFIED live on prod 2026-09-13:
|
||||
# count(hubble_drop_total) => (empty)
|
||||
# count({__name__=~"hubble.*"}) => (empty)
|
||||
# count({__name__=~"cilium.*"}) => (empty)
|
||||
# count by (job) (up) => 20 jobs, none Cilium or Hubble
|
||||
# (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
|
||||
# So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
|
||||
# platform has NO policy-drop visibility at all today.
|
||||
#
|
||||
# WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
|
||||
# /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
|
||||
# switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
|
||||
# `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
|
||||
# `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
|
||||
# them on means:
|
||||
# * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
|
||||
# which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
|
||||
# three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
|
||||
# the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
|
||||
# OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
|
||||
# * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
|
||||
# carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
|
||||
# against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
|
||||
# a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
|
||||
# invisible until the next node rebuild.
|
||||
# * per-flow label cardinality: `drop` metrics carry source/destination context, and the
|
||||
# default contexts can multiply series across every pod pair that gets denied — on a
|
||||
# default-deny cluster, denials are routine, not exceptional.
|
||||
# RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
|
||||
# on the cilium Kustomization, with the Ansible parity edit in the same commit, the
|
||||
# labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
|
||||
# and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
|
||||
# inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
|
||||
# audit), which also means a NetworkPolicy change cannot be observed to have started dropping
|
||||
# legitimate traffic.
|
||||
- alert: HubbleDeniedFlowSpike
|
||||
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
||||
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
||||
|
||||
Reference in new issue
Block a user