diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml new file mode 100644 index 0000000..f0fa6dc --- /dev/null +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml @@ -0,0 +1,119 @@ +# ============================================================================= +# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is +# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no +# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and +# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is +# right), it survives `promtool test rules` (the unit test supplies the series itself), and it +# survives every dashboard (nobody looks for a panel that was never built). +# +# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert: +# count(longhorn_volume_robustness) => (empty) 4 rules +# count({__name__=~"certmanager.*"}) => (empty) 3 rules +# count(hubble_drop_total) => (empty) 1 rule +# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble +# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD +# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from +# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules) +# was the same defect and was fixed separately in a79693d. +# +# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*), +# WHEN each rule's PromQL is parsed for the metric names it selects, +# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's +# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain +# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from +# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four +# storage alerts; +# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without +# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are +# both set, Longhorn's own NetworkPolicies drop the scrape unless +# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless +# kube-state-metrics' metricLabelsAllowlist names the label; +# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a +# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded); +# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric +# check above can never see this failure: where the repo authors a Service and a +# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service +# actually defines. All 12 apps/* `-down` SLI rules fail this today (the monitors name +# port `metrics`; no app Service defines one), so they are registered in +# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist; +# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`, +# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate +# invocation, and ci/gate.sh blocks the merge when the l0 stage fails. +# +# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as +# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until +# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends — +# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate +# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases. +# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed +# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE +# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot +# into fiction. +# +# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as +# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh +# run it directly, and this Chainsaw Test wraps the SAME script. +# +# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating +# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE +# TARGET exists, never that the exporter emits that exact SERIES NAME. +# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target +# and did not exist at all on the deployed KubeVirt (corrected here to +# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching +# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check +# that diffs alert metrics against /api/v1/label/__name__/values. +# +# Also not proven: that a target is UP. This is static — it asserts the +# enabling declaration is in git, never that the endpoint answers, that the chart still honours the +# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads +# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series, +# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart +# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to +# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human +# (R-OBS-02 / the L3 alerting test). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: alert-metrics-have-scrape-targets + labels: + conformance.home.k3s/R-OBS-04: "true" + annotations: + conformance.home.k3s/requirements: "R-OBS-04" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over + # every scrape-enabling declaration that feeds them, which live in eight different components + # (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero, + # platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement + # path-maps to that set. + # + # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and + # platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under + # apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to + # false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS + # CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes + # ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 = + # UNWIRED, never green) rather than silently green. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every metric name queried by every alert rule is claimed by a scrape target whose enabling + declaration is still present in this repo, every `up{job=...}` selector names a job a repo + monitor can actually produce, and the two gaps that remain are enumerated in registers that + can only shrink — so an alert can never again be authored, reviewed and merged against a + metric nothing scrapes. +spec: + steps: + - name: audit-every-alert-metric-against-the-configured-scrape-targets + description: >- + Parse every metric name out of every alerting rule, resolve each against the declared + scrape-owner table, prove each owner's enabling declaration (and its negative conditions) + still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port + agreement, and ratchet both registers of declared gaps. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh new file mode 100755 index 0000000..e4fe89b --- /dev/null +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh @@ -0,0 +1,702 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets +# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs, +# VPN)." — spec docs/10-observability-and-alerting.md) +# +# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE +# ----------------------------------------------------------------------------------------------- +# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series +# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and +# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is +# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit +# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because +# nobody looks for a panel that was never built. +# +# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert: +# count(longhorn_volume_robustness) => (empty) LonghornVolumeDegraded/Faulted/ +# CapacityHigh, LonghornNodeStorageHigh +# count({__name__=~"certmanager.*"}) => (empty) CertificateExpiringSoon/ +# ExpiryImminent/NotReady +# count(hubble_drop_total) => (empty) HubbleDeniedFlowSpike +# count by (job) (up) => 20 jobs, none Longhorn, cert-manager or Hubble +# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS +# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no +# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed +# separately in a79693d. +# +# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a +# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is +# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule +# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different +# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is +# what the owner table below does. +# +# GIVEN every alerting rule in every PrometheusRule in the repo, +# WHEN each rule's PromQL is parsed for the metric names it selects, +# THEN (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the +# owner's `require` regexes still match its file — so deleting the enablement from +# platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four +# storage alerts; +# (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be +# defeated without being deleted (cert-manager's chart `fail`s if podmonitor and +# servicemonitor are both enabled; an `enabled: false` inside the exporter's own block); +# (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only); +# (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to +# this — by requiring that where this repo authors a Service and a ServiceMonitor in the +# SAME file, the monitor's endpoint port is a port that Service actually defines; +# violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only); +# AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo` +# so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure. +# +# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as +# alert-rules-unit-tested faced: +# * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need +# Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr — +# none of which speak Prometheus natively. The gate would be red for every change until that +# work lands, so it would get disabled, and a disabled gate protects nothing. +# * merely REPORT: that is precisely how these rules stayed inert across phases 4-6. +# * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT +# GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair +# with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted — +# also FAILS, so neither register can rot into fiction. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): +# - It does not prove a target is UP, that an endpoint answers, that the chart still honours the +# values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads +# recorded in the commit message: with the enablements applied, longhorn_volume_robustness +# returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read +# up==1. A live probe needs a cluster, which L0 by definition does not have. +# - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this +# repo: the port name lives outside the tree. That is why (4) is scoped to same-file +# Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the +# platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their +# liveness is covered by the fact that their metrics have owners under (1). +# - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a +# metric in a scraped family whose exact name the exporter does not emit still passes. This is +# a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}` +# (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the +# series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the +# only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing +# names are the phase_transition_time histograms. Corrected in the same change to +# `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F +# matcher selects nothing and is just as inert). Catching that class needs the exporter's own +# metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that +# diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to +# every LABEL VALUE an alert matches on. +# - The metric-name parser is a PromQL approximation, not a parser: it strips string literals, +# label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as +# metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would +# be missed. No rule in this repo does that, and assertion (0) fails if the parser stops +# finding metrics at all. +# - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the +# L3 alerting test own that). +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). +# +# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test +# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the +# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all. +if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then + echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures. + +PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour: +alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the +kind of approximate parser that silently stops matching — the exact failure class this check +exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code. +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree + +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def read(p: Path) -> str: + try: + return p.read_text() + except (OSError, UnicodeDecodeError): + return "" + + +# ============================================================================================= +# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2). +# +# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose +# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This +# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the +# part that makes the check a REGRESSION guard rather than a one-time audit: every `require` +# regex is re-evaluated against the working tree on every gate run. +# +# `require` — ALL must match the file. The enablement itself. +# `forbid` — NONE may match. For enablements that can be defeated without being deleted. +# +# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_. +# ============================================================================================= +KPS = "platform/monitoring/kube-prometheus-stack.yaml" + +SCRAPE_OWNERS: list[dict] = [ + { + "owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor", + "prefixes": ("node_",), + "file": KPS, + # The subchart block must be present and configured here (it is where the exporter's + # resources live). The chart default is enabled:true, so the regression to guard is a + # DISABLE, not a missing enable. + "require": [r"^\s*prometheus-node-exporter:\s*$"], + "forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"], + "why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure " + "rules that exist because of prod-1's etcd WAL starvation", + }, + { + "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)", + "prefixes": ("kubelet_",), + "file": KPS, + # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only + # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were. + "require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here + "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"], + "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three " + "pvc-capacity-alerts rules", + }, + { + "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules", + "prefixes": ("kube_velero_schedule_",), + "file": KPS, + "require": [ + r"customResourceState:\s*\n\s*enabled:\s*true", + r"metricNamePrefix:\s*kube_velero_schedule", + # The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the + # custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent, + # and the inert-alert failure mode all over again (measured; see the block's header). + r'apiGroups:\s*\["velero\.io"\]', + r'resources:\s*\["schedules"\]', + ], + "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"], + "why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER " + "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); " + "Velero itself publishes nothing until a backup is attempted", + }, + { + "owner": "kube-prometheus-stack: bundled kube-state-metrics", + "prefixes": ("kube_",), + "file": KPS, + "require": [r"^\s*kube-state-metrics:\s*$"], + "forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"], + # kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no + # custom node labels by default, so the metric does not exist unless metricLabelsAllowlist + # names the label. The prefix claim alone would pass while VPNTunnelDown's join matched + # nothing. + "why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule " + "rules", + }, + { + "owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)", + "exact": ("kube_node_labels",), + "file": KPS, + "require": [ + r"metricLabelsAllowlist:", + r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]", + ], + "forbid": [], + "why": "KSM exposes NO custom labels by default. Without the allowlist entry " + "kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently " + "matches nothing — a rule that is present, evaluating, and structurally inert", + }, + { + "owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)", + "prefixes": ("cnpg_",), + "file": "platform/keycloak/postgres-cluster.yaml", + "require": [r"^\s*enablePodMonitor:\s*true\s*$"], + "forbid": [], + "why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog " + "and no-recoverability-point rules that guard R-STATE-03", + }, + { + "owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)", + "prefixes": ("gotk_",), + "file": "platform/monitoring/flux-podmonitor.yaml", + "require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"], + "forbid": [], + "why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three " + "automated-update rules — on a platform whose entire delivery model is Flux", + }, + { + "owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease", + "prefixes": ("certmanager_",), + "file": "platform/cert-manager/helmrelease.yaml", + "require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"], + # The chart calls `fail` when both monitor kinds are enabled, which would break the whole + # HelmRelease rather than just the scrape. Measured against v1.21.2. + "forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"], + "why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only " + "warning that an ACME DNS-01 renewal of the live wildcard has quietly failed", + }, + { + "owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease", + "prefixes": ("longhorn_",), + "file": "platform/longhorn/helmrelease.yaml", + "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"], + "forbid": [], + "why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode " + "is a disk", + # The ServiceMonitor ALONE is not sufficient here — see the companion assertion below. + "companion": { + "file": "platform/longhorn/kustomization.yaml", + "require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"], + "why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies " + "(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on " + "networkPolicies.enabled) whose longhorn-manager policy admits only other " + "Longhorn components in the same namespace. MEASURED on prod: with the " + "ServiceMonitor applied and no allowance, all three targets were DISCOVERED and " + "every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not " + "a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml " + "the four storage alerts are inert even though the monitor exists", + }, + }, + { + "owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml", + "prefixes": ("kubevirt_",), + "file": "platform/kubevirt/servicemonitor.yaml", + "require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"], + "forbid": [], + "why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use " + "the kubevirt_cdi_* names, not bare cdi_*)", + }, + { + "owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease", + "prefixes": ("velero_",), + "file": "platform/velero/helmrelease.yaml", + "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"], + "forbid": [], + "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules " + "that failed to notice prod running with zero volume backups", + }, +] + +# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1) +# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4). +UNIVERSAL = {"up"} + +# ============================================================================================= +# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and +# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes. +# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert, +# or that gains an owner, FAILS as stale. +# ============================================================================================= +KNOWN_UNWIRED_METRICS: dict[str, str] = { + "hubble_drop_total": + "HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression " + "visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and " + "only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on " + "in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} " + "but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on " + "OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes " + "— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; " + "cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — " + "and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same " + "commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled " + "one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's " + "header in platform/monitoring/alerts.yaml.", +} + +# ============================================================================================= +# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor +# names an endpoint port that the Service it selects does not define, so the monitor discovers +# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the +# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks +# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered +# has no up series to be 0. +# +# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names +# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of +# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)` +# listed 20 jobs and none of these twelve appeared; all twelve `-down` rules returned no +# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent, +# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and +# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery` +# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter. +# ============================================================================================= +KNOWN_UNWIRED_JOBS: dict[str, str] = { + "gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000", + "jackett": "apps/wave-0/jackett.yaml — no exporter", + "pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter", + "plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint", + "qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter", + "vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter", + "keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / " + "keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names " + "neither, so it is a rename away from being live rather than an exporter away", + "nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)", + "radarr": "apps/wave-1/radarr.yaml — needs exportarr", + "semaphore": "apps/wave-1/semaphore.yaml — no exporter", + "sonarr": "apps/wave-1/sonarr.yaml — needs exportarr", + "seerr": "apps/wave-2/seerr.yaml — no exporter", +} + +# ============================================================================================= +# (0) PARSE EVERY ALERT RULE +# ============================================================================================= +# PromQL metric-name approximation. Documented as a gap in the header: string literals, +# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every +# remaining bare identifier that is not a PromQL keyword or function is treated as a metric +# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an +# approximate parser silently stops guarding. +PROMQL_KEYWORDS = set(""" +abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week +day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum +histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate +label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn +sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count +count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio +last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time +quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time +double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time +by without on ignoring group_left group_right offset bool and or unless if atan2 start end +inf nan NaN Inf +""".split()) + +IDENT_RE = re.compile(r"(? list[str]: + e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr) + e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e) + e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e) + e = re.sub(r"\{[^{}]*\}", " ", e) # label matchers + for _ in range(6): # nested aggregations + e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e) + out, seen = [], set() + for m in IDENT_RE.finditer(e): + n = m.group(1) + if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen: + continue + seen.add(n) + out.append(n) + return out + + +def is_golden_fixture(p: Path) -> bool: + """conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests. + + conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a + rendered `-down` PrometheusRule. They are the assertion data of another test, never + reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the + same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/ + fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/. + """ + parts = p.parts + return "conformance" in parts and "golden" in parts + + +rule_files = sorted( + p for p in ROOT.rglob("*.yaml") + if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p) +) +check(len(rule_files) > 0, + "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; " + "this check would silently audit nothing") + +# metric -> [ ":", ... ] ; and the same shape for up{job="X"} selectors +refs: dict[str, list[str]] = {} +job_refs: dict[str, list[str]] = {} +n_rules = 0 +for f in rule_files: + rel = str(f.relative_to(ROOT)) + try: + docs = list(yaml.safe_load_all(read(f))) + except yaml.YAMLError as exc: + check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited") + continue + for d in docs: + if not isinstance(d, dict) or d.get("kind") != "PrometheusRule": + continue + for grp in (d.get("spec") or {}).get("groups") or []: + for r in grp.get("rules") or []: + if "alert" not in r: + continue # recording rules: not this check's subject + n_rules += 1 + site = f"{rel}:{r['alert']}" + expr = str(r.get("expr", "")) + for n in metric_names(expr): + refs.setdefault(n, []).append(site) + for j in UP_JOB_RE.findall(expr): + job_refs.setdefault(j, []).append(site) + +check(n_rules > 0, + "no alerting rule parsed out of any PrometheusRule — the parse contract broke and this " + "check would pass vacuously while every alert's metric went unaudited") +check(len(refs) >= 20, + f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The " + f"PromQL approximation has stopped matching — which is exactly how an approximate parser " + f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it " + f"with a real parser") + +# ============================================================================================= +# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO +# ============================================================================================= +def resolve(metric: str) -> dict | None: + """Longest-prefix / exact-name match against SCRAPE_OWNERS.""" + best, best_len = None, -1 + for o in SCRAPE_OWNERS: + if metric in o.get("exact", ()): # exact names win outright + return o + for pfx in o.get("prefixes", ()): + if metric.startswith(pfx) and len(pfx) > best_len: + best, best_len = o, len(pfx) + return best + + +# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing +# at a deleted file is a claim nothing backs. +for o in SCRAPE_OWNERS: + check((ROOT / o["file"]).is_file(), + f"SCRAPE_OWNERS names {o['file']} as the scrape owner for " + f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the " + f"claim is unbacked and the alerts that depend on it are inert") + +claimed: dict[str, dict] = {} +unowned: list[str] = [] +for metric in sorted(refs): + if metric in UNIVERSAL: + continue + o = resolve(metric) + if o is None: + if metric in KNOWN_UNWIRED_METRICS: + continue # declared gap; ratcheted below + unowned.append(metric) + check(False, + f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is " + f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape " + f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no " + f"vector, stays silent forever, and is indistinguishable from a healthy result. " + f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — " + f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported " + f"instead of silent") + continue + check(metric not in KNOWN_UNWIRED_METRICS, + f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by " + f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is " + f"the one that shows up") + claimed[metric] = o + +# The owners that are actually load-bearing right now get their declarations verified. +for o in SCRAPE_OWNERS: + if o not in claimed.values(): + continue # nothing queries it today; nothing to guard + body = read(ROOT / o["file"]) + for rx in o.get("require", []): + check(re.search(rx, body, re.M) is not None, + f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. " + f"Removing it re-inerts every alert on " + f"{o.get('prefixes') or o.get('exact')} — {o['why']}") + for rx in o.get("forbid", []): + check(re.search(rx, body, re.M) is None, + f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} " + f"even though the enablement is still authored. {o['why']}") + comp = o.get("companion") + if comp: + cbody = read(ROOT / comp["file"]) + check((ROOT / comp["file"]).is_file(), + f"{comp['file']} is missing — {comp['why']}") + for rx in comp.get("require", []): + check(re.search(rx, cbody, re.M) is not None, + f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: " + f"{comp['why']}") + +# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the +# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off. +NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml" +if any(m.startswith("longhorn_") for m in refs): + np = read(NP) + check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its " + f"four alerts are inert") + check(re.search(r"kind:\s*NetworkPolicy", np) is not None, + "platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy") + check(re.search(r"app:\s*longhorn-manager", np) is not None, + "the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not " + "open the pods that serve /metrics") + check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None, + "the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus " + "still cannot reach longhorn-manager and the storage alerts stay inert") + check(re.search(r"port:\s*9500", np) is not None, + "the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` " + "port, which serves /metrics)") + # LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports. + check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None + and "Egress" not in np, + "the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the " + "scrape path and nothing else (R-SEC-02)") + # Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that + # they stay closed, and a substring search over prose would fail on the documentation of the + # very property it is asserting. + np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines()) + for closed in ("9502", "9503"): + check(closed not in np_spec, + f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / " + f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring " + f"is reach this observability path does not need (R-SEC-02)") + +# stale KNOWN_UNWIRED_METRICS entries +for m in sorted(KNOWN_UNWIRED_METRICS): + check(m in refs, + f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any " + f"more — a stale exemption hides the next real gap; delete the entry") + +# ============================================================================================= +# (4) up{job="X"} — the axis assertion (1) is structurally blind to. +# +# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/* +# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose +# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap). +# ============================================================================================= +def yaml_docs(path: Path) -> list[dict]: + try: + return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)] + except yaml.YAMLError: + return [] + + +# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define) +same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {} +for p in sorted(ROOT.rglob("*.yaml")): + if ".git" in p.parts: + continue + txt = read(p) + if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt: + continue + docs = yaml_docs(p) + monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"] + services = [d for d in docs if d.get("kind") == "Service"] + if not monitors or not services: + continue + svc_ports: set[str] = set() + for s in services: + for port in (s.get("spec") or {}).get("ports") or []: + if isinstance(port, dict) and port.get("name"): + svc_ports.add(str(port["name"])) + for m in monitors: + ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or [] + if isinstance(e, dict) and e.get("port")} + name = (m.get("metadata") or {}).get("name") + if name: + same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports) + +for job in sorted(job_refs): + sites = ", ".join(sorted(set(job_refs[job]))) + mon = same_file_monitors.get(job) + if mon is None: + # No same-file Service+ServiceMonitor pair is named for this job. It may well be produced + # by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all + # are), which this check cannot see — so this is NOT a failure, only unauditable. + continue + rel, ep_ports, svc_ports = mon + missing = sorted(ep_ports - svc_ports) + if missing: + check(job in KNOWN_UNWIRED_JOBS, + f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor " + f"scrapes port(s) {missing}, and the Service(s) in that same file define only " + f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO " + f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is " + f"structurally inert — note PrometheusTargetsMissing cannot see this either, because " + f"a target that was never discovered has no up series to be 0. Give the Service a " + f"named port the workload actually serves metrics on (and an exporter if it has " + f"none), or declare it in KNOWN_UNWIRED_JOBS") + else: + check(job not in KNOWN_UNWIRED_JOBS, + f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) " + f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: " + f"delete the entry") + +# stale KNOWN_UNWIRED_JOBS entries +for job in sorted(KNOWN_UNWIRED_JOBS): + check(job in job_refs, + f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any " + f"more — a stale exemption hides the next real gap; delete the entry") + +# ============================================================================================= +# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves — +# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used +# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and +# HONOURED by ci/l0.sh's --include-scope union. +# ============================================================================================= +SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo" +SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets" + / "chainsaw-test.yaml") +check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None, + "alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare " + "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a " + "change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and " + "platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a " + "chart's serviceMonitor.enabled back to false would be filtered out of exactly the change " + "that re-inerts the alerts") +L0 = read(ROOT / "ci" / "l0.sh") +GATE = read(ROOT / "ci" / "gate.sh") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a " + "purely requirement-scoped run again and the scope annotation above is inert decoration") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None + and re.search(r"fail\).*BLOCK=1", GATE) is not None, + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ======================================================================== report +owners_used = {id(o): o for o in claimed.values()} +print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} " + f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)") +print(f" claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}" + f" owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}") +for o in SCRAPE_OWNERS: + ms = sorted(m for m, oo in claimed.items() if oo is o) + mark = " " if ms else "· " + print(f" {mark}{o['file']:<46} {len(ms):>2} metric(s) {o['owner'][:64]}") +print(f"\n DECLARED GAPS (shrink-only registers):") +print(f" KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — " + f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}") +print(f" KNOWN_UNWIRED_JOBS: {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — " + f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}") +if unowned: + print(f"\n UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr) + +print() +if not FAILURES: + print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)") + raise SystemExit(0) +print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/platform/cert-manager/helmrelease.yaml b/platform/cert-manager/helmrelease.yaml index bafe554..3cbe222 100644 --- a/platform/cert-manager/helmrelease.yaml +++ b/platform/cert-manager/helmrelease.yaml @@ -31,3 +31,53 @@ spec: # cluster/split-horizon DNS (docs/09 split-horizon note; realises the TODO that was here). dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53" dns01RecursiveNameserversOnly: true + # --- SCRAPE THE TLS ISSUER, or every certificate alert is INERT (added 2026-09-13) ---------- + # platform/monitoring/alerts.yaml ships CertificateExpiringSoon, CertificateExpiryImminent + # (critical) and CertificateNotReady — and NONE of them could ever fire. The chart exposes the + # controller's metrics port by default (`prometheus.enabled: true`, confirmed on prod: + # Service/cert-manager listens on 9402) but defaults `prometheus.servicemonitor.enabled: false`, + # and no ServiceMonitor existed in ns cert-manager. VERIFIED live on prod 2026-09-13: + # `count({__name__=~"certmanager.*"})` returned NOTHING and `count by (job) (up)` listed no + # cert-manager job — so certmanager_certificate_expiration_timestamp_seconds and + # certmanager_certificate_ready_status were never in the TSDB. + # + # WHY THIS IS PART OF THE TLS PATH: the live wildcard `wildcard-home` (ns gateway) expires + # 2026-12-12T02:16:57Z and is renewed by an ACME DNS-01 order that can fail quietly — a Cloudflare + # token rotation, a DNS-01 self-check timeout (the very failure the two lines above work around), + # a rate limit. With no scrape, the FIRST signal of a failed renewal is a browser TLS error for + # every *.${BASE_DOMAIN} service. That is a user-discovered outage on a fully automated path, + # which is exactly what R-TLS-01/R-OBS-04 exist to prevent. + # + # Chart mechanics, verified by rendering cert-manager v1.21.2 (the version this range resolves to): + # * templates/servicemonitor.yaml renders on `prometheus.enabled AND + # prometheus.servicemonitor.enabled`. Measured: `helm template --set + # prometheus.servicemonitor.enabled=true` emits exactly 1 ServiceMonitor, 0 without it. + # `prometheus.enabled` is already true by chart default, so this one line is sufficient. + # * podmonitor and servicemonitor are MUTUALLY EXCLUSIVE — the chart calls `fail` if both are + # set. Do not add `prometheus.podmonitor.enabled` alongside this. + # * the rendered ServiceMonitor selects the controller/webhook/cainjector Services by + # app.kubernetes.io/{name,instance,component} on targetPort `http-metrics`, and sets + # `jobLabel: app.kubernetes.io/name` — so the scrape job lands as `cert-manager`. + # * it carries the label `prometheus: default` (servicemonitor.prometheusInstance). Harmless + # here: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues false, so Prometheus + # selects every ServiceMonitor regardless of labels. Left at the chart default deliberately — + # overriding it would imply a label-selecting Prometheus this cluster does not have. + # Cost: 3 scrape targets, one certmanager_* series set — negligible. + # + # PROVEN END TO END ON PROD, 2026-09-13 21:0xZ, by applying the rendered ServiceMonitor by hand + # (then removing it, so the merge's Helm upgrade owns it) — no NetworkPolicy was needed, as + # neither ns cert-manager nor ns monitoring carries one: + # up{namespace="cert-manager"} => 3 targets, all 1 + # (jobs cert-manager / cainjector / webhook, + # because jobLabel is app.kubernetes.io/name) + # count({__name__=~"certmanager.*"}) => 104 (was EMPTY) + # (certmanager_certificate_expiration_timestamp_seconds - time())/86400 + # => 89.2 days for `wildcard-home` (ns gateway, + # ClusterIssuer letsencrypt-prod) + # certmanager_certificate_ready_status{condition="True"} => 1 + # So the three Certificate* rules now evaluate against a real series and are correctly QUIET: + # the wildcard is healthy and 89 days out, which is exactly what CertificateExpiringSoon's 14-day + # threshold is meant to catch and could not have. + prometheus: + servicemonitor: + enabled: true diff --git a/platform/cert-manager/values.yaml b/platform/cert-manager/values.yaml index ddc0d08..bf949e3 100644 --- a/platform/cert-manager/values.yaml +++ b/platform/cert-manager/values.yaml @@ -4,3 +4,8 @@ # dns01RecursiveNameservers / -Only # TODO(WS-SEC): may be needed with split-horizon DNS # ClusterIssuers (letsencrypt-staging/-prod) live in clusterissuers.yaml, not here. # Cloudflare API token: SOPS-encrypted secret `cloudflare-api-token` (WS-SEC, OQ-4) — name only. +# prometheus.servicemonitor.enabled: true # scrape the controller's 9402 metrics port; without it +# # the certmanager_* family is absent from the TSDB and +# # the three Certificate* alert rules in +# # platform/monitoring/alerts.yaml are INERT. Mutually +# # exclusive with prometheus.podmonitor.enabled. diff --git a/platform/kubevirt/alerts.yaml b/platform/kubevirt/alerts.yaml index 03e9ac0..093659c 100644 --- a/platform/kubevirt/alerts.yaml +++ b/platform/kubevirt/alerts.yaml @@ -22,8 +22,22 @@ spec: rules: # Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any # sustained Failed VMI is page-worthy — the IaaS path is broken for that workload. + # + # METRIC CORRECTED 2026-09-13 (was `kubevirt_vmi_phase_count{phase="Failed"}`). That series + # DOES NOT EXIST on the deployed KubeVirt, so this rule was structurally INERT — unlike the + # Longhorn/cert-manager/Flux gaps in the same audit, NOT for want of a scrape target: the + # `kubevirt-prometheus-metrics` job is up and its siblings below (virt_api_ready_status, + # virt_controller_ready_status, cdi_import_pods_high_restart) all resolve. KubeVirt simply + # retired the phase-count gauge in favour of an info-style series carrying `phase` as a label. + # VERIFIED live on prod 2026-09-13: + # count(kubevirt_vmi_phase_count) => (empty) + # count(kubevirt_vmi_info) => 1 + # kubevirt_vmi_info{...,name="unifi",namespace="unifi",phase="running",...} => 1 + # Note `phase="running"` — KubeVirt LOWERCASES the phase in this label, so the matcher is + # "failed", not "Failed". A capital-F matcher here would silently select nothing and leave the + # rule just as inert as the metric name did; that is the whole point of this class of defect. - alert: KubeVirtVMLaunchFailing - expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0 + expr: sum by (namespace) (kubevirt_vmi_info{phase="failed"}) > 0 for: 5m labels: severity: critical diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index 8eac06d..a9e58d6 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -102,6 +102,54 @@ spec: # Reclaim replica space of deleted/failed volumes promptly in dev. orphanAutoDeletion: true + # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ---------- + # platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical), + # LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh — + # and NONE of them could ever fire, because the chart defaults + # `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system. + # VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn; + # `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING. + # So the entire longhorn_* family was absent from the TSDB and four alert rules were decoration. + # + # WHY THIS IS PART OF THE STORAGE PATH, NOT AN OBSERVABILITY NICE-TO-HAVE: this cluster's + # recurring failure mode is a disk. prod-1's failing SSD starved etcd's WAL fsync and wedged the + # node (DECISIONS-LOG; the prod-1 drive replacement), and a degraded/faulted replica is precisely + # the first symptom Longhorn publishes. R-STATE-02 promises 3-way replication survives a node + # loss; with no scrape, a volume can sit at 2 replicas — or faulted — indefinitely, unalerted. + # platform/monitoring/pvc-capacity-alerts.yaml's header already flagged this exact gap as "a + # separate defect (a missing Longhorn ServiceMonitor/metrics config)". This closes it. + # + # Chart mechanics, verified by rendering longhorn 1.12.1 (the version this range resolves to): + # * templates/servicemonitor.yaml is gated on `{{- if .Values.metrics.serviceMonitor.enabled }}` + # and NOTHING else — no `.Capabilities` probe. Measured: `helm template --set + # metrics.serviceMonitor.enabled=true` emits exactly 1 ServiceMonitor, and 0 without the flag. + # So this line is both necessary and sufficient; unlike velero's, it needs no API capability. + # * the rendered ServiceMonitor selects `app: longhorn-manager` in the release namespace on the + # port named `manager`. CONFIRMED against prod: Service/longhorn-backend carries + # `app: longhorn-manager` and a port `manager` (9500) — the selector resolves to real + # endpoints, it is not a dangling monitor (which is how the apps/ SLI monitors fail: they name + # a `metrics` port their Service does not have, so they discover zero targets and their + # `up{job=...}` alerts are inert in the very same way). + # * no `additionalLabels` needed: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues + # false, so Prometheus selects every ServiceMonitor regardless of labels. + # Cost: one scrape target per manager pod (3), the longhorn_* families — low hundreds of series + # on this cluster's volume count. + # + # PROVEN END TO END ON PROD, 2026-09-13 21:0xZ. The rendered ServiceMonitor was applied to prod + # by hand (then removed again, so the merge's Helm upgrade creates and OWNS it rather than + # colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place: + # count(up{job="longhorn-backend"}) => 3 (all three managers, up==1) + # count({__name__=~"longhorn.*"}) => 8717 (was EMPTY) + # count(longhorn_volume_robustness) => 44 + # count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached) + # and the four alert rules went from "no series, cannot fire" to evaluating for real — one of + # them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB + # volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming + # freed blocks. Reported, NOT silenced by moving the threshold). + metrics: + serviceMonitor: + enabled: true + # Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind # oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here. diff --git a/platform/longhorn/kustomization.yaml b/platform/longhorn/kustomization.yaml index a567a3c..2e44244 100644 --- a/platform/longhorn/kustomization.yaml +++ b/platform/longhorn/kustomization.yaml @@ -5,4 +5,10 @@ resources: - namespace.yaml - helmrepository.yaml - helmrelease.yaml + # Opens ns monitoring -> longhorn-manager:9500 so Prometheus can actually scrape the metrics the + # HelmRelease's `metrics.serviceMonitor.enabled` exposes. Longhorn 1.12.1's own (effectively + # unconditional) NetworkPolicies drop that path, so WITHOUT this the ServiceMonitor discovers 3 + # targets that all sit at up==0 and the Longhorn alerts stay inert. See the file header for the + # measured proof. (2026-09-13, from the inert-alert audit.) + - networkpolicy-metrics.yaml - storageclasses.yaml diff --git a/platform/longhorn/networkpolicy-metrics.yaml b/platform/longhorn/networkpolicy-metrics.yaml new file mode 100644 index 0000000..6431df6 --- /dev/null +++ b/platform/longhorn/networkpolicy-metrics.yaml @@ -0,0 +1,76 @@ +--- +# Allow Prometheus to reach longhorn-manager's metrics port (R-OBS-04, R-STATE-02). +# +# WITHOUT THIS FILE, `metrics.serviceMonitor.enabled: true` IN helmrelease.yaml IS NOT ENOUGH and the +# four Longhorn alert rules in platform/monitoring/alerts.yaml STAY INERT. This was measured, not +# predicted — prod, 2026-09-13, with the ServiceMonitor in place: +# +# count by (job) (up) => {job="longhorn-backend"} => 3 (targets DISCOVERED) +# up{job="longhorn-backend"} => 0, 0, 0 (every scrape FAILING) +# count(longhorn_volume_robustness) => (empty) (still nothing in the TSDB) +# /api/v1/targets lastError => 'Get "http://10.0.0.169:9500/metrics": context deadline +# exceeded' (10.0007s — a DROP, not a refusal) +# +# The endpoint itself is healthy and fast; only the path from ns monitoring is blocked. Proven by +# curling it from inside the manager pod: +# http://:9500/metrics -> 200 in 0.031s (the manager binds the POD IP, not localhost — +# http://localhost:9500/metrics -> 000 which is why a localhost probe misleads here) +# +# WHY IT IS BLOCKED: longhorn 1.12.1 ships SIX NetworkPolicies of its own, and the one selecting +# `app: longhorn-manager` is Ingress-only with a from-list of nothing but other Longhorn components +# (longhorn-manager, longhorn-ui, longhorn-csi-plugin, the recurring-job/job-task pods, +# longhorn-driver-deployer) — all bare podSelectors, i.e. same-namespace. Prometheus lives in ns +# monitoring, so it matches no clause and its packets are dropped. +# +# THE CHART OFFERS NO KNOB FOR THIS, and the obvious-looking one is a trap. `networkPolicies.enabled` +# (default false) does NOT gate these templates — verified by rendering 1.12.1: `helm template` emits +# the same 6 NetworkPolicies with the flag on AND off, because templates/network-policies/* are +# guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) instead. So the policies are +# effectively unconditional, and nothing in the chart adds a Prometheus allowance when +# metrics.serviceMonitor.enabled is set. +# +# WHY NOT `networkPolicies.restrictInternalTraffic: false`: that is the one-line fix and it is the +# wrong one. It deletes Longhorn's entire internal-traffic restriction across all six components to +# open one port to one namespace — a straight R-SEC-02 (least network reach) regression, trading a +# security control for an observability one when the two are not in conflict. +# +# THIS IS ADDITIVE AND NARROW. Kubernetes NetworkPolicies are OR'd (Cilium enforces the UNION — see +# platform/headlamp/networkpolicy.yaml's header), so this grants exactly one new path on top of +# Longhorn's floor: ns monitoring -> app=longhorn-manager, TCP 9500 only. Longhorn's own policy is +# left untouched, every other component stays restricted, and nothing else in longhorn-system becomes +# reachable. +# +# TIER: a core networking.k8s.io kind with no CRD dependency, so it belongs to the `platform` +# Kustomization alongside the HelmRelease (referenced from platform/longhorn/kustomization.yaml) — +# NOT platform-config, which exists for CRs of platform-installed CRDs. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: longhorn-manager-allow-prometheus-scrape + namespace: longhorn-system + labels: + app.kubernetes.io/name: longhorn + # Marks this as an observability path, not a data path — so a future reviewer trimming Longhorn + # network reach can see at a glance what breaks (the storage alerts) if they delete it. + homelab.keithjackson.dev/purpose: metrics-scrape +spec: + podSelector: + matchLabels: + app: longhorn-manager + policyTypes: + - Ingress + ingress: + - from: + # The whole monitoring namespace rather than the Prometheus pod labels: the operator owns + # those labels and rewrites them on every kube-prometheus-stack bump (they carry the chart's + # instance name and a controller-revision hash). A pod-label selector here would silently + # stop matching on an upgrade and put the storage alerts back to inert — the exact regression + # class this whole change exists to close. The namespace holds only the monitoring stack. + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - protocol: TCP + # longhorn-manager's container port `manager` (Service longhorn-backend:9500), which serves + # /metrics. NOT 9502/9503 (admission webhook / recovery backend) — those stay closed. + port: 9500 diff --git a/platform/longhorn/values.yaml b/platform/longhorn/values.yaml index 93728bc..f6f318d 100644 --- a/platform/longhorn/values.yaml +++ b/platform/longhorn/values.yaml @@ -22,3 +22,10 @@ # defaultDataLocality: disabled # keep replicas spread for durability # orphanAutoDeletion: true # reclaim orphaned replica space promptly # backupTarget: # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003) +# +# metrics: +# serviceMonitor: +# enabled: true # scrape longhorn-manager (port `manager`, 9500); +# # without it the whole longhorn_* family is absent +# # from the TSDB and the four Longhorn alert rules +# # in platform/monitoring/alerts.yaml are INERT diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index af66d5e..25aa82c 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -333,6 +333,47 @@ spec: # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown` # above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath # health is covered by the Cilium platform alerts. + # *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a + # *** ServiceMonitor: a monitor alone cannot make this rule fire. + # Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets + # (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a + # stale entry there FAILS the gate, so this cannot quietly become fiction. + # + # VERIFIED live on prod 2026-09-13: + # count(hubble_drop_total) => (empty) + # count({__name__=~"hubble.*"}) => (empty) + # count({__name__=~"cilium.*"}) => (empty) + # count by (job) (up) => 20 jobs, none Cilium or Hubble + # (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading) + # So the rule evaluates `rate()` and returns no series: it cannot fire, and the + # platform has NO policy-drop visibility at all today. + # + # WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve + # /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is + # switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets + # `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not + # `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning + # them on means: + # * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`, + # which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all + # three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster: + # the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its + # OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius. + # * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml + # carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true` + # against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes + # a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is + # invisible until the next node rebuild. + # * per-flow label cardinality: `drop` metrics carry source/destination context, and the + # default contexts can multiply series across every pod pair that gets denied — on a + # default-deny cluster, denials are routine, not exceptional. + # RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change + # on the cilium Kustomization, with the Ansible parity edit in the same commit, the + # labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`), + # and a one-node canary before the full DaemonSet roll. Until then this alert is honestly + # inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security + # audit), which also means a NetworkPolicy change cannot be observed to have started dropping + # legitimate traffic. - alert: HubbleDeniedFlowSpike # Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03). expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5