feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
f4f6e5f7a3
commit
a92fa11701
10 files changed
+1069
-1
No files matched your search
@@ -31,3 +31,53 @@ spec:
|
||||
# cluster/split-horizon DNS (docs/09 split-horizon note; realises the TODO that was here).
|
||||
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
||||
dns01RecursiveNameserversOnly: true
|
||||
# --- SCRAPE THE TLS ISSUER, or every certificate alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships CertificateExpiringSoon, CertificateExpiryImminent
|
||||
# (critical) and CertificateNotReady — and NONE of them could ever fire. The chart exposes the
|
||||
# controller's metrics port by default (`prometheus.enabled: true`, confirmed on prod:
|
||||
# Service/cert-manager listens on 9402) but defaults `prometheus.servicemonitor.enabled: false`,
|
||||
# and no ServiceMonitor existed in ns cert-manager. VERIFIED live on prod 2026-09-13:
|
||||
# `count({__name__=~"certmanager.*"})` returned NOTHING and `count by (job) (up)` listed no
|
||||
# cert-manager job — so certmanager_certificate_expiration_timestamp_seconds and
|
||||
# certmanager_certificate_ready_status were never in the TSDB.
|
||||
#
|
||||
# WHY THIS IS PART OF THE TLS PATH: the live wildcard `wildcard-home` (ns gateway) expires
|
||||
# 2026-12-12T02:16:57Z and is renewed by an ACME DNS-01 order that can fail quietly — a Cloudflare
|
||||
# token rotation, a DNS-01 self-check timeout (the very failure the two lines above work around),
|
||||
# a rate limit. With no scrape, the FIRST signal of a failed renewal is a browser TLS error for
|
||||
# every *.${BASE_DOMAIN} service. That is a user-discovered outage on a fully automated path,
|
||||
# which is exactly what R-TLS-01/R-OBS-04 exist to prevent.
|
||||
#
|
||||
# Chart mechanics, verified by rendering cert-manager v1.21.2 (the version this range resolves to):
|
||||
# * templates/servicemonitor.yaml renders on `prometheus.enabled AND
|
||||
# prometheus.servicemonitor.enabled`. Measured: `helm template --set
|
||||
# prometheus.servicemonitor.enabled=true` emits exactly 1 ServiceMonitor, 0 without it.
|
||||
# `prometheus.enabled` is already true by chart default, so this one line is sufficient.
|
||||
# * podmonitor and servicemonitor are MUTUALLY EXCLUSIVE — the chart calls `fail` if both are
|
||||
# set. Do not add `prometheus.podmonitor.enabled` alongside this.
|
||||
# * the rendered ServiceMonitor selects the controller/webhook/cainjector Services by
|
||||
# app.kubernetes.io/{name,instance,component} on targetPort `http-metrics`, and sets
|
||||
# `jobLabel: app.kubernetes.io/name` — so the scrape job lands as `cert-manager`.
|
||||
# * it carries the label `prometheus: default` (servicemonitor.prometheusInstance). Harmless
|
||||
# here: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues false, so Prometheus
|
||||
# selects every ServiceMonitor regardless of labels. Left at the chart default deliberately —
|
||||
# overriding it would imply a label-selecting Prometheus this cluster does not have.
|
||||
# Cost: 3 scrape targets, one certmanager_* series set — negligible.
|
||||
#
|
||||
# PROVEN END TO END ON PROD, 2026-09-13 21:0xZ, by applying the rendered ServiceMonitor by hand
|
||||
# (then removing it, so the merge's Helm upgrade owns it) — no NetworkPolicy was needed, as
|
||||
# neither ns cert-manager nor ns monitoring carries one:
|
||||
# up{namespace="cert-manager"} => 3 targets, all 1
|
||||
# (jobs cert-manager / cainjector / webhook,
|
||||
# because jobLabel is app.kubernetes.io/name)
|
||||
# count({__name__=~"certmanager.*"}) => 104 (was EMPTY)
|
||||
# (certmanager_certificate_expiration_timestamp_seconds - time())/86400
|
||||
# => 89.2 days for `wildcard-home` (ns gateway,
|
||||
# ClusterIssuer letsencrypt-prod)
|
||||
# certmanager_certificate_ready_status{condition="True"} => 1
|
||||
# So the three Certificate* rules now evaluate against a real series and are correctly QUIET:
|
||||
# the wildcard is healthy and 89 days out, which is exactly what CertificateExpiringSoon's 14-day
|
||||
# threshold is meant to catch and could not have.
|
||||
prometheus:
|
||||
servicemonitor:
|
||||
enabled: true
|
||||
@@ -4,3 +4,8 @@
|
||||
# dns01RecursiveNameservers / -Only # TODO(WS-SEC): may be needed with split-horizon DNS
|
||||
# ClusterIssuers (letsencrypt-staging/-prod) live in clusterissuers.yaml, not here.
|
||||
# Cloudflare API token: SOPS-encrypted secret `cloudflare-api-token` (WS-SEC, OQ-4) — name only.
|
||||
# prometheus.servicemonitor.enabled: true # scrape the controller's 9402 metrics port; without it
|
||||
# # the certmanager_* family is absent from the TSDB and
|
||||
# # the three Certificate* alert rules in
|
||||
# # platform/monitoring/alerts.yaml are INERT. Mutually
|
||||
# # exclusive with prometheus.podmonitor.enabled.
|
||||
@@ -22,8 +22,22 @@ spec:
|
||||
rules:
|
||||
# Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
|
||||
# sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
|
||||
#
|
||||
# METRIC CORRECTED 2026-09-13 (was `kubevirt_vmi_phase_count{phase="Failed"}`). That series
|
||||
# DOES NOT EXIST on the deployed KubeVirt, so this rule was structurally INERT — unlike the
|
||||
# Longhorn/cert-manager/Flux gaps in the same audit, NOT for want of a scrape target: the
|
||||
# `kubevirt-prometheus-metrics` job is up and its siblings below (virt_api_ready_status,
|
||||
# virt_controller_ready_status, cdi_import_pods_high_restart) all resolve. KubeVirt simply
|
||||
# retired the phase-count gauge in favour of an info-style series carrying `phase` as a label.
|
||||
# VERIFIED live on prod 2026-09-13:
|
||||
# count(kubevirt_vmi_phase_count) => (empty)
|
||||
# count(kubevirt_vmi_info) => 1
|
||||
# kubevirt_vmi_info{...,name="unifi",namespace="unifi",phase="running",...} => 1
|
||||
# Note `phase="running"` — KubeVirt LOWERCASES the phase in this label, so the matcher is
|
||||
# "failed", not "Failed". A capital-F matcher here would silently select nothing and leave the
|
||||
# rule just as inert as the metric name did; that is the whole point of this class of defect.
|
||||
- alert: KubeVirtVMLaunchFailing
|
||||
expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
|
||||
expr: sum by (namespace) (kubevirt_vmi_info{phase="failed"}) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
|
||||
@@ -102,6 +102,54 @@ spec:
|
||||
# Reclaim replica space of deleted/failed volumes promptly in dev.
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
|
||||
# LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
|
||||
# and NONE of them could ever fire, because the chart defaults
|
||||
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
||||
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
||||
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
||||
# So the entire longhorn_* family was absent from the TSDB and four alert rules were decoration.
|
||||
#
|
||||
# WHY THIS IS PART OF THE STORAGE PATH, NOT AN OBSERVABILITY NICE-TO-HAVE: this cluster's
|
||||
# recurring failure mode is a disk. prod-1's failing SSD starved etcd's WAL fsync and wedged the
|
||||
# node (DECISIONS-LOG; the prod-1 drive replacement), and a degraded/faulted replica is precisely
|
||||
# the first symptom Longhorn publishes. R-STATE-02 promises 3-way replication survives a node
|
||||
# loss; with no scrape, a volume can sit at 2 replicas — or faulted — indefinitely, unalerted.
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml's header already flagged this exact gap as "a
|
||||
# separate defect (a missing Longhorn ServiceMonitor/metrics config)". This closes it.
|
||||
#
|
||||
# Chart mechanics, verified by rendering longhorn 1.12.1 (the version this range resolves to):
|
||||
# * templates/servicemonitor.yaml is gated on `{{- if .Values.metrics.serviceMonitor.enabled }}`
|
||||
# and NOTHING else — no `.Capabilities` probe. Measured: `helm template --set
|
||||
# metrics.serviceMonitor.enabled=true` emits exactly 1 ServiceMonitor, and 0 without the flag.
|
||||
# So this line is both necessary and sufficient; unlike velero's, it needs no API capability.
|
||||
# * the rendered ServiceMonitor selects `app: longhorn-manager` in the release namespace on the
|
||||
# port named `manager`. CONFIRMED against prod: Service/longhorn-backend carries
|
||||
# `app: longhorn-manager` and a port `manager` (9500) — the selector resolves to real
|
||||
# endpoints, it is not a dangling monitor (which is how the apps/ SLI monitors fail: they name
|
||||
# a `metrics` port their Service does not have, so they discover zero targets and their
|
||||
# `up{job=...}` alerts are inert in the very same way).
|
||||
# * no `additionalLabels` needed: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues
|
||||
# false, so Prometheus selects every ServiceMonitor regardless of labels.
|
||||
# Cost: one scrape target per manager pod (3), the longhorn_* families — low hundreds of series
|
||||
# on this cluster's volume count.
|
||||
#
|
||||
# PROVEN END TO END ON PROD, 2026-09-13 21:0xZ. The rendered ServiceMonitor was applied to prod
|
||||
# by hand (then removed again, so the merge's Helm upgrade creates and OWNS it rather than
|
||||
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
||||
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
||||
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
||||
# count(longhorn_volume_robustness) => 44
|
||||
# count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached)
|
||||
# and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
|
||||
# them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
|
||||
# volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
|
||||
# freed blocks. Reported, NOT silenced by moving the threshold).
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
|
||||
# Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
|
||||
# oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.
|
||||
|
||||
|
||||
@@ -5,4 +5,10 @@ resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml
|
||||
# Opens ns monitoring -> longhorn-manager:9500 so Prometheus can actually scrape the metrics the
|
||||
# HelmRelease's `metrics.serviceMonitor.enabled` exposes. Longhorn 1.12.1's own (effectively
|
||||
# unconditional) NetworkPolicies drop that path, so WITHOUT this the ServiceMonitor discovers 3
|
||||
# targets that all sit at up==0 and the Longhorn alerts stay inert. See the file header for the
|
||||
# measured proof. (2026-09-13, from the inert-alert audit.)
|
||||
- networkpolicy-metrics.yaml
|
||||
- storageclasses.yaml
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
# Allow Prometheus to reach longhorn-manager's metrics port (R-OBS-04, R-STATE-02).
|
||||
#
|
||||
# WITHOUT THIS FILE, `metrics.serviceMonitor.enabled: true` IN helmrelease.yaml IS NOT ENOUGH and the
|
||||
# four Longhorn alert rules in platform/monitoring/alerts.yaml STAY INERT. This was measured, not
|
||||
# predicted — prod, 2026-09-13, with the ServiceMonitor in place:
|
||||
#
|
||||
# count by (job) (up) => {job="longhorn-backend"} => 3 (targets DISCOVERED)
|
||||
# up{job="longhorn-backend"} => 0, 0, 0 (every scrape FAILING)
|
||||
# count(longhorn_volume_robustness) => (empty) (still nothing in the TSDB)
|
||||
# /api/v1/targets lastError => 'Get "http://10.0.0.169:9500/metrics": context deadline
|
||||
# exceeded' (10.0007s — a DROP, not a refusal)
|
||||
#
|
||||
# The endpoint itself is healthy and fast; only the path from ns monitoring is blocked. Proven by
|
||||
# curling it from inside the manager pod:
|
||||
# http://<podIP>:9500/metrics -> 200 in 0.031s (the manager binds the POD IP, not localhost —
|
||||
# http://localhost:9500/metrics -> 000 which is why a localhost probe misleads here)
|
||||
#
|
||||
# WHY IT IS BLOCKED: longhorn 1.12.1 ships SIX NetworkPolicies of its own, and the one selecting
|
||||
# `app: longhorn-manager` is Ingress-only with a from-list of nothing but other Longhorn components
|
||||
# (longhorn-manager, longhorn-ui, longhorn-csi-plugin, the recurring-job/job-task pods,
|
||||
# longhorn-driver-deployer) — all bare podSelectors, i.e. same-namespace. Prometheus lives in ns
|
||||
# monitoring, so it matches no clause and its packets are dropped.
|
||||
#
|
||||
# THE CHART OFFERS NO KNOB FOR THIS, and the obvious-looking one is a trap. `networkPolicies.enabled`
|
||||
# (default false) does NOT gate these templates — verified by rendering 1.12.1: `helm template` emits
|
||||
# the same 6 NetworkPolicies with the flag on AND off, because templates/network-policies/* are
|
||||
# guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) instead. So the policies are
|
||||
# effectively unconditional, and nothing in the chart adds a Prometheus allowance when
|
||||
# metrics.serviceMonitor.enabled is set.
|
||||
#
|
||||
# WHY NOT `networkPolicies.restrictInternalTraffic: false`: that is the one-line fix and it is the
|
||||
# wrong one. It deletes Longhorn's entire internal-traffic restriction across all six components to
|
||||
# open one port to one namespace — a straight R-SEC-02 (least network reach) regression, trading a
|
||||
# security control for an observability one when the two are not in conflict.
|
||||
#
|
||||
# THIS IS ADDITIVE AND NARROW. Kubernetes NetworkPolicies are OR'd (Cilium enforces the UNION — see
|
||||
# platform/headlamp/networkpolicy.yaml's header), so this grants exactly one new path on top of
|
||||
# Longhorn's floor: ns monitoring -> app=longhorn-manager, TCP 9500 only. Longhorn's own policy is
|
||||
# left untouched, every other component stays restricted, and nothing else in longhorn-system becomes
|
||||
# reachable.
|
||||
#
|
||||
# TIER: a core networking.k8s.io kind with no CRD dependency, so it belongs to the `platform`
|
||||
# Kustomization alongside the HelmRelease (referenced from platform/longhorn/kustomization.yaml) —
|
||||
# NOT platform-config, which exists for CRs of platform-installed CRDs.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: longhorn-manager-allow-prometheus-scrape
|
||||
namespace: longhorn-system
|
||||
labels:
|
||||
app.kubernetes.io/name: longhorn
|
||||
# Marks this as an observability path, not a data path — so a future reviewer trimming Longhorn
|
||||
# network reach can see at a glance what breaks (the storage alerts) if they delete it.
|
||||
homelab.keithjackson.dev/purpose: metrics-scrape
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: longhorn-manager
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
# The whole monitoring namespace rather than the Prometheus pod labels: the operator owns
|
||||
# those labels and rewrites them on every kube-prometheus-stack bump (they carry the chart's
|
||||
# instance name and a controller-revision hash). A pod-label selector here would silently
|
||||
# stop matching on an upgrade and put the storage alerts back to inert — the exact regression
|
||||
# class this whole change exists to close. The namespace holds only the monitoring stack.
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
# longhorn-manager's container port `manager` (Service longhorn-backend:9500), which serves
|
||||
# /metrics. NOT 9502/9503 (admission webhook / recovery backend) — those stay closed.
|
||||
port: 9500
|
||||
@@ -22,3 +22,10 @@
|
||||
# defaultDataLocality: disabled # keep replicas spread for durability
|
||||
# orphanAutoDeletion: true # reclaim orphaned replica space promptly
|
||||
# backupTarget: <off-cluster> # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
|
||||
#
|
||||
# metrics:
|
||||
# serviceMonitor:
|
||||
# enabled: true # scrape longhorn-manager (port `manager`, 9500);
|
||||
# # without it the whole longhorn_* family is absent
|
||||
# # from the TSDB and the four Longhorn alert rules
|
||||
# # in platform/monitoring/alerts.yaml are INERT
|
||||
@@ -333,6 +333,47 @@ spec:
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
||||
# health is covered by the Cilium platform alerts.
|
||||
# *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
|
||||
# *** ServiceMonitor: a monitor alone cannot make this rule fire.
|
||||
# Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
|
||||
# (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
|
||||
# stale entry there FAILS the gate, so this cannot quietly become fiction.
|
||||
#
|
||||
# VERIFIED live on prod 2026-09-13:
|
||||
# count(hubble_drop_total) => (empty)
|
||||
# count({__name__=~"hubble.*"}) => (empty)
|
||||
# count({__name__=~"cilium.*"}) => (empty)
|
||||
# count by (job) (up) => 20 jobs, none Cilium or Hubble
|
||||
# (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
|
||||
# So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
|
||||
# platform has NO policy-drop visibility at all today.
|
||||
#
|
||||
# WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
|
||||
# /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
|
||||
# switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
|
||||
# `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
|
||||
# `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
|
||||
# them on means:
|
||||
# * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
|
||||
# which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
|
||||
# three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
|
||||
# the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
|
||||
# OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
|
||||
# * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
|
||||
# carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
|
||||
# against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
|
||||
# a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
|
||||
# invisible until the next node rebuild.
|
||||
# * per-flow label cardinality: `drop` metrics carry source/destination context, and the
|
||||
# default contexts can multiply series across every pod pair that gets denied — on a
|
||||
# default-deny cluster, denials are routine, not exceptional.
|
||||
# RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
|
||||
# on the cilium Kustomization, with the Ansible parity edit in the same commit, the
|
||||
# labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
|
||||
# and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
|
||||
# inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
|
||||
# audit), which also means a NetworkPolicy change cannot be observed to have started dropping
|
||||
# legitimate traffic.
|
||||
- alert: HubbleDeniedFlowSpike
|
||||
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
||||
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
||||
|
||||
Reference in new issue
Block a user