fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
72f0ba4e1f
commit
4687521d65
11 files changed
+913
-113
No files matched your search
@@ -103,9 +103,9 @@ spec:
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
|
||||
# LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
|
||||
# and NONE of them could ever fire, because the chart defaults
|
||||
# platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
|
||||
# LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
|
||||
# LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
|
||||
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
||||
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
||||
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
||||
@@ -140,12 +140,26 @@ spec:
|
||||
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
||||
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
||||
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
||||
# count(longhorn_volume_robustness) => 44
|
||||
# count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached)
|
||||
# and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
|
||||
# them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
|
||||
# volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
|
||||
# freed blocks. Reported, NOT silenced by moving the threshold).
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 `state` labels)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
|
||||
# "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
|
||||
# metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
|
||||
# `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
|
||||
# zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
|
||||
# volumes. The consequence was that enabling this scrape did NOT un-inert
|
||||
# LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
|
||||
# Fixed in platform/monitoring/alerts.yaml and pinned by
|
||||
# platform/monitoring/tests/longhorn.test.yaml.
|
||||
#
|
||||
# Of the rules that WERE evaluating for real, one immediately found a true positive and one
|
||||
# permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
|
||||
# Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
|
||||
# not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
|
||||
# replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
|
||||
# it measured allocation and called it capacity, the filesystem question is already answered by
|
||||
# kubelet_volume_stats_*, and the allocation question now lives per disk as
|
||||
# LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
|
||||
+116
-12
@@ -147,9 +147,33 @@ spec:
|
||||
groups:
|
||||
- name: platform-health.longhorn
|
||||
rules:
|
||||
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
|
||||
# -------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
||||
#
|
||||
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
||||
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
||||
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
||||
# label and its value is only ever 0 or 1:
|
||||
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
||||
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
||||
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# count(longhorn_volume_robustness == 2) => (empty)
|
||||
# count(longhorn_volume_robustness == 3) => (empty)
|
||||
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
||||
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
||||
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
||||
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
||||
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
||||
#
|
||||
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
||||
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornVolumeDegraded
|
||||
expr: longhorn_volume_robustness == 2
|
||||
expr: longhorn_volume_robustness{state="degraded"} == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -158,7 +182,7 @@ spec:
|
||||
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
- alert: LonghornVolumeFaulted
|
||||
expr: longhorn_volume_robustness == 3
|
||||
expr: longhorn_volume_robustness{state="faulted"} == 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -166,23 +190,103 @@ spec:
|
||||
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- alert: LonghornVolumeCapacityHigh
|
||||
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
|
||||
# -------------------------------------------------------------------------------------
|
||||
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
||||
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
||||
#
|
||||
# The deleted rule was
|
||||
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
||||
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
||||
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
||||
# MEASURED on prod 2026-09-13:
|
||||
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
||||
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
||||
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
||||
# ratio 0.9731 <- what the alert saw
|
||||
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
||||
# the SAME filesystem, from the kubelet:
|
||||
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
||||
#
|
||||
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
||||
# every snapshot still pinned. On this volume,
|
||||
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
||||
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
||||
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
||||
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
||||
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
||||
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
||||
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
||||
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
||||
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
||||
# to ignore the storage row, which is the row that matters on this cluster.
|
||||
#
|
||||
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
||||
# asserted:
|
||||
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
||||
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
||||
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
||||
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
||||
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
||||
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
||||
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
||||
# Block volumes are covered at the pool level, below.
|
||||
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
||||
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
||||
# against its own nominal size, where it is a tautology.
|
||||
# -------------------------------------------------------------------------------------
|
||||
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
||||
#
|
||||
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
||||
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
||||
#
|
||||
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
||||
# replica files plus their pinned snapshots — because that is what consumes the node's
|
||||
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
||||
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
||||
# "disk allocation high" says what it measures.
|
||||
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
||||
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
||||
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
||||
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
||||
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
||||
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
||||
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
||||
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
||||
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
||||
# gives lead time on the way to Longhorn's own limit.
|
||||
#
|
||||
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
||||
# node capacity usage ratio scheduled (nominal)
|
||||
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
||||
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
||||
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornDiskAllocationHigh
|
||||
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
|
||||
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- alert: LonghornNodeStorageHigh
|
||||
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
|
||||
for: 30m
|
||||
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
||||
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
||||
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
||||
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
||||
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
||||
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
||||
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
||||
# a longhorn-manager restarts.
|
||||
- alert: LonghornDiskUnschedulable
|
||||
expr: longhorn_disk_status{condition="schedulable"} == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
|
||||
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
|
||||
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
@@ -10,12 +10,26 @@
|
||||
# the SSO database with nothing watching it.
|
||||
#
|
||||
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
|
||||
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
|
||||
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
|
||||
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
|
||||
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
|
||||
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
|
||||
# working on both clusters.
|
||||
# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two
|
||||
# different reasons in sequence, and it is now GONE:
|
||||
# * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified
|
||||
# read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule
|
||||
# group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in
|
||||
# platform/longhorn/helmrelease.yaml.
|
||||
# * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It
|
||||
# divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head
|
||||
# plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and
|
||||
# called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet
|
||||
# read the same filesystem 20.8% used — by design and permanently, because `retentionSize:
|
||||
# 45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the
|
||||
# comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now
|
||||
# lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to
|
||||
# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`.
|
||||
# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in
|
||||
# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in
|
||||
# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no
|
||||
# filesystem for anyone to measure (see the inodes guard below).
|
||||
#
|
||||
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
|
||||
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
|
||||
|
||||
@@ -0,0 +1,465 @@
|
||||
# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml)
|
||||
# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity"
|
||||
# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file,
|
||||
# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml).
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THIS FILE IS FOR
|
||||
# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for
|
||||
# the first time. Two things became visible within minutes:
|
||||
# 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the
|
||||
# kubelet reported the same filesystem 20.8% used. The rule was named and documented as a
|
||||
# CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can
|
||||
# never clear, so it was a PERMANENTLY-FIRING alert. It is deleted.
|
||||
# 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series
|
||||
# present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state`
|
||||
# label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed.
|
||||
# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule
|
||||
# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a
|
||||
# constructed bad state, with the `for:` durations asserted as real.
|
||||
#
|
||||
# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so
|
||||
# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the
|
||||
# literal bytes measured on prod 2026-09-13. Both are noted at the point of use.
|
||||
#
|
||||
# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so
|
||||
# nothing is ever stale inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT.
|
||||
# This is the "not permanently firing" half of the proof for every rule below, and it is the
|
||||
# test that would have caught the deleted rule: the SAME fixture carries the real
|
||||
# actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted
|
||||
# expression evaluating to 0.9731 on a volume whose filesystem is 20% used.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet
|
||||
input_series:
|
||||
# --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) ---
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
# --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod ---
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) ---
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '100326506496+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '456002658304+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '105149128704+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '105464528896+0x120'
|
||||
# --- Longhorn's own scheduling verdict: all three disks accepting replicas ---
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
# `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition`
|
||||
# selector is load-bearing. Drop the selector and the rule would still be quiet on THIS
|
||||
# fixture, so test 5 constructs the case where it matters.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
# --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free ---
|
||||
# Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080.
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '42949672960+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod)
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '4993977548+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
# The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the
|
||||
# per-volume Longhorn rule does not trade a false alarm for a blind spot on this state.
|
||||
- eval_time: 2h
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
# …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the
|
||||
# whole argument for the deletion, and it is now an assertion rather than a paragraph.
|
||||
- expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
value: 0.2
|
||||
# THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) …
|
||||
- expr: count(longhorn_volume_robustness)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 8
|
||||
# … and the OLD expressions still match nothing on this metric shape. This is the regression
|
||||
# guard: if someone "simplifies" the rules back to a numeric enum, these two go red.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
- expr: count(longhorn_volume_robustness == 3) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
# The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules
|
||||
# are quiet because the state is healthy, not because they select nothing.
|
||||
- expr: count(longhorn_volume_robustness{state="degraded"})
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`.
|
||||
# Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 10m` is real: nothing at 5m.
|
||||
- eval_time: 5m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-2
|
||||
pvc: keycloak-db-1
|
||||
pvc_namespace: keycloak
|
||||
volume: pvc-f30d3eb7
|
||||
state: degraded
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume degraded: pvc-f30d3eb7"
|
||||
description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
# Degraded must not masquerade as faulted: the critical tier stays silent.
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The pre-fix expression, on the state it was written for. Empty. This is the inert rule that
|
||||
# reached prod, reproduced offline.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume faulted - Faulted fires after 1m, Degraded stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 1m` is real: pending, not firing, at the first evaluation.
|
||||
- eval_time: 0m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
pvc: nextcloud-db-1
|
||||
pvc_namespace: nextcloud-db
|
||||
volume: pvc-d6621428
|
||||
state: faulted
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume FAULTED: pvc-d6621428"
|
||||
description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK.
|
||||
# Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the
|
||||
# promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the
|
||||
# node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level
|
||||
# form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it
|
||||
input_series:
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '288000000000+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '92000000000+0x120'
|
||||
# Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires
|
||||
# on its own, ahead of the terminal signal tested in case 5.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 30m` is real: nothing at 20m.
|
||||
- eval_time: 20m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-1
|
||||
disk: default-disk-new
|
||||
exp_annotations:
|
||||
summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new"
|
||||
description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# Per disk: one at 0.72, one at 0.23.
|
||||
- expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
value: 0.23
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
value: 0.72
|
||||
# Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent
|
||||
# at the old 0.85. That is the blind spot the metric change closes.
|
||||
- expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 0.475
|
||||
|
||||
# ===========================================================================================
|
||||
# 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires.
|
||||
# The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so
|
||||
# a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the
|
||||
# schedulable series only by luck of which conditions happen to exist. Allocation is nowhere
|
||||
# near the threshold, so this rule is proven to stand on its own signal.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone
|
||||
input_series:
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '92000000000+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart.
|
||||
- eval_time: 10m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
disk: default-disk-59b6657d
|
||||
condition: schedulable
|
||||
exp_annotations:
|
||||
summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d"
|
||||
description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule
|
||||
# fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the
|
||||
# allocation number cannot tell these two situations apart, and the kubelet number can. So the
|
||||
# signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for
|
||||
# the monitoring namespace and by the pvc-capacity rules for every other namespace.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged
|
||||
input_series:
|
||||
# allocation IDENTICAL to the healthy case — 0.9731, indistinguishable
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15%
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# a non-monitoring Longhorn PVC: 10% free -> warning tier
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '536870912+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
# another one at 3% free -> critical tier as well as warning
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '322122547+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '655360+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` on all three: nothing at 10m.
|
||||
- eval_time: 10m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: monitoring
|
||||
node: k3s-prod-1
|
||||
persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
|
||||
exp_annotations:
|
||||
summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
|
||||
description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
||||
- eval_time: 10m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
# Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the
|
||||
# exact-partition property pvc-capacity-alerts.yaml documents.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: keycloak
|
||||
node: k3s-prod-2
|
||||
persistentvolumeclaim: keycloak-db-1
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: keycloak/keycloak-db-1"
|
||||
description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# Only the 3%-free one reaches the critical tier — the 10% one must not.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC almost full: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
promql_expr_test:
|
||||
# The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the
|
||||
# filesystem went from 80% free to 10% free. One number moved; the other cannot.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
Reference in new issue
Block a user