fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
72f0ba4e1f
commit
4687521d65
11 files changed
+913
-113
No files matched your search
+116
-12
@@ -147,9 +147,33 @@ spec:
|
||||
groups:
|
||||
- name: platform-health.longhorn
|
||||
rules:
|
||||
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
|
||||
# -------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
||||
#
|
||||
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
||||
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
||||
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
||||
# label and its value is only ever 0 or 1:
|
||||
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
||||
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
||||
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# count(longhorn_volume_robustness == 2) => (empty)
|
||||
# count(longhorn_volume_robustness == 3) => (empty)
|
||||
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
||||
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
||||
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
||||
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
||||
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
||||
#
|
||||
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
||||
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornVolumeDegraded
|
||||
expr: longhorn_volume_robustness == 2
|
||||
expr: longhorn_volume_robustness{state="degraded"} == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -158,7 +182,7 @@ spec:
|
||||
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
- alert: LonghornVolumeFaulted
|
||||
expr: longhorn_volume_robustness == 3
|
||||
expr: longhorn_volume_robustness{state="faulted"} == 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -166,23 +190,103 @@ spec:
|
||||
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- alert: LonghornVolumeCapacityHigh
|
||||
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
|
||||
# -------------------------------------------------------------------------------------
|
||||
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
||||
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
||||
#
|
||||
# The deleted rule was
|
||||
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
||||
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
||||
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
||||
# MEASURED on prod 2026-09-13:
|
||||
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
||||
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
||||
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
||||
# ratio 0.9731 <- what the alert saw
|
||||
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
||||
# the SAME filesystem, from the kubelet:
|
||||
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
||||
#
|
||||
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
||||
# every snapshot still pinned. On this volume,
|
||||
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
||||
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
||||
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
||||
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
||||
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
||||
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
||||
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
||||
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
||||
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
||||
# to ignore the storage row, which is the row that matters on this cluster.
|
||||
#
|
||||
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
||||
# asserted:
|
||||
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
||||
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
||||
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
||||
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
||||
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
||||
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
||||
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
||||
# Block volumes are covered at the pool level, below.
|
||||
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
||||
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
||||
# against its own nominal size, where it is a tautology.
|
||||
# -------------------------------------------------------------------------------------
|
||||
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
||||
#
|
||||
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
||||
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
||||
#
|
||||
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
||||
# replica files plus their pinned snapshots — because that is what consumes the node's
|
||||
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
||||
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
||||
# "disk allocation high" says what it measures.
|
||||
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
||||
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
||||
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
||||
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
||||
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
||||
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
||||
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
||||
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
||||
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
||||
# gives lead time on the way to Longhorn's own limit.
|
||||
#
|
||||
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
||||
# node capacity usage ratio scheduled (nominal)
|
||||
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
||||
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
||||
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornDiskAllocationHigh
|
||||
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
|
||||
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- alert: LonghornNodeStorageHigh
|
||||
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
|
||||
for: 30m
|
||||
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
||||
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
||||
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
||||
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
||||
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
||||
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
||||
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
||||
# a longhorn-manager restarts.
|
||||
- alert: LonghornDiskUnschedulable
|
||||
expr: longhorn_disk_status{condition="schedulable"} == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
|
||||
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
|
||||
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
Reference in new issue
Block a user