merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md
This commit is contained in:
jakman85 committed
commit
52398d4426
11 files changed
+912
-113
No files matched your search
@@ -103,9 +103,9 @@ spec:
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
|
||||
# LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
|
||||
# and NONE of them could ever fire, because the chart defaults
|
||||
# platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
|
||||
# LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
|
||||
# LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
|
||||
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
||||
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
||||
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
||||
@@ -140,12 +140,26 @@ spec:
|
||||
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
||||
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
||||
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
||||
# count(longhorn_volume_robustness) => 44
|
||||
# count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached)
|
||||
# and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
|
||||
# them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
|
||||
# volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
|
||||
# freed blocks. Reported, NOT silenced by moving the threshold).
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 `state` labels)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
|
||||
# "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
|
||||
# metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
|
||||
# `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
|
||||
# zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
|
||||
# volumes. The consequence was that enabling this scrape did NOT un-inert
|
||||
# LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
|
||||
# Fixed in platform/monitoring/alerts.yaml and pinned by
|
||||
# platform/monitoring/tests/longhorn.test.yaml.
|
||||
#
|
||||
# Of the rules that WERE evaluating for real, one immediately found a true positive and one
|
||||
# permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
|
||||
# Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
|
||||
# not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
|
||||
# replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
|
||||
# it measured allocation and called it capacity, the filesystem question is already answered by
|
||||
# kubelet_volume_stats_*, and the allocation question now lives per disk as
|
||||
# LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
|
||||
+116
-12
@@ -147,9 +147,33 @@ spec:
|
||||
groups:
|
||||
- name: platform-health.longhorn
|
||||
rules:
|
||||
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
|
||||
# -------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
||||
#
|
||||
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
||||
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
||||
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
||||
# label and its value is only ever 0 or 1:
|
||||
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
||||
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
||||
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# count(longhorn_volume_robustness == 2) => (empty)
|
||||
# count(longhorn_volume_robustness == 3) => (empty)
|
||||
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
||||
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
||||
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
||||
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
||||
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
||||
#
|
||||
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
||||
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornVolumeDegraded
|
||||
expr: longhorn_volume_robustness == 2
|
||||
expr: longhorn_volume_robustness{state="degraded"} == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -158,7 +182,7 @@ spec:
|
||||
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
- alert: LonghornVolumeFaulted
|
||||
expr: longhorn_volume_robustness == 3
|
||||
expr: longhorn_volume_robustness{state="faulted"} == 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -166,23 +190,103 @@ spec:
|
||||
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- alert: LonghornVolumeCapacityHigh
|
||||
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
|
||||
# -------------------------------------------------------------------------------------
|
||||
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
||||
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
||||
#
|
||||
# The deleted rule was
|
||||
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
||||
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
||||
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
||||
# MEASURED on prod 2026-09-13:
|
||||
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
||||
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
||||
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
||||
# ratio 0.9731 <- what the alert saw
|
||||
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
||||
# the SAME filesystem, from the kubelet:
|
||||
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
||||
#
|
||||
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
||||
# every snapshot still pinned. On this volume,
|
||||
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
||||
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
||||
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
||||
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
||||
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
||||
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
||||
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
||||
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
||||
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
||||
# to ignore the storage row, which is the row that matters on this cluster.
|
||||
#
|
||||
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
||||
# asserted:
|
||||
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
||||
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
||||
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
||||
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
||||
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
||||
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
||||
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
||||
# Block volumes are covered at the pool level, below.
|
||||
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
||||
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
||||
# against its own nominal size, where it is a tautology.
|
||||
# -------------------------------------------------------------------------------------
|
||||
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
||||
#
|
||||
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
||||
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
||||
#
|
||||
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
||||
# replica files plus their pinned snapshots — because that is what consumes the node's
|
||||
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
||||
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
||||
# "disk allocation high" says what it measures.
|
||||
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
||||
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
||||
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
||||
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
||||
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
||||
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
||||
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
||||
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
||||
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
||||
# gives lead time on the way to Longhorn's own limit.
|
||||
#
|
||||
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
||||
# node capacity usage ratio scheduled (nominal)
|
||||
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
||||
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
||||
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornDiskAllocationHigh
|
||||
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
|
||||
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- alert: LonghornNodeStorageHigh
|
||||
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
|
||||
for: 30m
|
||||
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
||||
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
||||
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
||||
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
||||
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
||||
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
||||
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
||||
# a longhorn-manager restarts.
|
||||
- alert: LonghornDiskUnschedulable
|
||||
expr: longhorn_disk_status{condition="schedulable"} == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
|
||||
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
|
||||
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
@@ -10,12 +10,26 @@
|
||||
# the SSO database with nothing watching it.
|
||||
#
|
||||
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
|
||||
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
|
||||
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
|
||||
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
|
||||
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
|
||||
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
|
||||
# working on both clusters.
|
||||
# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two
|
||||
# different reasons in sequence, and it is now GONE:
|
||||
# * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified
|
||||
# read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule
|
||||
# group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in
|
||||
# platform/longhorn/helmrelease.yaml.
|
||||
# * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It
|
||||
# divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head
|
||||
# plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and
|
||||
# called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet
|
||||
# read the same filesystem 20.8% used — by design and permanently, because `retentionSize:
|
||||
# 45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the
|
||||
# comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now
|
||||
# lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to
|
||||
# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`.
|
||||
# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in
|
||||
# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in
|
||||
# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no
|
||||
# filesystem for anyone to measure (see the inodes guard below).
|
||||
#
|
||||
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
|
||||
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
|
||||
|
||||
@@ -0,0 +1,465 @@
|
||||
# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml)
|
||||
# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity"
|
||||
# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file,
|
||||
# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml).
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THIS FILE IS FOR
|
||||
# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for
|
||||
# the first time. Two things became visible within minutes:
|
||||
# 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the
|
||||
# kubelet reported the same filesystem 20.8% used. The rule was named and documented as a
|
||||
# CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can
|
||||
# never clear, so it was a PERMANENTLY-FIRING alert. It is deleted.
|
||||
# 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series
|
||||
# present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state`
|
||||
# label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed.
|
||||
# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule
|
||||
# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a
|
||||
# constructed bad state, with the `for:` durations asserted as real.
|
||||
#
|
||||
# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so
|
||||
# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the
|
||||
# literal bytes measured on prod 2026-09-13. Both are noted at the point of use.
|
||||
#
|
||||
# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so
|
||||
# nothing is ever stale inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT.
|
||||
# This is the "not permanently firing" half of the proof for every rule below, and it is the
|
||||
# test that would have caught the deleted rule: the SAME fixture carries the real
|
||||
# actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted
|
||||
# expression evaluating to 0.9731 on a volume whose filesystem is 20% used.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet
|
||||
input_series:
|
||||
# --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) ---
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
# --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod ---
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) ---
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '100326506496+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '456002658304+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '105149128704+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '105464528896+0x120'
|
||||
# --- Longhorn's own scheduling verdict: all three disks accepting replicas ---
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
# `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition`
|
||||
# selector is load-bearing. Drop the selector and the rule would still be quiet on THIS
|
||||
# fixture, so test 5 constructs the case where it matters.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
# --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free ---
|
||||
# Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080.
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '42949672960+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod)
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '4993977548+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
# The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the
|
||||
# per-volume Longhorn rule does not trade a false alarm for a blind spot on this state.
|
||||
- eval_time: 2h
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
# …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the
|
||||
# whole argument for the deletion, and it is now an assertion rather than a paragraph.
|
||||
- expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
value: 0.2
|
||||
# THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) …
|
||||
- expr: count(longhorn_volume_robustness)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 8
|
||||
# … and the OLD expressions still match nothing on this metric shape. This is the regression
|
||||
# guard: if someone "simplifies" the rules back to a numeric enum, these two go red.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
- expr: count(longhorn_volume_robustness == 3) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
# The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules
|
||||
# are quiet because the state is healthy, not because they select nothing.
|
||||
- expr: count(longhorn_volume_robustness{state="degraded"})
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`.
|
||||
# Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 10m` is real: nothing at 5m.
|
||||
- eval_time: 5m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-2
|
||||
pvc: keycloak-db-1
|
||||
pvc_namespace: keycloak
|
||||
volume: pvc-f30d3eb7
|
||||
state: degraded
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume degraded: pvc-f30d3eb7"
|
||||
description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
# Degraded must not masquerade as faulted: the critical tier stays silent.
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The pre-fix expression, on the state it was written for. Empty. This is the inert rule that
|
||||
# reached prod, reproduced offline.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume faulted - Faulted fires after 1m, Degraded stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 1m` is real: pending, not firing, at the first evaluation.
|
||||
- eval_time: 0m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
pvc: nextcloud-db-1
|
||||
pvc_namespace: nextcloud-db
|
||||
volume: pvc-d6621428
|
||||
state: faulted
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume FAULTED: pvc-d6621428"
|
||||
description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK.
|
||||
# Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the
|
||||
# promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the
|
||||
# node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level
|
||||
# form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it
|
||||
input_series:
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '288000000000+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '92000000000+0x120'
|
||||
# Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires
|
||||
# on its own, ahead of the terminal signal tested in case 5.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 30m` is real: nothing at 20m.
|
||||
- eval_time: 20m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-1
|
||||
disk: default-disk-new
|
||||
exp_annotations:
|
||||
summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new"
|
||||
description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# Per disk: one at 0.72, one at 0.23.
|
||||
- expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
value: 0.23
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
value: 0.72
|
||||
# Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent
|
||||
# at the old 0.85. That is the blind spot the metric change closes.
|
||||
- expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 0.475
|
||||
|
||||
# ===========================================================================================
|
||||
# 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires.
|
||||
# The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so
|
||||
# a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the
|
||||
# schedulable series only by luck of which conditions happen to exist. Allocation is nowhere
|
||||
# near the threshold, so this rule is proven to stand on its own signal.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone
|
||||
input_series:
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '92000000000+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart.
|
||||
- eval_time: 10m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
disk: default-disk-59b6657d
|
||||
condition: schedulable
|
||||
exp_annotations:
|
||||
summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d"
|
||||
description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule
|
||||
# fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the
|
||||
# allocation number cannot tell these two situations apart, and the kubelet number can. So the
|
||||
# signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for
|
||||
# the monitoring namespace and by the pvc-capacity rules for every other namespace.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged
|
||||
input_series:
|
||||
# allocation IDENTICAL to the healthy case — 0.9731, indistinguishable
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15%
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# a non-monitoring Longhorn PVC: 10% free -> warning tier
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '536870912+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
# another one at 3% free -> critical tier as well as warning
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '322122547+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '655360+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` on all three: nothing at 10m.
|
||||
- eval_time: 10m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: monitoring
|
||||
node: k3s-prod-1
|
||||
persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
|
||||
exp_annotations:
|
||||
summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
|
||||
description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
||||
- eval_time: 10m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
# Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the
|
||||
# exact-partition property pvc-capacity-alerts.yaml documents.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: keycloak
|
||||
node: k3s-prod-2
|
||||
persistentvolumeclaim: keycloak-db-1
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: keycloak/keycloak-db-1"
|
||||
description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# Only the 3%-free one reaches the critical tier — the 10% one must not.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC almost full: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
promql_expr_test:
|
||||
# The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the
|
||||
# filesystem went from 80% free to 10% free. One number moved; the other cannot.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
Reference in new issue
Block a user