merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md
This commit is contained in:
jakman85 committed
commit
52398d4426
11 files changed
+912
-113
No files matched your search
@@ -0,0 +1,465 @@
|
||||
# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml)
|
||||
# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity"
|
||||
# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file,
|
||||
# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml).
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THIS FILE IS FOR
|
||||
# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for
|
||||
# the first time. Two things became visible within minutes:
|
||||
# 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the
|
||||
# kubelet reported the same filesystem 20.8% used. The rule was named and documented as a
|
||||
# CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can
|
||||
# never clear, so it was a PERMANENTLY-FIRING alert. It is deleted.
|
||||
# 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series
|
||||
# present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state`
|
||||
# label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed.
|
||||
# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule
|
||||
# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a
|
||||
# constructed bad state, with the `for:` durations asserted as real.
|
||||
#
|
||||
# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so
|
||||
# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the
|
||||
# literal bytes measured on prod 2026-09-13. Both are noted at the point of use.
|
||||
#
|
||||
# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so
|
||||
# nothing is ever stale inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT.
|
||||
# This is the "not permanently firing" half of the proof for every rule below, and it is the
|
||||
# test that would have caught the deleted rule: the SAME fixture carries the real
|
||||
# actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted
|
||||
# expression evaluating to 0.9731 on a volume whose filesystem is 20% used.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet
|
||||
input_series:
|
||||
# --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) ---
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
# --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod ---
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) ---
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '100326506496+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '456002658304+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '105149128704+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '105464528896+0x120'
|
||||
# --- Longhorn's own scheduling verdict: all three disks accepting replicas ---
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
# `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition`
|
||||
# selector is load-bearing. Drop the selector and the rule would still be quiet on THIS
|
||||
# fixture, so test 5 constructs the case where it matters.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
# --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free ---
|
||||
# Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080.
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '42949672960+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod)
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '4993977548+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
# The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the
|
||||
# per-volume Longhorn rule does not trade a false alarm for a blind spot on this state.
|
||||
- eval_time: 2h
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
# …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the
|
||||
# whole argument for the deletion, and it is now an assertion rather than a paragraph.
|
||||
- expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
value: 0.2
|
||||
# THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) …
|
||||
- expr: count(longhorn_volume_robustness)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 8
|
||||
# … and the OLD expressions still match nothing on this metric shape. This is the regression
|
||||
# guard: if someone "simplifies" the rules back to a numeric enum, these two go red.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
- expr: count(longhorn_volume_robustness == 3) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
# The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules
|
||||
# are quiet because the state is healthy, not because they select nothing.
|
||||
- expr: count(longhorn_volume_robustness{state="degraded"})
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`.
|
||||
# Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 10m` is real: nothing at 5m.
|
||||
- eval_time: 5m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-2
|
||||
pvc: keycloak-db-1
|
||||
pvc_namespace: keycloak
|
||||
volume: pvc-f30d3eb7
|
||||
state: degraded
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume degraded: pvc-f30d3eb7"
|
||||
description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
# Degraded must not masquerade as faulted: the critical tier stays silent.
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The pre-fix expression, on the state it was written for. Empty. This is the inert rule that
|
||||
# reached prod, reproduced offline.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume faulted - Faulted fires after 1m, Degraded stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 1m` is real: pending, not firing, at the first evaluation.
|
||||
- eval_time: 0m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
pvc: nextcloud-db-1
|
||||
pvc_namespace: nextcloud-db
|
||||
volume: pvc-d6621428
|
||||
state: faulted
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume FAULTED: pvc-d6621428"
|
||||
description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK.
|
||||
# Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the
|
||||
# promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the
|
||||
# node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level
|
||||
# form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it
|
||||
input_series:
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '288000000000+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '92000000000+0x120'
|
||||
# Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires
|
||||
# on its own, ahead of the terminal signal tested in case 5.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 30m` is real: nothing at 20m.
|
||||
- eval_time: 20m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-1
|
||||
disk: default-disk-new
|
||||
exp_annotations:
|
||||
summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new"
|
||||
description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# Per disk: one at 0.72, one at 0.23.
|
||||
- expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
value: 0.23
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
value: 0.72
|
||||
# Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent
|
||||
# at the old 0.85. That is the blind spot the metric change closes.
|
||||
- expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 0.475
|
||||
|
||||
# ===========================================================================================
|
||||
# 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires.
|
||||
# The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so
|
||||
# a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the
|
||||
# schedulable series only by luck of which conditions happen to exist. Allocation is nowhere
|
||||
# near the threshold, so this rule is proven to stand on its own signal.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone
|
||||
input_series:
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '92000000000+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart.
|
||||
- eval_time: 10m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
disk: default-disk-59b6657d
|
||||
condition: schedulable
|
||||
exp_annotations:
|
||||
summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d"
|
||||
description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule
|
||||
# fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the
|
||||
# allocation number cannot tell these two situations apart, and the kubelet number can. So the
|
||||
# signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for
|
||||
# the monitoring namespace and by the pvc-capacity rules for every other namespace.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged
|
||||
input_series:
|
||||
# allocation IDENTICAL to the healthy case — 0.9731, indistinguishable
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15%
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# a non-monitoring Longhorn PVC: 10% free -> warning tier
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '536870912+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
# another one at 3% free -> critical tier as well as warning
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '322122547+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '655360+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` on all three: nothing at 10m.
|
||||
- eval_time: 10m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: monitoring
|
||||
node: k3s-prod-1
|
||||
persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
|
||||
exp_annotations:
|
||||
summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
|
||||
description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
||||
- eval_time: 10m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
# Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the
|
||||
# exact-partition property pvc-capacity-alerts.yaml documents.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: keycloak
|
||||
node: k3s-prod-2
|
||||
persistentvolumeclaim: keycloak-db-1
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: keycloak/keycloak-db-1"
|
||||
description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# Only the 3%-free one reaches the critical tier — the 10% one must not.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC almost full: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
promql_expr_test:
|
||||
# The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the
|
||||
# filesystem went from 80% free to 10% free. One number moved; the other cannot.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
Reference in new issue
Block a user