# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml) # and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity" # rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file, # PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml). # # Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via # conformance/tests/l0-static/alert-rules-unit-tested) # # WHAT THIS FILE IS FOR # On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for # the first time. Two things became visible within minutes: # 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the # kubelet reported the same filesystem 20.8% used. The rule was named and documented as a # CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can # never clear, so it was a PERMANENTLY-FIRING alert. It is deleted. # 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series # present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state` # label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed. # Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule # touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a # constructed bad state, with the `for:` durations asserted as real. # # NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so # the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the # literal bytes measured on prod 2026-09-13. Both are noted at the point of use. # # CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so # nothing is ever stale inside Prometheus' 5m lookback. rule_files: - generated.rules.yaml evaluation_interval: 1m tests: # =========================================================================================== # 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT. # This is the "not permanently firing" half of the proof for every rule below, and it is the # test that would have caught the deleted rule: the SAME fixture carries the real # actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted # expression evaluating to 0.9731 on a volume whose filesystem is 20% used. # =========================================================================================== - interval: 1m name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet input_series: # --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) --- - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}' values: '1+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}' values: '1+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}' values: '0+0x120' # --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod --- - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' values: '52243681280+0x120' - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' values: '53687091200+0x120' # --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) --- - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' values: '445098295296+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' values: '100326506496+0x120' - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}' values: '456002658304+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}' values: '105149128704+0x120' - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' values: '445098295296+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' values: '105464528896+0x120' # --- Longhorn's own scheduling verdict: all three disks accepting replicas --- - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}' values: '1+0x120' - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}' values: '1+0x120' - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}' values: '1+0x120' # `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition` # selector is load-bearing. Drop the selector and the rule would still be quiet on THIS # fixture, so test 5 constructs the case where it matters. - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}' values: '1+0x120' # --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free --- # Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080. - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '53687091200+0x120' - series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '10737418240+0x120' - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '42949672960+0x120' - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '3276800+0x120' # and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod) - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '5368709120+0x120' - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '4993977548+0x120' - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '327680+0x120' alert_rule_test: - eval_time: 2h alertname: LonghornVolumeDegraded exp_alerts: [] - eval_time: 2h alertname: LonghornVolumeFaulted exp_alerts: [] - eval_time: 2h alertname: LonghornDiskAllocationHigh exp_alerts: [] - eval_time: 2h alertname: LonghornDiskUnschedulable exp_alerts: [] # The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the # per-volume Longhorn rule does not trade a false alarm for a blind spot on this state. - eval_time: 2h alertname: MonitoringVolumeFillingUp exp_alerts: [] - eval_time: 2h alertname: PersistentVolumeFillingUp exp_alerts: [] - eval_time: 2h alertname: PersistentVolumeAlmostFull exp_alerts: [] promql_expr_test: # THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on. - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes eval_time: 1h exp_samples: - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' value: 0.9731143951416016 # …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the # whole argument for the deletion, and it is now an assertion rather than a paragraph. - expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes eval_time: 1h exp_samples: - labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' value: 0.2 # THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) … - expr: count(longhorn_volume_robustness) eval_time: 1h exp_samples: - labels: '{}' value: 8 # … and the OLD expressions still match nothing on this metric shape. This is the regression # guard: if someone "simplifies" the rules back to a numeric enum, these two go red. - expr: count(longhorn_volume_robustness == 2) or vector(-1) eval_time: 1h exp_samples: - labels: '{}' value: -1 - expr: count(longhorn_volume_robustness == 3) or vector(-1) eval_time: 1h exp_samples: - labels: '{}' value: -1 # The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules # are quiet because the state is healthy, not because they select nothing. - expr: count(longhorn_volume_robustness{state="degraded"}) eval_time: 1h exp_samples: - labels: '{}' value: 2 # =========================================================================================== # 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`. # Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point. # =========================================================================================== - interval: 1m name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet input_series: - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}' values: '1+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}' values: '0+0x120' alert_rule_test: # `for: 10m` is real: nothing at 5m. - eval_time: 5m alertname: LonghornVolumeDegraded exp_alerts: [] - eval_time: 30m alertname: LonghornVolumeDegraded exp_alerts: - exp_labels: severity: warning job: longhorn-backend namespace: longhorn-system node: k3s-prod-2 pvc: keycloak-db-1 pvc_namespace: keycloak volume: pvc-f30d3eb7 state: degraded exp_annotations: summary: "Longhorn volume degraded: pvc-f30d3eb7" description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded" # Degraded must not masquerade as faulted: the critical tier stays silent. - eval_time: 30m alertname: LonghornVolumeFaulted exp_alerts: [] promql_expr_test: # The pre-fix expression, on the state it was written for. Empty. This is the inert rule that # reached prod, reproduced offline. - expr: count(longhorn_volume_robustness == 2) or vector(-1) eval_time: 30m exp_samples: - labels: '{}' value: -1 # =========================================================================================== # 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not. # =========================================================================================== - interval: 1m name: one volume faulted - Faulted fires after 1m, Degraded stays quiet input_series: - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}' values: '0+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}' values: '1+0x120' - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}' values: '0+0x120' alert_rule_test: # `for: 1m` is real: pending, not firing, at the first evaluation. - eval_time: 0m alertname: LonghornVolumeFaulted exp_alerts: [] - eval_time: 10m alertname: LonghornVolumeFaulted exp_alerts: - exp_labels: severity: critical job: longhorn-backend namespace: longhorn-system node: k3s-prod-3 pvc: nextcloud-db-1 pvc_namespace: nextcloud-db volume: pvc-d6621428 state: faulted exp_annotations: summary: "Longhorn volume FAULTED: pvc-d6621428" description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted" - eval_time: 10m alertname: LonghornVolumeDegraded exp_alerts: [] # =========================================================================================== # 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK. # Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the # promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the # node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level # form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff. # =========================================================================================== - interval: 1m name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it input_series: - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' values: '400000000000+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' values: '288000000000+0x120' - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' values: '400000000000+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' values: '92000000000+0x120' # Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires # on its own, ahead of the terminal signal tested in case 5. - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}' values: '1+0x120' alert_rule_test: # `for: 30m` is real: nothing at 20m. - eval_time: 20m alertname: LonghornDiskAllocationHigh exp_alerts: [] - eval_time: 45m alertname: LonghornDiskAllocationHigh exp_alerts: - exp_labels: severity: warning job: longhorn-backend namespace: longhorn-system node: k3s-prod-1 disk: default-disk-new exp_annotations: summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new" description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" - eval_time: 45m alertname: LonghornDiskUnschedulable exp_alerts: [] promql_expr_test: # Per disk: one at 0.72, one at 0.23. - expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) eval_time: 45m exp_samples: - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' value: 0.23 - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' value: 0.72 # Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent # at the old 0.85. That is the blind spot the metric change closes. - expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes) eval_time: 45m exp_samples: - labels: '{node="k3s-prod-1"}' value: 0.475 # =========================================================================================== # 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires. # The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so # a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the # schedulable series only by luck of which conditions happen to exist. Allocation is nowhere # near the threshold, so this rule is proven to stand on its own signal. # =========================================================================================== - interval: 1m name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone input_series: - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}' values: '0+0x120' - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}' values: '1+0x120' - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' values: '400000000000+0x120' - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' values: '92000000000+0x120' alert_rule_test: # `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart. - eval_time: 10m alertname: LonghornDiskUnschedulable exp_alerts: [] - eval_time: 30m alertname: LonghornDiskUnschedulable exp_alerts: - exp_labels: severity: warning job: longhorn-backend namespace: longhorn-system node: k3s-prod-3 disk: default-disk-59b6657d condition: schedulable exp_annotations: summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d" description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" - eval_time: 30m alertname: LonghornDiskAllocationHigh exp_alerts: [] # =========================================================================================== # 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule # fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the # allocation number cannot tell these two situations apart, and the kubelet number can. So the # signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for # the monitoring namespace and by the pvc-capacity rules for every other namespace. # =========================================================================================== - interval: 1m name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged input_series: # allocation IDENTICAL to the healthy case — 0.9731, indistinguishable - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' values: '52243681280+0x120' - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' values: '53687091200+0x120' # monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15% - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '53687091200+0x120' - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '5368709120+0x120' - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' values: '3276800+0x120' # a non-monitoring Longhorn PVC: 10% free -> warning tier - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '5368709120+0x120' - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '536870912+0x120' - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' values: '327680+0x120' # another one at 3% free -> critical tier as well as warning - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' values: '10737418240+0x120' - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' values: '322122547+0x120' - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' values: '655360+0x120' alert_rule_test: # `for: 15m` on all three: nothing at 10m. - eval_time: 10m alertname: MonitoringVolumeFillingUp exp_alerts: [] - eval_time: 30m alertname: MonitoringVolumeFillingUp exp_alerts: - exp_labels: severity: warning job: kubelet namespace: monitoring node: k3s-prod-1 persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 exp_annotations: summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0" description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind." runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity" - eval_time: 10m alertname: PersistentVolumeFillingUp exp_alerts: [] # Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the # exact-partition property pvc-capacity-alerts.yaml documents. - eval_time: 30m alertname: PersistentVolumeFillingUp exp_alerts: - exp_labels: severity: warning job: kubelet namespace: gitea node: k3s-prod-3 persistentvolumeclaim: gitea-data exp_annotations: summary: "PVC filling up: gitea/gitea-data" description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database." runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" - exp_labels: severity: warning job: kubelet namespace: keycloak node: k3s-prod-2 persistentvolumeclaim: keycloak-db-1 exp_annotations: summary: "PVC filling up: keycloak/keycloak-db-1" description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database." runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" # Only the 3%-free one reaches the critical tier — the 10% one must not. - eval_time: 30m alertname: PersistentVolumeAlmostFull exp_alerts: - exp_labels: severity: critical job: kubelet namespace: gitea node: k3s-prod-3 persistentvolumeclaim: gitea-data exp_annotations: summary: "PVC almost full: gitea/gitea-data" description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down." runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" promql_expr_test: # The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the # filesystem went from 80% free to 10% free. One number moved; the other cannot. - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes eval_time: 30m exp_samples: - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' value: 0.9731143951416016