fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity

LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.

DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.

AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
  count(longhorn_volume_robustness)                  => 96  (24 volumes x 4 states)
  count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
  count(longhorn_volume_robustness == 2)             => (empty)
  count(longhorn_volume_robustness == 3)             => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.

The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
  LonghornDiskAllocationHigh  (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
  LonghornDiskUnschedulable   longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.

PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.

REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/longhorn/helmrelease.yaml

@@ -103,9 +103,9 @@ spec:
       orphanAutoDeletion: true
 
     # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
-    # platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
-    # LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
-    # and NONE of them could ever fire, because the chart defaults
+    # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
+    # LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
+    # LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
     # `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
     # VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
     # `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
@@ -140,12 +140,26 @@ spec:
     # colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
     #   count(up{job="longhorn-backend"})              => 3      (all three managers, up==1)
     #   count({__name__=~"longhorn.*"})                => 8717   (was EMPTY)
-    #   count(longhorn_volume_robustness)              => 44
-    #   count_values("v", longhorn_volume_robustness)  => 24 healthy(1), 72 unknown(0, detached)
-    # and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
-    # them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
-    # volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
-    # freed blocks. Reported, NOT silenced by moving the threshold).
+    #   count(longhorn_volume_robustness)              => 96     (24 volumes x 4 `state` labels)
+    #   count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
+    # CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
+    # "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
+    # metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
+    # `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
+    # zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
+    # volumes. The consequence was that enabling this scrape did NOT un-inert
+    # LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
+    # Fixed in platform/monitoring/alerts.yaml and pinned by
+    # platform/monitoring/tests/longhorn.test.yaml.
+    #
+    # Of the rules that WERE evaluating for real, one immediately found a true positive and one
+    # permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
+    # Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
+    # not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
+    # replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
+    # it measured allocation and called it capacity, the filesystem question is already answered by
+    # kubelet_volume_stats_*, and the allocation question now lives per disk as
+    # LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
     metrics:
       serviceMonitor:
         enabled: true