fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity

LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.

DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.

AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
  count(longhorn_volume_robustness)                  => 96  (24 volumes x 4 states)
  count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
  count(longhorn_volume_robustness == 2)             => (empty)
  count(longhorn_volume_robustness == 3)             => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.

The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
  LonghornDiskAllocationHigh  (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
  LonghornDiskUnschedulable   longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.

PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.

REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3244,3 +3244,146 @@ standing as a deliberate red.** Writing a test tagged R-TLS-02 that silently ver
 and renewal would report the requirement as met while a third of its normative text is
 unimplemented — which is the one thing this harness exists to prevent. The choice is the
 architect's, not an agent's.
+
+## 2026-09-13 — `LonghornVolumeCapacityHigh` measured ALLOCATION and called it capacity; and the robustness rules were inert a SECOND time (branch `fix/longhorn-capacity-alert-measures-allocation`)
+
+Longhorn metrics began being scraped earlier today. Within minutes `LonghornVolumeCapacityHigh` went
+`pending`. Three separate defects came out of re-measuring it, two of them the exact failure classes
+this repo spent the day eliminating.
+
+### 1. The per-volume "capacity" rule was permanently firing, by design
+
+MEASURED on prod, `kubectl -n monitoring exec … promtool query instant`:
+
+| what | value |
+|---|---|
+| `longhorn_volume_actual_size_bytes` (Prometheus TSDB volume) | 52243681280 (48.66 GiB) |
+| `longhorn_volume_capacity_bytes` | 53687091200 (50 GiB) |
+| the rule's ratio | **0.9731** |
+| next-highest volume in the cluster | 0.2818 (keycloak-db-1) — a lone outlier |
+| `kubelet_volume_stats_used_bytes / …_capacity_bytes`, SAME filesystem | **0.2080** |
+| `longhorn_snapshot_actual_size_bytes{user_created="false"}` on that volume | 40510484480 (**37.7 GiB**) |
+| growth of actual_size, `deriv(…[2h])` | ~152 MB/h, monotonic |
+
+`actual_size` is head + pinned snapshots, not usage. ONE Longhorn system snapshot — the kind taken for
+a full replica rebuild, which prod-1's drive replacement forced — holds 37.7 GiB; the live head is the
+remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn returns no freed blocks without
+discard/TRIM, and `retentionSize: 45GiB` on a 50Gi volume deliberately targets 90%, so the ratio is
+pinned near 1.0 and **cannot fall back under 0.85**. Same shape as `CNPGNoRecoverabilityPoint`.
+
+DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation.
+The redundancy was proven rather than asserted: of 24 Longhorn volumes on prod, 23 appear in
+`kubelet_volume_stats_inodes > 0` and are therefore inside `MonitoringVolumeFillingUp`
+(namespace="monitoring") or `PersistentVolumeFillingUp` / `…AlmostFull` (everything else). The 24th is
+`unifi/unifi-rootdisk`, volumeMode Block — no filesystem for anyone to measure, and the deleted rule
+was not covering it either (0.2097 was raw-device allocation). Block volumes are covered at the pool
+level.
+
+### 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT after the scrape landed
+
+The bigger finding. On longhorn 1.12.1 `longhorn_volume_robustness` is **one-hot over a `state`
+label** with values 0/1 — it is NOT the 0-3 enum the rules and their comment assumed:
+
+```
+count(longhorn_volume_robustness)                  => 96      (24 volumes x 4 states)
+count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
+count(longhorn_volume_robustness == 2)             => (empty)
+count(longhorn_volume_robustness == 3)             => (empty)
+```
+
+So a FAULTED volume — the most serious storage state on this cluster — would have alerted nobody,
+with the scrape working and the series present. The reading recorded in
+`platform/longhorn/helmrelease.yaml` ("44 series … 24 healthy(1), 72 unknown(0, detached)") was this
+same misreading; those zeros are the other three state labels of the same 24 healthy volumes. Both
+rules are now keyed on the label, and a regression case asserts the old `== 2` form matches nothing.
+
+### 3. The node-level allocation alarm fired AFTER the cliff, and averaged disks together
+
+`LonghornNodeStorageHigh` at `> 0.85` was past Longhorn's own limit: `storage-minimal-available-percentage`
+is **25**, so a disk leaves the scheduling pool at 75% used. Re-keyed to the per-disk series (the
+node-level one SUMS a node's disks, so one full disk hides behind a half-empty sibling) and renamed:
+
+* `LonghornDiskAllocationHigh` — `(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`, 30m
+* `LonghornDiskUnschedulable` — `longhorn_disk_status{condition="schedulable"} == 0`, 15m — Longhorn's
+  own verdict, instead of re-deriving a scheduling formula that folds in three settings no metric
+  exports (reserved 20, over-provisioning 200, minimal-available 25).
+
+MEASURED prod usage: 0.2254 / 0.2306 / 0.2369 on prod-1/2/3; 268.9 GiB nominal scheduled per node.
+
+### Proof, both directions, for every rule touched
+
+`platform/monitoring/tests/longhorn.test.yaml` (6 cases, run by
+`platform/monitoring/tests/run.sh`; the L0 register shrank by 7 and the repo is now 14/58 alerts
+unit-tested). Live: all four new expressions return EMPTY on prod, while
+`count(robustness{state="degraded"}) + count(robustness{state="faulted"}) + count(disk ratio) +
+count(disk_status{condition="schedulable"})` returns **54** — quiet, and provably not inert.
+Mutation-checked 11 ways (each turns the suite RED): both old enum expressions, the 0.85 threshold,
+the node-summed metric, a threshold low enough to fire on today's healthy state, the wrong
+`condition` label, three shortened/lengthened `for:` values, and two mutations of the kubelet rules.
+
+### OWNER DECISION 1 — is `retentionSize: 45GiB` on a 50Gi volume right?
+
+**Recommendation: lower it to `36GiB` (about 72% of the volume). Not changed here — retention is how
+much history you keep, so it is your call.**
+
+The reasoning, and it is not about the alert:
+
+* 45GiB leaves 5 GiB of headroom on the volume, and `retentionSize` does **not** count the WAL, the
+  in-memory head's on-disk segments, or compaction scratch. A compaction of a large block can need
+  roughly the size of the blocks being merged. At 90% full, a single bad compaction is
+  "no space left on device", which wedges every WAL commit — scrapes still return 200, nothing is
+  stored, the head freezes, and every `absent()`/`== 0` platform alert fires falsely. The file's own
+  comment says the cap is "kept BELOW the PVC size … for WAL + head + compaction scratch"; 10% is not
+  below enough for that list.
+* The volume is 3x replicated on mini-PCs whose data partition also carries the etcd WAL. Every GiB of
+  TSDB is 3 GiB of allocation across the pool and more rebuild I/O when a replica resyncs — and
+  etcd fsync starvation is this cluster's known way of losing a node (2026-09-07).
+* It is not currently constraining anything: the TSDB is **11.2 GiB** at 30d retention on a 3-node
+  cluster. 36GiB is still triple today's size, so 30d would remain the binding limit in practice and
+  no history would actually be lost.
+* If more history is wanted, grow the PVC (`allowVolumeExpansion: true`) rather than the fraction —
+  the knobs are `MONITORING_PROMETHEUS_PVC_SIZE` and `MONITORING_PROMETHEUS_RETENTION_SIZE` in
+  `clusters/prod/kustomization.yaml`, and the base comment already requires the second to stay below
+  the first.
+
+### OWNER DECISION 2 — enable discard/TRIM on the `longhorn` StorageClass?
+
+**Recommendation: do NOT enable it globally now. It is feasible, but it buys a metric and risks the
+one subsystem this cluster keeps losing.** Not enabled here; it changes behaviour for every Longhorn
+volume on the cluster.
+
+* Feasible. The class already sets `fsType: "ext4"`, and Longhorn passes mount options through, so
+  continuous discard is a StorageClass change (`mountOptions: ["discard"]`) plus a REMOUNT of every
+  volume to take effect — i.e. a restart of every stateful workload. Longhorn also exposes a
+  per-volume `Trim Filesystem` action, which is the same reclaim without touching the class.
+* The setting that actually matters here is `remove-snapshots-during-filesystem-trim`, read live on
+  prod as **false**. With it false, a trim reclaims nothing that a snapshot is holding — so trim, in
+  any form, would NOT have reclaimed the 37.7 GiB. Flipping that one to true is a far larger decision
+  than enabling discard: it makes a routine filesystem trim DELETE SNAPSHOTS, i.e. quietly discard
+  recovery points.
+* Also read live: `auto-cleanup-system-generated-snapshot` is already **true**, and the 37.7 GiB
+  system snapshot is still present — so it is most likely the replica's base snapshot, which cannot be
+  removed while it is the parent of the volume head. If so, no trim setting and no discard flag will
+  release it; only a full replica rebuild onto a fresh file would, which is the expensive operation
+  this cluster is trying to avoid.
+* What it would buy: `actual_size` would track real usage. That is now worth nothing to alerting —
+  no rule reads `actual_size` any more, and the pool-level rules read disk usage, which trim improves
+  only for blocks no snapshot is holding. **The 37.7 GiB that started all this is held by a SNAPSHOT,
+  which discard cannot release** (see the two settings below).
+* What it risks: `discard` makes every filesystem delete issue a synchronous block-level discard on
+  all three replicas, on nodes where `/data/k3s` (etcd WAL) and `/data/longhorn` share one device. The
+  failure mode we already have is etcd fsync starvation under Longhorn write load. Turning on
+  continuous discard for 24 volumes is exactly the wrong direction for that, and a mass remount is a
+  rebuild-storm trigger in its own right (every node reboot degrades all volumes today).
+* If reclaim is wanted later, the safe order is: (1) check whether the big snapshot is a removable
+  snapshot or the replica's base, (2) purge what is removable from the Longhorn UI, (3) per-volume
+  `Trim Filesystem` during a quiet window, (4) only then consider a periodic `fstrim` timer — never
+  continuous `discard`, and never `remove-snapshots-during-filesystem-trim: true`, while etcd shares
+  the spindle. Revisit if/when `docs/plans/longhorn-ssd-tiering.md` separates etcd from Longhorn.
+
+### Also noticed, NOT changed (drive-by, for whoever owns it)
+
+`runbooks/longhorn-degraded.md` states that `snapshotHeavyTaskConcurrentLimit` was tightened 5 → 1
+after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is
+**5**. Either the change never landed or it was reverted; the runbook is currently telling an operator
+a knob is set that is not. Out of scope for this branch.