merge: Longhorn alerting measured the wrong things, three ways

(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a
capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the
same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM
snapshot — the kind taken for the full replica rebuild that prod-1's drive
replacement forced — growing +152 MB/h while the filesystem stayed flat. The
rule was therefore unclearable by design, i.e. permanently firing. Deleted as
redundant, shown rather than asserted: 23 of 24 volumes fall under the
kubelet-based filling rules, and the 24th is volumeMode Block, which the
deleted rule was not covering either.

(2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with
the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a
state label with values 0/1, not the 0-3 enum the rules and their comment
assumed, so a faulted volume would have alerted nobody. This survived the
scrape fix landing hours earlier.

(3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the
75% at which storage-minimal-available-percentage=25 already removes a disk
from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's
disks, so one full disk hid behind a half-empty sibling. Replaced with
per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable,
the latter reading Longhorn's own verdict instead of re-deriving a formula
from three settings no metric exports.

Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and
discard/TRIM are both REPORTED with recommendations, not changed: TRIM would
not have helped here, since remove-snapshots-during-filesystem-trim is false
and the 37.7 GiB is a snapshot.

# Conflicts:
#	DECISIONS-LOG.md

DECISIONS-LOG.md

@@ -3332,3 +3332,145 @@ there may be a reason both entries exist. The three options, in the order I woul
    Cheapest, and relies on a comment — which is the mechanism this repo has already watched fail
    (the bulk StorageClass sat as a REPLACE-ME stub behind four hand-written comments).
 `vm-demo` is likely just an undeployed demo and may only need option 3.
+## 2026-09-13 — `LonghornVolumeCapacityHigh` measured ALLOCATION and called it capacity; and the robustness rules were inert a SECOND time (branch `fix/longhorn-capacity-alert-measures-allocation`)
+
+Longhorn metrics began being scraped earlier today. Within minutes `LonghornVolumeCapacityHigh` went
+`pending`. Three separate defects came out of re-measuring it, two of them the exact failure classes
+this repo spent the day eliminating.
+
+### 1. The per-volume "capacity" rule was permanently firing, by design
+
+MEASURED on prod, `kubectl -n monitoring exec … promtool query instant`:
+
+| what | value |
+|---|---|
+| `longhorn_volume_actual_size_bytes` (Prometheus TSDB volume) | 52243681280 (48.66 GiB) |
+| `longhorn_volume_capacity_bytes` | 53687091200 (50 GiB) |
+| the rule's ratio | **0.9731** |
+| next-highest volume in the cluster | 0.2818 (keycloak-db-1) — a lone outlier |
+| `kubelet_volume_stats_used_bytes / …_capacity_bytes`, SAME filesystem | **0.2080** |
+| `longhorn_snapshot_actual_size_bytes{user_created="false"}` on that volume | 40510484480 (**37.7 GiB**) |
+| growth of actual_size, `deriv(…[2h])` | ~152 MB/h, monotonic |
+
+`actual_size` is head + pinned snapshots, not usage. ONE Longhorn system snapshot — the kind taken for
+a full replica rebuild, which prod-1's drive replacement forced — holds 37.7 GiB; the live head is the
+remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn returns no freed blocks without
+discard/TRIM, and `retentionSize: 45GiB` on a 50Gi volume deliberately targets 90%, so the ratio is
+pinned near 1.0 and **cannot fall back under 0.85**. Same shape as `CNPGNoRecoverabilityPoint`.
+
+DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation.
+The redundancy was proven rather than asserted: of 24 Longhorn volumes on prod, 23 appear in
+`kubelet_volume_stats_inodes > 0` and are therefore inside `MonitoringVolumeFillingUp`
+(namespace="monitoring") or `PersistentVolumeFillingUp` / `…AlmostFull` (everything else). The 24th is
+`unifi/unifi-rootdisk`, volumeMode Block — no filesystem for anyone to measure, and the deleted rule
+was not covering it either (0.2097 was raw-device allocation). Block volumes are covered at the pool
+level.
+
+### 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT after the scrape landed
+
+The bigger finding. On longhorn 1.12.1 `longhorn_volume_robustness` is **one-hot over a `state`
+label** with values 0/1 — it is NOT the 0-3 enum the rules and their comment assumed:
+
+```
+count(longhorn_volume_robustness)                  => 96      (24 volumes x 4 states)
+count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
+count(longhorn_volume_robustness == 2)             => (empty)
+count(longhorn_volume_robustness == 3)             => (empty)
+```
+
+So a FAULTED volume — the most serious storage state on this cluster — would have alerted nobody,
+with the scrape working and the series present. The reading recorded in
+`platform/longhorn/helmrelease.yaml` ("44 series … 24 healthy(1), 72 unknown(0, detached)") was this
+same misreading; those zeros are the other three state labels of the same 24 healthy volumes. Both
+rules are now keyed on the label, and a regression case asserts the old `== 2` form matches nothing.
+
+### 3. The node-level allocation alarm fired AFTER the cliff, and averaged disks together
+
+`LonghornNodeStorageHigh` at `> 0.85` was past Longhorn's own limit: `storage-minimal-available-percentage`
+is **25**, so a disk leaves the scheduling pool at 75% used. Re-keyed to the per-disk series (the
+node-level one SUMS a node's disks, so one full disk hides behind a half-empty sibling) and renamed:
+
+* `LonghornDiskAllocationHigh` — `(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`, 30m
+* `LonghornDiskUnschedulable` — `longhorn_disk_status{condition="schedulable"} == 0`, 15m — Longhorn's
+  own verdict, instead of re-deriving a scheduling formula that folds in three settings no metric
+  exports (reserved 20, over-provisioning 200, minimal-available 25).
+
+MEASURED prod usage: 0.2254 / 0.2306 / 0.2369 on prod-1/2/3; 268.9 GiB nominal scheduled per node.
+
+### Proof, both directions, for every rule touched
+
+`platform/monitoring/tests/longhorn.test.yaml` (6 cases, run by
+`platform/monitoring/tests/run.sh`; the L0 register shrank by 7 and the repo is now 14/58 alerts
+unit-tested). Live: all four new expressions return EMPTY on prod, while
+`count(robustness{state="degraded"}) + count(robustness{state="faulted"}) + count(disk ratio) +
+count(disk_status{condition="schedulable"})` returns **54** — quiet, and provably not inert.
+Mutation-checked 11 ways (each turns the suite RED): both old enum expressions, the 0.85 threshold,
+the node-summed metric, a threshold low enough to fire on today's healthy state, the wrong
+`condition` label, three shortened/lengthened `for:` values, and two mutations of the kubelet rules.
+
+### OWNER DECISION 1 — is `retentionSize: 45GiB` on a 50Gi volume right?
+
+**Recommendation: lower it to `36GiB` (about 72% of the volume). Not changed here — retention is how
+much history you keep, so it is your call.**
+
+The reasoning, and it is not about the alert:
+
+* 45GiB leaves 5 GiB of headroom on the volume, and `retentionSize` does **not** count the WAL, the
+  in-memory head's on-disk segments, or compaction scratch. A compaction of a large block can need
+  roughly the size of the blocks being merged. At 90% full, a single bad compaction is
+  "no space left on device", which wedges every WAL commit — scrapes still return 200, nothing is
+  stored, the head freezes, and every `absent()`/`== 0` platform alert fires falsely. The file's own
+  comment says the cap is "kept BELOW the PVC size … for WAL + head + compaction scratch"; 10% is not
+  below enough for that list.
+* The volume is 3x replicated on mini-PCs whose data partition also carries the etcd WAL. Every GiB of
+  TSDB is 3 GiB of allocation across the pool and more rebuild I/O when a replica resyncs — and
+  etcd fsync starvation is this cluster's known way of losing a node (2026-09-07).
+* It is not currently constraining anything: the TSDB is **11.2 GiB** at 30d retention on a 3-node
+  cluster. 36GiB is still triple today's size, so 30d would remain the binding limit in practice and
+  no history would actually be lost.
+* If more history is wanted, grow the PVC (`allowVolumeExpansion: true`) rather than the fraction —
+  the knobs are `MONITORING_PROMETHEUS_PVC_SIZE` and `MONITORING_PROMETHEUS_RETENTION_SIZE` in
+  `clusters/prod/kustomization.yaml`, and the base comment already requires the second to stay below
+  the first.
+
+### OWNER DECISION 2 — enable discard/TRIM on the `longhorn` StorageClass?
+
+**Recommendation: do NOT enable it globally now. It is feasible, but it buys a metric and risks the
+one subsystem this cluster keeps losing.** Not enabled here; it changes behaviour for every Longhorn
+volume on the cluster.
+
+* Feasible. The class already sets `fsType: "ext4"`, and Longhorn passes mount options through, so
+  continuous discard is a StorageClass change (`mountOptions: ["discard"]`) plus a REMOUNT of every
+  volume to take effect — i.e. a restart of every stateful workload. Longhorn also exposes a
+  per-volume `Trim Filesystem` action, which is the same reclaim without touching the class.
+* The setting that actually matters here is `remove-snapshots-during-filesystem-trim`, read live on
+  prod as **false**. With it false, a trim reclaims nothing that a snapshot is holding — so trim, in
+  any form, would NOT have reclaimed the 37.7 GiB. Flipping that one to true is a far larger decision
+  than enabling discard: it makes a routine filesystem trim DELETE SNAPSHOTS, i.e. quietly discard
+  recovery points.
+* Also read live: `auto-cleanup-system-generated-snapshot` is already **true**, and the 37.7 GiB
+  system snapshot is still present — so it is most likely the replica's base snapshot, which cannot be
+  removed while it is the parent of the volume head. If so, no trim setting and no discard flag will
+  release it; only a full replica rebuild onto a fresh file would, which is the expensive operation
+  this cluster is trying to avoid.
+* What it would buy: `actual_size` would track real usage. That is now worth nothing to alerting —
+  no rule reads `actual_size` any more, and the pool-level rules read disk usage, which trim improves
+  only for blocks no snapshot is holding. **The 37.7 GiB that started all this is held by a SNAPSHOT,
+  which discard cannot release** (see the two settings below).
+* What it risks: `discard` makes every filesystem delete issue a synchronous block-level discard on
+  all three replicas, on nodes where `/data/k3s` (etcd WAL) and `/data/longhorn` share one device. The
+  failure mode we already have is etcd fsync starvation under Longhorn write load. Turning on
+  continuous discard for 24 volumes is exactly the wrong direction for that, and a mass remount is a
+  rebuild-storm trigger in its own right (every node reboot degrades all volumes today).
+* If reclaim is wanted later, the safe order is: (1) check whether the big snapshot is a removable
+  snapshot or the replica's base, (2) purge what is removable from the Longhorn UI, (3) per-volume
+  `Trim Filesystem` during a quiet window, (4) only then consider a periodic `fstrim` timer — never
+  continuous `discard`, and never `remove-snapshots-during-filesystem-trim: true`, while etcd shares
+  the spindle. Revisit if/when `docs/plans/longhorn-ssd-tiering.md` separates etcd from Longhorn.
+
+### Also noticed, NOT changed (drive-by, for whoever owns it)
+
+`runbooks/longhorn-degraded.md` states that `snapshotHeavyTaskConcurrentLimit` was tightened 5 → 1
+after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is
+**5**. Either the change never landed or it was reverted; the runbook is currently telling an operator
+a knob is set that is not. Out of scope for this branch.