fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity

LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.

DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.

AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
  count(longhorn_volume_robustness)                  => 96  (24 volumes x 4 states)
  count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
  count(longhorn_volume_robustness == 2)             => (empty)
  count(longhorn_volume_robustness == 3)             => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.

The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
  LonghornDiskAllocationHigh  (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
  LonghornDiskUnschedulable   longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.

PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.

REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 17:19:03 -05:00
1 parent 72f0ba4e1f
commit 4687521d65
11 files changed
+913 -113

No files matched your search

+23 -16
View File
@@ -2,7 +2,7 @@
**Alert:** `LonghornVolumeFaulted` (critical, 1m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 3`
**Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum)
## What this means here
@@ -22,26 +22,33 @@ Understand what is and is not recoverable before you touch anything:
- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
harmless — re-run the import.
### Verify the alert can even fire (measured 2026-09-13)
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a
faulted volume — the most serious storage state on this cluster — alerted nobody, and announced
itself only as an application outage:
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's
`metrics.serviceMonitor.enabled` defaulted false. Fixed in
`platform/longhorn/helmrelease.yaml`.
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
`longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and
`count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label.
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the
live series before trusting this alert's silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'count by (state) (longhorn_volume_robustness)'
```
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group
has no data. A faulted volume today announces itself as an application outage. That is a known open
defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence.
Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this
alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev