merge: Longhorn alerting measured the wrong things, three ways
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a
capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the
same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM
snapshot — the kind taken for the full replica rebuild that prod-1's drive
replacement forced — growing +152 MB/h while the filesystem stayed flat. The
rule was therefore unclearable by design, i.e. permanently firing. Deleted as
redundant, shown rather than asserted: 23 of 24 volumes fall under the
kubelet-based filling rules, and the 24th is volumeMode Block, which the
deleted rule was not covering either.

(2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with
the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a
state label with values 0/1, not the 0-3 enum the rules and their comment
assumed, so a faulted volume would have alerted nobody. This survived the
scrape fix landing hours earlier.

(3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the
75% at which storage-minimal-available-percentage=25 already removes a disk
from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's
disks, so one full disk hid behind a half-empty sibling. Replaced with
per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable,
the latter reading Longhorn's own verdict instead of re-deriving a formula
from three settings no metric exports.

Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and
discard/TRIM are both REPORTED with recommendations, not changed: TRIM would
not have helped here, since remove-snapshots-during-filesystem-trim is false
and the 37.7 GiB is a snapshot.

# Conflicts:
#	DECISIONS-LOG.md
This commit is contained in:
jakman85 committed 2026-09-13 17:23:37 -05:00
commit 52398d4426
11 files changed
+912 -113

No files matched your search

@@ -168,8 +168,12 @@ UNTESTED_FILES = {
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
# already-"tested" file untested: anything not listed here must appear in a covering test.
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
# rules file by the suite convention but has no assertions of its own yet.
# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
# redundancy meant asserting on the rules that survive). Only
# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
UNTESTED_ALERTS = {
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -183,10 +187,6 @@ UNTESTED_ALERTS = {
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
@@ -198,13 +198,10 @@ UNTESTED_ALERTS = {
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
}