merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md
This commit is contained in:
jakman85 committed
commit
52398d4426
11 files changed
+912
-113
No files matched your search
@@ -3332,3 +3332,145 @@ there may be a reason both entries exist. The three options, in the order I woul
|
||||
Cheapest, and relies on a comment — which is the mechanism this repo has already watched fail
|
||||
(the bulk StorageClass sat as a REPLACE-ME stub behind four hand-written comments).
|
||||
`vm-demo` is likely just an undeployed demo and may only need option 3.
|
||||
## 2026-09-13 — `LonghornVolumeCapacityHigh` measured ALLOCATION and called it capacity; and the robustness rules were inert a SECOND time (branch `fix/longhorn-capacity-alert-measures-allocation`)
|
||||
|
||||
Longhorn metrics began being scraped earlier today. Within minutes `LonghornVolumeCapacityHigh` went
|
||||
`pending`. Three separate defects came out of re-measuring it, two of them the exact failure classes
|
||||
this repo spent the day eliminating.
|
||||
|
||||
### 1. The per-volume "capacity" rule was permanently firing, by design
|
||||
|
||||
MEASURED on prod, `kubectl -n monitoring exec … promtool query instant`:
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| `longhorn_volume_actual_size_bytes` (Prometheus TSDB volume) | 52243681280 (48.66 GiB) |
|
||||
| `longhorn_volume_capacity_bytes` | 53687091200 (50 GiB) |
|
||||
| the rule's ratio | **0.9731** |
|
||||
| next-highest volume in the cluster | 0.2818 (keycloak-db-1) — a lone outlier |
|
||||
| `kubelet_volume_stats_used_bytes / …_capacity_bytes`, SAME filesystem | **0.2080** |
|
||||
| `longhorn_snapshot_actual_size_bytes{user_created="false"}` on that volume | 40510484480 (**37.7 GiB**) |
|
||||
| growth of actual_size, `deriv(…[2h])` | ~152 MB/h, monotonic |
|
||||
|
||||
`actual_size` is head + pinned snapshots, not usage. ONE Longhorn system snapshot — the kind taken for
|
||||
a full replica rebuild, which prod-1's drive replacement forced — holds 37.7 GiB; the live head is the
|
||||
remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn returns no freed blocks without
|
||||
discard/TRIM, and `retentionSize: 45GiB` on a 50Gi volume deliberately targets 90%, so the ratio is
|
||||
pinned near 1.0 and **cannot fall back under 0.85**. Same shape as `CNPGNoRecoverabilityPoint`.
|
||||
|
||||
DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation.
|
||||
The redundancy was proven rather than asserted: of 24 Longhorn volumes on prod, 23 appear in
|
||||
`kubelet_volume_stats_inodes > 0` and are therefore inside `MonitoringVolumeFillingUp`
|
||||
(namespace="monitoring") or `PersistentVolumeFillingUp` / `…AlmostFull` (everything else). The 24th is
|
||||
`unifi/unifi-rootdisk`, volumeMode Block — no filesystem for anyone to measure, and the deleted rule
|
||||
was not covering it either (0.2097 was raw-device allocation). Block volumes are covered at the pool
|
||||
level.
|
||||
|
||||
### 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT after the scrape landed
|
||||
|
||||
The bigger finding. On longhorn 1.12.1 `longhorn_volume_robustness` is **one-hot over a `state`
|
||||
label** with values 0/1 — it is NOT the 0-3 enum the rules and their comment assumed:
|
||||
|
||||
```
|
||||
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
||||
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
count(longhorn_volume_robustness == 2) => (empty)
|
||||
count(longhorn_volume_robustness == 3) => (empty)
|
||||
```
|
||||
|
||||
So a FAULTED volume — the most serious storage state on this cluster — would have alerted nobody,
|
||||
with the scrape working and the series present. The reading recorded in
|
||||
`platform/longhorn/helmrelease.yaml` ("44 series … 24 healthy(1), 72 unknown(0, detached)") was this
|
||||
same misreading; those zeros are the other three state labels of the same 24 healthy volumes. Both
|
||||
rules are now keyed on the label, and a regression case asserts the old `== 2` form matches nothing.
|
||||
|
||||
### 3. The node-level allocation alarm fired AFTER the cliff, and averaged disks together
|
||||
|
||||
`LonghornNodeStorageHigh` at `> 0.85` was past Longhorn's own limit: `storage-minimal-available-percentage`
|
||||
is **25**, so a disk leaves the scheduling pool at 75% used. Re-keyed to the per-disk series (the
|
||||
node-level one SUMS a node's disks, so one full disk hides behind a half-empty sibling) and renamed:
|
||||
|
||||
* `LonghornDiskAllocationHigh` — `(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`, 30m
|
||||
* `LonghornDiskUnschedulable` — `longhorn_disk_status{condition="schedulable"} == 0`, 15m — Longhorn's
|
||||
own verdict, instead of re-deriving a scheduling formula that folds in three settings no metric
|
||||
exports (reserved 20, over-provisioning 200, minimal-available 25).
|
||||
|
||||
MEASURED prod usage: 0.2254 / 0.2306 / 0.2369 on prod-1/2/3; 268.9 GiB nominal scheduled per node.
|
||||
|
||||
### Proof, both directions, for every rule touched
|
||||
|
||||
`platform/monitoring/tests/longhorn.test.yaml` (6 cases, run by
|
||||
`platform/monitoring/tests/run.sh`; the L0 register shrank by 7 and the repo is now 14/58 alerts
|
||||
unit-tested). Live: all four new expressions return EMPTY on prod, while
|
||||
`count(robustness{state="degraded"}) + count(robustness{state="faulted"}) + count(disk ratio) +
|
||||
count(disk_status{condition="schedulable"})` returns **54** — quiet, and provably not inert.
|
||||
Mutation-checked 11 ways (each turns the suite RED): both old enum expressions, the 0.85 threshold,
|
||||
the node-summed metric, a threshold low enough to fire on today's healthy state, the wrong
|
||||
`condition` label, three shortened/lengthened `for:` values, and two mutations of the kubelet rules.
|
||||
|
||||
### OWNER DECISION 1 — is `retentionSize: 45GiB` on a 50Gi volume right?
|
||||
|
||||
**Recommendation: lower it to `36GiB` (about 72% of the volume). Not changed here — retention is how
|
||||
much history you keep, so it is your call.**
|
||||
|
||||
The reasoning, and it is not about the alert:
|
||||
|
||||
* 45GiB leaves 5 GiB of headroom on the volume, and `retentionSize` does **not** count the WAL, the
|
||||
in-memory head's on-disk segments, or compaction scratch. A compaction of a large block can need
|
||||
roughly the size of the blocks being merged. At 90% full, a single bad compaction is
|
||||
"no space left on device", which wedges every WAL commit — scrapes still return 200, nothing is
|
||||
stored, the head freezes, and every `absent()`/`== 0` platform alert fires falsely. The file's own
|
||||
comment says the cap is "kept BELOW the PVC size … for WAL + head + compaction scratch"; 10% is not
|
||||
below enough for that list.
|
||||
* The volume is 3x replicated on mini-PCs whose data partition also carries the etcd WAL. Every GiB of
|
||||
TSDB is 3 GiB of allocation across the pool and more rebuild I/O when a replica resyncs — and
|
||||
etcd fsync starvation is this cluster's known way of losing a node (2026-09-07).
|
||||
* It is not currently constraining anything: the TSDB is **11.2 GiB** at 30d retention on a 3-node
|
||||
cluster. 36GiB is still triple today's size, so 30d would remain the binding limit in practice and
|
||||
no history would actually be lost.
|
||||
* If more history is wanted, grow the PVC (`allowVolumeExpansion: true`) rather than the fraction —
|
||||
the knobs are `MONITORING_PROMETHEUS_PVC_SIZE` and `MONITORING_PROMETHEUS_RETENTION_SIZE` in
|
||||
`clusters/prod/kustomization.yaml`, and the base comment already requires the second to stay below
|
||||
the first.
|
||||
|
||||
### OWNER DECISION 2 — enable discard/TRIM on the `longhorn` StorageClass?
|
||||
|
||||
**Recommendation: do NOT enable it globally now. It is feasible, but it buys a metric and risks the
|
||||
one subsystem this cluster keeps losing.** Not enabled here; it changes behaviour for every Longhorn
|
||||
volume on the cluster.
|
||||
|
||||
* Feasible. The class already sets `fsType: "ext4"`, and Longhorn passes mount options through, so
|
||||
continuous discard is a StorageClass change (`mountOptions: ["discard"]`) plus a REMOUNT of every
|
||||
volume to take effect — i.e. a restart of every stateful workload. Longhorn also exposes a
|
||||
per-volume `Trim Filesystem` action, which is the same reclaim without touching the class.
|
||||
* The setting that actually matters here is `remove-snapshots-during-filesystem-trim`, read live on
|
||||
prod as **false**. With it false, a trim reclaims nothing that a snapshot is holding — so trim, in
|
||||
any form, would NOT have reclaimed the 37.7 GiB. Flipping that one to true is a far larger decision
|
||||
than enabling discard: it makes a routine filesystem trim DELETE SNAPSHOTS, i.e. quietly discard
|
||||
recovery points.
|
||||
* Also read live: `auto-cleanup-system-generated-snapshot` is already **true**, and the 37.7 GiB
|
||||
system snapshot is still present — so it is most likely the replica's base snapshot, which cannot be
|
||||
removed while it is the parent of the volume head. If so, no trim setting and no discard flag will
|
||||
release it; only a full replica rebuild onto a fresh file would, which is the expensive operation
|
||||
this cluster is trying to avoid.
|
||||
* What it would buy: `actual_size` would track real usage. That is now worth nothing to alerting —
|
||||
no rule reads `actual_size` any more, and the pool-level rules read disk usage, which trim improves
|
||||
only for blocks no snapshot is holding. **The 37.7 GiB that started all this is held by a SNAPSHOT,
|
||||
which discard cannot release** (see the two settings below).
|
||||
* What it risks: `discard` makes every filesystem delete issue a synchronous block-level discard on
|
||||
all three replicas, on nodes where `/data/k3s` (etcd WAL) and `/data/longhorn` share one device. The
|
||||
failure mode we already have is etcd fsync starvation under Longhorn write load. Turning on
|
||||
continuous discard for 24 volumes is exactly the wrong direction for that, and a mass remount is a
|
||||
rebuild-storm trigger in its own right (every node reboot degrades all volumes today).
|
||||
* If reclaim is wanted later, the safe order is: (1) check whether the big snapshot is a removable
|
||||
snapshot or the replica's base, (2) purge what is removable from the Longhorn UI, (3) per-volume
|
||||
`Trim Filesystem` during a quiet window, (4) only then consider a periodic `fstrim` timer — never
|
||||
continuous `discard`, and never `remove-snapshots-during-filesystem-trim: true`, while etcd shares
|
||||
the spindle. Revisit if/when `docs/plans/longhorn-ssd-tiering.md` separates etcd from Longhorn.
|
||||
|
||||
### Also noticed, NOT changed (drive-by, for whoever owns it)
|
||||
|
||||
`runbooks/longhorn-degraded.md` states that `snapshotHeavyTaskConcurrentLimit` was tightened 5 → 1
|
||||
after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is
|
||||
**5**. Either the change never landed or it was reverted; the runbook is currently telling an operator
|
||||
a knob is set that is not. Out of scope for this branch.
|
||||
@@ -168,8 +168,12 @@ UNTESTED_FILES = {
|
||||
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
||||
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
||||
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
||||
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
||||
# rules file by the suite convention but has no assertions of its own yet.
|
||||
# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
|
||||
# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
|
||||
# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
|
||||
# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
|
||||
# redundancy meant asserting on the rules that survive). Only
|
||||
# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -183,10 +187,6 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
||||
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
||||
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
@@ -198,13 +198,10 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
||||
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
||||
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
||||
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
||||
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
||||
}
|
||||
|
||||
|
||||
@@ -103,9 +103,9 @@ spec:
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
|
||||
# platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
|
||||
# LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
|
||||
# and NONE of them could ever fire, because the chart defaults
|
||||
# platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
|
||||
# LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
|
||||
# LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
|
||||
# `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
|
||||
# VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
|
||||
# `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
|
||||
@@ -140,12 +140,26 @@ spec:
|
||||
# colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
|
||||
# count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
|
||||
# count({__name__=~"longhorn.*"}) => 8717 (was EMPTY)
|
||||
# count(longhorn_volume_robustness) => 44
|
||||
# count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached)
|
||||
# and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
|
||||
# them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
|
||||
# volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
|
||||
# freed blocks. Reported, NOT silenced by moving the threshold).
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 `state` labels)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
|
||||
# "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
|
||||
# metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
|
||||
# `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
|
||||
# zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
|
||||
# volumes. The consequence was that enabling this scrape did NOT un-inert
|
||||
# LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
|
||||
# Fixed in platform/monitoring/alerts.yaml and pinned by
|
||||
# platform/monitoring/tests/longhorn.test.yaml.
|
||||
#
|
||||
# Of the rules that WERE evaluating for real, one immediately found a true positive and one
|
||||
# permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
|
||||
# Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
|
||||
# not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
|
||||
# replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
|
||||
# it measured allocation and called it capacity, the filesystem question is already answered by
|
||||
# kubelet_volume_stats_*, and the allocation question now lives per disk as
|
||||
# LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
|
||||
+116
-12
@@ -147,9 +147,33 @@ spec:
|
||||
groups:
|
||||
- name: platform-health.longhorn
|
||||
rules:
|
||||
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
|
||||
# -------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
||||
#
|
||||
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
||||
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
||||
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
||||
# label and its value is only ever 0 or 1:
|
||||
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
||||
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
||||
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
||||
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
||||
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
||||
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
||||
# count(longhorn_volume_robustness == 2) => (empty)
|
||||
# count(longhorn_volume_robustness == 3) => (empty)
|
||||
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
||||
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
||||
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
||||
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
||||
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
||||
#
|
||||
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
||||
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornVolumeDegraded
|
||||
expr: longhorn_volume_robustness == 2
|
||||
expr: longhorn_volume_robustness{state="degraded"} == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -158,7 +182,7 @@ spec:
|
||||
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
- alert: LonghornVolumeFaulted
|
||||
expr: longhorn_volume_robustness == 3
|
||||
expr: longhorn_volume_robustness{state="faulted"} == 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -166,23 +190,103 @@ spec:
|
||||
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- alert: LonghornVolumeCapacityHigh
|
||||
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
|
||||
# -------------------------------------------------------------------------------------
|
||||
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
||||
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
||||
#
|
||||
# The deleted rule was
|
||||
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
||||
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
||||
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
||||
# MEASURED on prod 2026-09-13:
|
||||
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
||||
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
||||
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
||||
# ratio 0.9731 <- what the alert saw
|
||||
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
||||
# the SAME filesystem, from the kubelet:
|
||||
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
||||
#
|
||||
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
||||
# every snapshot still pinned. On this volume,
|
||||
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
||||
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
||||
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
||||
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
||||
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
||||
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
||||
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
||||
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
||||
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
||||
# to ignore the storage row, which is the row that matters on this cluster.
|
||||
#
|
||||
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
||||
# asserted:
|
||||
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
||||
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
||||
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
||||
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
||||
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
||||
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
||||
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
||||
# Block volumes are covered at the pool level, below.
|
||||
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
||||
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
||||
# against its own nominal size, where it is a tautology.
|
||||
# -------------------------------------------------------------------------------------
|
||||
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
||||
#
|
||||
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
||||
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
||||
#
|
||||
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
||||
# replica files plus their pinned snapshots — because that is what consumes the node's
|
||||
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
||||
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
||||
# "disk allocation high" says what it measures.
|
||||
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
||||
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
||||
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
||||
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
||||
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
||||
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
||||
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
||||
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
||||
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
||||
# gives lead time on the way to Longhorn's own limit.
|
||||
#
|
||||
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
||||
# node capacity usage ratio scheduled (nominal)
|
||||
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
||||
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
||||
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
||||
# -------------------------------------------------------------------------------------
|
||||
- alert: LonghornDiskAllocationHigh
|
||||
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
|
||||
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
|
||||
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- alert: LonghornNodeStorageHigh
|
||||
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
|
||||
for: 30m
|
||||
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
||||
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
||||
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
||||
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
||||
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
||||
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
||||
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
||||
# a longhorn-manager restarts.
|
||||
- alert: LonghornDiskUnschedulable
|
||||
expr: longhorn_disk_status{condition="schedulable"} == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
|
||||
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
|
||||
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
||||
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
@@ -10,12 +10,26 @@
|
||||
# the SSO database with nothing watching it.
|
||||
#
|
||||
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
|
||||
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
|
||||
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
|
||||
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
|
||||
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
|
||||
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
|
||||
# working on both clusters.
|
||||
# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two
|
||||
# different reasons in sequence, and it is now GONE:
|
||||
# * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified
|
||||
# read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule
|
||||
# group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in
|
||||
# platform/longhorn/helmrelease.yaml.
|
||||
# * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It
|
||||
# divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head
|
||||
# plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and
|
||||
# called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet
|
||||
# read the same filesystem 20.8% used — by design and permanently, because `retentionSize:
|
||||
# 45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the
|
||||
# comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now
|
||||
# lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
|
||||
# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to
|
||||
# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`.
|
||||
# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in
|
||||
# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in
|
||||
# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no
|
||||
# filesystem for anyone to measure (see the inodes guard below).
|
||||
#
|
||||
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
|
||||
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
|
||||
|
||||
@@ -0,0 +1,465 @@
|
||||
# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml)
|
||||
# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity"
|
||||
# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file,
|
||||
# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml).
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THIS FILE IS FOR
|
||||
# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for
|
||||
# the first time. Two things became visible within minutes:
|
||||
# 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the
|
||||
# kubelet reported the same filesystem 20.8% used. The rule was named and documented as a
|
||||
# CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can
|
||||
# never clear, so it was a PERMANENTLY-FIRING alert. It is deleted.
|
||||
# 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series
|
||||
# present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state`
|
||||
# label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed.
|
||||
# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule
|
||||
# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a
|
||||
# constructed bad state, with the `for:` durations asserted as real.
|
||||
#
|
||||
# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so
|
||||
# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the
|
||||
# literal bytes measured on prod 2026-09-13. Both are noted at the point of use.
|
||||
#
|
||||
# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so
|
||||
# nothing is ever stale inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT.
|
||||
# This is the "not permanently firing" half of the proof for every rule below, and it is the
|
||||
# test that would have caught the deleted rule: the SAME fixture carries the real
|
||||
# actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted
|
||||
# expression evaluating to 0.9731 on a volume whose filesystem is 20% used.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet
|
||||
input_series:
|
||||
# --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) ---
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
# --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod ---
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) ---
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '100326506496+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '456002658304+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
|
||||
values: '105149128704+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '445098295296+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '105464528896+0x120'
|
||||
# --- Longhorn's own scheduling verdict: all three disks accepting replicas ---
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
# `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition`
|
||||
# selector is load-bearing. Drop the selector and the rule would still be quiet on THIS
|
||||
# fixture, so test 5 constructs the case where it matters.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
# --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free ---
|
||||
# Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080.
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '42949672960+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod)
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '4993977548+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
alert_rule_test:
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
# The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the
|
||||
# per-volume Longhorn rule does not trade a false alarm for a blind spot on this state.
|
||||
- eval_time: 2h
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 2h
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
# …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the
|
||||
# whole argument for the deletion, and it is now an assertion rather than a paragraph.
|
||||
- expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
value: 0.2
|
||||
# THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) …
|
||||
- expr: count(longhorn_volume_robustness)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 8
|
||||
# … and the OLD expressions still match nothing on this metric shape. This is the regression
|
||||
# guard: if someone "simplifies" the rules back to a numeric enum, these two go red.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
- expr: count(longhorn_volume_robustness == 3) or vector(-1)
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
# The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules
|
||||
# are quiet because the state is healthy, not because they select nothing.
|
||||
- expr: count(longhorn_volume_robustness{state="degraded"})
|
||||
eval_time: 1h
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`.
|
||||
# Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 10m` is real: nothing at 5m.
|
||||
- eval_time: 5m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-2
|
||||
pvc: keycloak-db-1
|
||||
pvc_namespace: keycloak
|
||||
volume: pvc-f30d3eb7
|
||||
state: degraded
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume degraded: pvc-f30d3eb7"
|
||||
description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
||||
# Degraded must not masquerade as faulted: the critical tier stays silent.
|
||||
- eval_time: 30m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The pre-fix expression, on the state it was written for. Empty. This is the inert rule that
|
||||
# reached prod, reproduced offline.
|
||||
- expr: count(longhorn_volume_robustness == 2) or vector(-1)
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: -1
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one volume faulted - Faulted fires after 1m, Degraded stays quiet
|
||||
input_series:
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 1m` is real: pending, not firing, at the first evaluation.
|
||||
- eval_time: 0m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeFaulted
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
pvc: nextcloud-db-1
|
||||
pvc_namespace: nextcloud-db
|
||||
volume: pvc-d6621428
|
||||
state: faulted
|
||||
exp_annotations:
|
||||
summary: "Longhorn volume FAULTED: pvc-d6621428"
|
||||
description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
||||
- eval_time: 10m
|
||||
alertname: LonghornVolumeDegraded
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK.
|
||||
# Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the
|
||||
# promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the
|
||||
# node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level
|
||||
# form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it
|
||||
input_series:
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
values: '288000000000+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
values: '92000000000+0x120'
|
||||
# Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires
|
||||
# on its own, ahead of the terminal signal tested in case 5.
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
|
||||
values: '1+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 30m` is real: nothing at 20m.
|
||||
- eval_time: 20m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-1
|
||||
disk: default-disk-new
|
||||
exp_annotations:
|
||||
summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new"
|
||||
description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 45m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# Per disk: one at 0.72, one at 0.23.
|
||||
- expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
|
||||
value: 0.23
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
|
||||
value: 0.72
|
||||
# Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent
|
||||
# at the old 0.85. That is the blind spot the metric change closes.
|
||||
- expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes)
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 0.475
|
||||
|
||||
# ===========================================================================================
|
||||
# 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires.
|
||||
# The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so
|
||||
# a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the
|
||||
# schedulable series only by luck of which conditions happen to exist. Allocation is nowhere
|
||||
# near the threshold, so this rule is proven to stand on its own signal.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone
|
||||
input_series:
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
|
||||
values: '0+0x120'
|
||||
- series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}'
|
||||
values: '1+0x120'
|
||||
- series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '400000000000+0x120'
|
||||
- series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
|
||||
values: '92000000000+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart.
|
||||
- eval_time: 10m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskUnschedulable
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: longhorn-backend
|
||||
namespace: longhorn-system
|
||||
node: k3s-prod-3
|
||||
disk: default-disk-59b6657d
|
||||
condition: schedulable
|
||||
exp_annotations:
|
||||
summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d"
|
||||
description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
||||
- eval_time: 30m
|
||||
alertname: LonghornDiskAllocationHigh
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule
|
||||
# fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the
|
||||
# allocation number cannot tell these two situations apart, and the kubelet number can. So the
|
||||
# signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for
|
||||
# the monitoring namespace and by the pvc-capacity rules for every other namespace.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged
|
||||
input_series:
|
||||
# allocation IDENTICAL to the healthy case — 0.9731, indistinguishable
|
||||
- series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '52243681280+0x120'
|
||||
- series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
values: '53687091200+0x120'
|
||||
# monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15%
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '53687091200+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
|
||||
values: '3276800+0x120'
|
||||
# a non-monitoring Longhorn PVC: 10% free -> warning tier
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '5368709120+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '536870912+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
|
||||
values: '327680+0x120'
|
||||
# another one at 3% free -> critical tier as well as warning
|
||||
- series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '10737418240+0x120'
|
||||
- series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '322122547+0x120'
|
||||
- series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
|
||||
values: '655360+0x120'
|
||||
alert_rule_test:
|
||||
# `for: 15m` on all three: nothing at 10m.
|
||||
- eval_time: 10m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: MonitoringVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: monitoring
|
||||
node: k3s-prod-1
|
||||
persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
|
||||
exp_annotations:
|
||||
summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
|
||||
description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
||||
- eval_time: 10m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts: []
|
||||
# Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the
|
||||
# exact-partition property pvc-capacity-alerts.yaml documents.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeFillingUp
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kubelet
|
||||
namespace: keycloak
|
||||
node: k3s-prod-2
|
||||
persistentvolumeclaim: keycloak-db-1
|
||||
exp_annotations:
|
||||
summary: "PVC filling up: keycloak/keycloak-db-1"
|
||||
description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# Only the 3%-free one reaches the critical tier — the 10% one must not.
|
||||
- eval_time: 30m
|
||||
alertname: PersistentVolumeAlmostFull
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
job: kubelet
|
||||
namespace: gitea
|
||||
node: k3s-prod-3
|
||||
persistentvolumeclaim: gitea-data
|
||||
exp_annotations:
|
||||
summary: "PVC almost full: gitea/gitea-data"
|
||||
description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
promql_expr_test:
|
||||
# The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the
|
||||
# filesystem went from 80% free to 10% free. One number moved; the other cannot.
|
||||
- expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
|
||||
eval_time: 30m
|
||||
exp_samples:
|
||||
- labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
|
||||
value: 0.9731143951416016
|
||||
+7
-4
@@ -37,12 +37,15 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||||
|
||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||
|
||||
Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or
|
||||
Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**:
|
||||
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
|
||||
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
|
||||
scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the
|
||||
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
|
||||
rule groups still evaluate against nothing and **cannot fire**:
|
||||
|
||||
| Missing scrape job | Inert alerts |
|
||||
|---|---|
|
||||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
|
||||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
|
||||
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
||||
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
||||
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
||||
@@ -65,7 +68,7 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
|
||||
expensive part. Freeze/thaw commands here.
|
||||
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
|
||||
- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity.
|
||||
- **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`).
|
||||
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
|
||||
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
|
||||
|
||||
|
||||
@@ -1,47 +1,62 @@
|
||||
# longhorn-capacity — A Longhorn volume or node is near its storage capacity
|
||||
# longhorn-capacity — A Longhorn disk is running out of room for replicas
|
||||
|
||||
**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m)
|
||||
**Alerts:** `LonghornDiskAllocationHigh` (warning, 30m) and `LonghornDiskUnschedulable` (warning, 15m)
|
||||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||||
**Expressions:**
|
||||
`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85`
|
||||
`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85`
|
||||
`(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`
|
||||
`longhorn_disk_status{condition="schedulable"} == 0`
|
||||
|
||||
## What this means here
|
||||
|
||||
Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different
|
||||
problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC;
|
||||
`node` means the cluster is running out of places to put replicas.
|
||||
**This is a POOL question, not a PVC question.** One of the three nodes' Longhorn data disks is
|
||||
filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert
|
||||
says anything about how full any single workload's filesystem is — for that, read
|
||||
`/runbooks/pvc-capacity` (`PersistentVolumeFillingUp`, `PersistentVolumeAlmostFull`) or
|
||||
`/runbooks/monitoring-capacity` (`MonitoringVolumeFillingUp`). Those are keyed on
|
||||
`kubelet_volume_stats_*`, which is the filesystem's own accounting.
|
||||
|
||||
The node case is bounded by settings that are already conservative:
|
||||
`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`,
|
||||
`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity.
|
||||
Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node
|
||||
alert firing means something has changed materially, not that the platform has slowly grown.
|
||||
`longhorn_disk_usage_bytes` counts bytes ALLOCATED: every replica file plus every snapshot still
|
||||
pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a
|
||||
system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only
|
||||
goes up until something is deleted. That is the correct question *here* — allocated bytes are what
|
||||
consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why
|
||||
`LonghornVolumeCapacityHigh` was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume
|
||||
whose filesystem was 20.8% used, by design, permanently). See the comment block in
|
||||
`platform/monitoring/alerts.yaml` for the measurements.
|
||||
|
||||
The severe consequence of running out of node storage is that a degraded volume **cannot finish
|
||||
rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a
|
||||
volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`.
|
||||
### Verify the alert can even fire (measured 2026-09-13)
|
||||
**Why 0.70 and not 0.85.** `storageMinimalAvailablePercentage: 25` means Longhorn itself refuses to
|
||||
schedule a new replica on a disk with under 25% available — i.e. at **75% used the disk is already
|
||||
out of the scheduling pool**. An 85% threshold could only ever fire after that had happened. 0.70 is
|
||||
the last point at which there is still lead time. The companion settings are
|
||||
`storageReservedPercentageForDefaultDisk: 20`, `storageOverProvisioningPercentage: 200`,
|
||||
`defaultDataLocality: disabled`, hard node anti-affinity.
|
||||
|
||||
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
|
||||
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
|
||||
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
|
||||
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
|
||||
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
|
||||
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
|
||||
rule depends on before you trust its silence:
|
||||
Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal
|
||||
replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means
|
||||
something changed materially, not that the platform slowly grew.
|
||||
|
||||
The severe consequence is that a degraded volume **cannot finish rebuilding** — hard anti-affinity
|
||||
needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas
|
||||
indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts.
|
||||
Cross-read `/runbooks/longhorn-degraded`.
|
||||
|
||||
### Verify the alert can fire (measured 2026-09-13)
|
||||
|
||||
These alerts were INERT until 2026-09-13, when `metrics.serviceMonitor.enabled` was set in
|
||||
`platform/longhorn/helmrelease.yaml`; the longhorn_* family was simply absent from the TSDB before
|
||||
that. Both are now live — verified on prod the same day:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
|
||||
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
|
||||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 \
|
||||
'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})'
|
||||
```
|
||||
|
||||
**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and
|
||||
`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 —
|
||||
Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from
|
||||
`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts
|
||||
are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.
|
||||
returned **6** (3 disks, both selectors resolving), while both alert expressions returned empty.
|
||||
If that count is ever empty again, the scrape has broken and this pair is silently back to inert —
|
||||
check `up{job="longhorn-backend"}` and `platform/longhorn/networkpolicy-metrics.yaml`.
|
||||
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||
@@ -73,14 +88,25 @@ Where this runbook gives a mutating command it is for an operator, and it says s
|
||||
|
||||
## First commands
|
||||
|
||||
Node-level allocation (the half that has no substitute alert):
|
||||
Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 \
|
||||
'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status'
|
||||
```
|
||||
|
||||
Node-level allocation, from the CRs:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
|
||||
```
|
||||
|
||||
Volume sizes versus actual usage:
|
||||
Volume nominal size versus ALLOCATED size. `actualSize` is head + pinned snapshots, NOT filesystem
|
||||
usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note
|
||||
above), so do not read this column as "full":
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
@@ -95,7 +121,17 @@ infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
|
||||
```
|
||||
|
||||
The filesystem view, which is the alert that actually works:
|
||||
Which snapshots are holding the allocation (this is the measured top cause on prod — one
|
||||
`user_created=false` system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13):
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 \
|
||||
'topk(10, longhorn_snapshot_actual_size_bytes)'
|
||||
```
|
||||
|
||||
The filesystem view — the separate question, answered by `/runbooks/pvc-capacity`:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
@@ -111,7 +147,15 @@ ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
### 1. A CNPG database is not recycling WAL (check this FIRST)
|
||||
### 0. Pinned snapshots, not live data (check this FIRST for an allocation alert)
|
||||
An allocation alert can fire with every filesystem in the cluster nearly empty. Run the
|
||||
`longhorn_snapshot_actual_size_bytes` query above before anything else. A replica rebuild — which is
|
||||
what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 —
|
||||
leaves a `user_created=false` system snapshot behind, and it keeps its blocks until it is purged.
|
||||
Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if
|
||||
this was the cause.
|
||||
|
||||
### 1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert)
|
||||
The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
|
||||
Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
|
||||
sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
|
||||
@@ -144,8 +188,9 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete.
|
||||
Free space or restore the other node before anything else.
|
||||
- **A disk over 70% while any volume is degraded is urgent**, because the rebuild cannot complete —
|
||||
and at 75% Longhorn takes the disk out of the scheduling pool entirely
|
||||
(`LonghornDiskUnschedulable`). Free space or restore the other node before anything else.
|
||||
- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
|
||||
- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
|
||||
the real fault.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Alert:** `LonghornVolumeDegraded` (warning, 10m)
|
||||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||||
**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted)
|
||||
**Expression:** `longhorn_volume_robustness{state="degraded"} == 1` (one-hot over `state`, NOT a 0-3 enum)
|
||||
|
||||
## What this means here
|
||||
|
||||
@@ -20,27 +20,34 @@ rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s`
|
||||
share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
|
||||
the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
|
||||
out". Cross-read `/runbooks/node-io`.
|
||||
### Verify the alert can even fire (measured 2026-09-13)
|
||||
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
|
||||
|
||||
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
|
||||
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
|
||||
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
|
||||
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
|
||||
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
|
||||
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
|
||||
rule depends on before you trust its silence:
|
||||
This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was
|
||||
enough to make a faulted or degraded volume alert nobody:
|
||||
|
||||
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series — the longhorn_* family was
|
||||
absent from the TSDB because the chart's `metrics.serviceMonitor.enabled` defaulted false. Fixed
|
||||
in `platform/longhorn/helmrelease.yaml`.
|
||||
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
|
||||
`longhorn_volume_robustness == 2` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
|
||||
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
|
||||
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states),
|
||||
`count by (state) (longhorn_volume_robustness == 1)` = `{state="healthy"} 24`, and
|
||||
`count(longhorn_volume_robustness == 2)` = **empty**. Fixed by keying on the label.
|
||||
|
||||
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`, which keeps
|
||||
a regression case asserting the old `== 2` form matches nothing. Check the live series before
|
||||
trusting this alert's silence:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
|
||||
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
|
||||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 \
|
||||
'count by (state) (longhorn_volume_robustness)'
|
||||
```
|
||||
|
||||
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
|
||||
on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no
|
||||
ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will
|
||||
probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not
|
||||
from this alert**. Do not infer "no alert, so volumes are fine".
|
||||
Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert
|
||||
again — do not infer "no alert, so volumes are fine".
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Alert:** `LonghornVolumeFaulted` (critical, 1m)
|
||||
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
|
||||
**Expression:** `longhorn_volume_robustness == 3`
|
||||
**Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum)
|
||||
|
||||
## What this means here
|
||||
|
||||
@@ -22,26 +22,33 @@ Understand what is and is not recoverable before you touch anything:
|
||||
- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
|
||||
disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
|
||||
harmless — re-run the import.
|
||||
### Verify the alert can even fire (measured 2026-09-13)
|
||||
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
|
||||
|
||||
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
|
||||
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
|
||||
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
|
||||
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
|
||||
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
|
||||
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
|
||||
rule depends on before you trust its silence:
|
||||
This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a
|
||||
faulted volume — the most serious storage state on this cluster — alerted nobody, and announced
|
||||
itself only as an application outage:
|
||||
|
||||
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's
|
||||
`metrics.serviceMonitor.enabled` defaulted false. Fixed in
|
||||
`platform/longhorn/helmrelease.yaml`.
|
||||
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
|
||||
`longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
|
||||
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
|
||||
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and
|
||||
`count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label.
|
||||
|
||||
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the
|
||||
live series before trusting this alert's silence:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
|
||||
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
|
||||
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
||||
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
||||
/bin/promtool query instant http://localhost:9090 \
|
||||
'count by (state) (longhorn_volume_robustness)'
|
||||
```
|
||||
|
||||
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
|
||||
on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group
|
||||
has no data. A faulted volume today announces itself as an application outage. That is a known open
|
||||
defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence.
|
||||
Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this
|
||||
alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence.
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
||||
|
||||
@@ -25,7 +25,7 @@ of 2026-09-13 — every one of them means the corresponding alerts are **inert**
|
||||
|
||||
| Missing scrape job | Alerts that cannot fire |
|
||||
|---|---|
|
||||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
|
||||
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
|
||||
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
||||
| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
||||
| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
||||
@@ -129,8 +129,9 @@ An exporter that is up but publishing the wrong series is the hardest case — s
|
||||
monitoring one.
|
||||
- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
|
||||
chart value). Do not "fix" a blind spot by silencing the alert.
|
||||
- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to
|
||||
hand-apply monitors into the cluster — every durable change goes through git.
|
||||
- The Longhorn and cert-manager scrapes were enabled on 2026-09-13 (the Longhorn row above is now
|
||||
history). The Flux-controller and Hubble/Cilium monitor gaps are still open work. Note them; do not
|
||||
try to hand-apply monitors into the cluster — every durable change goes through git.
|
||||
|
||||
---
|
||||
Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`
|
||||
|
||||
Reference in new issue
Block a user