Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity

Browse Source 
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.

DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.

AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
  count(longhorn_volume_robustness)                  => 96  (24 volumes x 4 states)
  count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
  count(longhorn_volume_robustness == 2)             => (empty)
  count(longhorn_volume_robustness == 3)             => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.

The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
  LonghornDiskAllocationHigh  (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
  LonghornDiskUnschedulable   longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.

PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.

REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 17:19:03 -05:00
1 parent 72f0ba4e1f
commit 4687521d65
11 files changed
+913 -113
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

runbooks/README.md
+7 -4
View File
@@ -37,12 +37,15 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
## Alerts you should not trust the silence of (measured 2026-09-13)
Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or
Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**:
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
rule groups still evaluate against nothing and **cannot fire**:
| Missing scrape job | Inert alerts |
|---|---|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
@@ -65,7 +68,7 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the
expensive part. Freeze/thaw commands here.
- **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive.
- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity.
- **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`).
- **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work.
- **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself.
runbooks/longhorn-capacity.md
+82 -37
View File
@@ -1,47 +1,62 @@
# longhorn-capacity — A Longhorn volume or node is near its storage capacity
# longhorn-capacity — A Longhorn disk is running out of room for replicas
**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m)
**Alerts:** `LonghornDiskAllocationHigh` (warning, 30m) and `LonghornDiskUnschedulable` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expressions:**
`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85`
`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85`
`(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`
`longhorn_disk_status{condition="schedulable"} == 0`
## What this means here
Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different
problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC;
`node` means the cluster is running out of places to put replicas.
**This is a POOL question, not a PVC question.** One of the three nodes' Longhorn data disks is
filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert
says anything about how full any single workload's filesystem is — for that, read
`/runbooks/pvc-capacity` (`PersistentVolumeFillingUp`, `PersistentVolumeAlmostFull`) or
`/runbooks/monitoring-capacity` (`MonitoringVolumeFillingUp`). Those are keyed on
`kubelet_volume_stats_*`, which is the filesystem's own accounting.
The node case is bounded by settings that are already conservative:
`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`,
`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity.
Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node
alert firing means something has changed materially, not that the platform has slowly grown.
`longhorn_disk_usage_bytes` counts bytes ALLOCATED: every replica file plus every snapshot still
pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a
system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only
goes up until something is deleted. That is the correct question *here* — allocated bytes are what
consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why
`LonghornVolumeCapacityHigh` was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume
whose filesystem was 20.8% used, by design, permanently). See the comment block in
`platform/monitoring/alerts.yaml` for the measurements.
The severe consequence of running out of node storage is that a degraded volume **cannot finish
rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a
volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`.
### Verify the alert can even fire (measured 2026-09-13)
**Why 0.70 and not 0.85.** `storageMinimalAvailablePercentage: 25` means Longhorn itself refuses to
schedule a new replica on a disk with under 25% available — i.e. at **75% used the disk is already
out of the scheduling pool**. An 85% threshold could only ever fire after that had happened. 0.70 is
the last point at which there is still lead time. The companion settings are
`storageReservedPercentageForDefaultDisk: 20`, `storageOverProvisioningPercentage: 200`,
`defaultDataLocality: disabled`, hard node anti-affinity.
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal
replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means
something changed materially, not that the platform slowly grew.
The severe consequence is that a degraded volume **cannot finish rebuilding** — hard anti-affinity
needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas
indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts.
Cross-read `/runbooks/longhorn-degraded`.
### Verify the alert can fire (measured 2026-09-13)
These alerts were INERT until 2026-09-13, when `metrics.serviceMonitor.enabled` was set in
`platform/longhorn/helmrelease.yaml`; the longhorn_* family was simply absent from the TSDB before
that. Both are now live — verified on prod the same day:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})'
```
**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and
`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 —
Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from
`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts
are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand.
returned **6** (3 disks, both selectors resolving), while both alert expressions returned empty.
If that count is ever empty again, the scrape has broken and this pair is silently back to inert —
check `up{job="longhorn-backend"}` and `platform/longhorn/networkpolicy-metrics.yaml`.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
@@ -73,14 +88,25 @@ Where this runbook gives a mutating command it is for an operator, and it says s
## First commands
Node-level allocation (the half that has no substitute alert):
Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read:
```
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status'
```
Node-level allocation, from the CRs:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"'
```
Volume sizes versus actual usage:
Volume nominal size versus ALLOCATED size. `actualSize` is head + pinned snapshots, NOT filesystem
usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note
above), so do not read this column as "full":
```
infra/devcontainer/dev.sh run bash -c \
@@ -95,7 +121,17 @@ infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName'
```
The filesystem view, which is the alert that actually works:
Which snapshots are holding the allocation (this is the measured top cause on prod — one
`user_created=false` system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13):
```
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'topk(10, longhorn_snapshot_actual_size_bytes)'
```
The filesystem view — the separate question, answered by `/runbooks/pvc-capacity`:
```
infra/devcontainer/dev.sh run bash -c \
@@ -111,7 +147,15 @@ ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev
## Likely causes, most likely first
### 1. A CNPG database is not recycling WAL (check this FIRST)
### 0. Pinned snapshots, not live data (check this FIRST for an allocation alert)
An allocation alert can fire with every filesystem in the cluster nearly empty. Run the
`longhorn_snapshot_actual_size_bytes` query above before anything else. A replica rebuild — which is
what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 —
leaves a `user_created=false` system snapshot behind, and it keeps its blocks until it is purged.
Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if
this was the cause.
### 1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert)
The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means
Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal`
sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to
@@ -144,8 +188,9 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor
## When to escalate or stop
- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete.
Free space or restore the other node before anything else.
- **A disk over 70% while any volume is degraded is urgent**, because the rebuild cannot complete —
and at 75% Longhorn takes the disk out of the scheduling pool entirely
(`LonghornDiskUnschedulable`). Free space or restore the other node before anything else.
- **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible.
- **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding
the real fault.
runbooks/longhorn-degraded.md
+24 -17
View File
@@ -2,7 +2,7 @@
**Alert:** `LonghornVolumeDegraded` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted)
**Expression:** `longhorn_volume_robustness{state="degraded"} == 1` (one-hot over `state`, NOT a 0-3 enum)
## What this means here
@@ -20,27 +20,34 @@ rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s`
share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So
the question to answer is never just "why is it degraded" but "is the rebuild about to take a node
out". Cross-read `/runbooks/node-io`.
### Verify the alert can even fire (measured 2026-09-13)
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was
enough to make a faulted or degraded volume alert nobody:
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series — the longhorn_* family was
absent from the TSDB because the chart's `metrics.serviceMonitor.enabled` defaulted false. Fixed
in `platform/longhorn/helmrelease.yaml`.
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
`longhorn_volume_robustness == 2` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states),
`count by (state) (longhorn_volume_robustness == 1)` = `{state="healthy"} 24`, and
`count(longhorn_volume_robustness == 2)` = **empty**. Fixed by keying on the label.
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`, which keeps
a regression case asserting the old `== 2` form matches nothing. Check the live series before
trusting this alert's silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'count by (state) (longhorn_volume_robustness)'
```
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no
ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will
probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not
from this alert**. Do not infer "no alert, so volumes are fine".
Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert
again — do not infer "no alert, so volumes are fine".
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
runbooks/longhorn-faulted.md
+23 -16
View File
@@ -2,7 +2,7 @@
**Alert:** `LonghornVolumeFaulted` (critical, 1m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn`
**Expression:** `longhorn_volume_robustness == 3`
**Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum)
## What this means here
@@ -22,26 +22,33 @@ Understand what is and is not recoverable before you touch anything:
- If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the
disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and
harmless — re-run the import.
### Verify the alert can even fire (measured 2026-09-13)
### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE)
Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`,
`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`,
`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`,
`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`,
`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`.
There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this
rule depends on before you trust its silence:
This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a
faulted volume — the most serious storage state on this cluster — alerted nobody, and announced
itself only as an application outage:
1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's
`metrics.serviceMonitor.enabled` defaulted false. Fixed in
`platform/longhorn/helmrelease.yaml`.
2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read
`longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric
is one-hot over a `state` label with values 0/1 only. Measured on prod the same day:
`count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and
`count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label.
Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the
live series before trusting this alert's silence:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \
sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query'
infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
/bin/promtool query instant http://localhost:9090 \
'count by (state) (longhorn_volume_robustness)'
```
**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod
on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group
has no data. A faulted volume today announces itself as an application outage. That is a known open
defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence.
Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this
alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
runbooks/targets-down.md
+4 -3
View File
@@ -25,7 +25,7 @@ of 2026-09-13 — every one of them means the corresponding alerts are **inert**
| Missing scrape job | Alerts that cannot fire |
|---|---|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
@@ -129,8 +129,9 @@ An exporter that is up but publishing the wrong series is the hardest case — s
monitoring one.
- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
chart value). Do not "fix" a blind spot by silencing the alert.
- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to
hand-apply monitors into the cluster — every durable change goes through git.
- The Longhorn and cert-manager scrapes were enabled on 2026-09-13 (the Longhorn row above is now
history). The Flux-controller and Hubble/Cilium monitor gaps are still open work. Note them; do not
try to hand-apply monitors into the cluster — every durable change goes through git.
---
Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`