diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index e51224c..b330e6b 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3332,3 +3332,145 @@ there may be a reason both entries exist. The three options, in the order I woul Cheapest, and relies on a comment — which is the mechanism this repo has already watched fail (the bulk StorageClass sat as a REPLACE-ME stub behind four hand-written comments). `vm-demo` is likely just an undeployed demo and may only need option 3. +## 2026-09-13 — `LonghornVolumeCapacityHigh` measured ALLOCATION and called it capacity; and the robustness rules were inert a SECOND time (branch `fix/longhorn-capacity-alert-measures-allocation`) + +Longhorn metrics began being scraped earlier today. Within minutes `LonghornVolumeCapacityHigh` went +`pending`. Three separate defects came out of re-measuring it, two of them the exact failure classes +this repo spent the day eliminating. + +### 1. The per-volume "capacity" rule was permanently firing, by design + +MEASURED on prod, `kubectl -n monitoring exec … promtool query instant`: + +| what | value | +|---|---| +| `longhorn_volume_actual_size_bytes` (Prometheus TSDB volume) | 52243681280 (48.66 GiB) | +| `longhorn_volume_capacity_bytes` | 53687091200 (50 GiB) | +| the rule's ratio | **0.9731** | +| next-highest volume in the cluster | 0.2818 (keycloak-db-1) — a lone outlier | +| `kubelet_volume_stats_used_bytes / …_capacity_bytes`, SAME filesystem | **0.2080** | +| `longhorn_snapshot_actual_size_bytes{user_created="false"}` on that volume | 40510484480 (**37.7 GiB**) | +| growth of actual_size, `deriv(…[2h])` | ~152 MB/h, monotonic | + +`actual_size` is head + pinned snapshots, not usage. ONE Longhorn system snapshot — the kind taken for +a full replica rebuild, which prod-1's drive replacement forced — holds 37.7 GiB; the live head is the +remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn returns no freed blocks without +discard/TRIM, and `retentionSize: 45GiB` on a 50Gi volume deliberately targets 90%, so the ratio is +pinned near 1.0 and **cannot fall back under 0.85**. Same shape as `CNPGNoRecoverabilityPoint`. + +DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation. +The redundancy was proven rather than asserted: of 24 Longhorn volumes on prod, 23 appear in +`kubelet_volume_stats_inodes > 0` and are therefore inside `MonitoringVolumeFillingUp` +(namespace="monitoring") or `PersistentVolumeFillingUp` / `…AlmostFull` (everything else). The 24th is +`unifi/unifi-rootdisk`, volumeMode Block — no filesystem for anyone to measure, and the deleted rule +was not covering it either (0.2097 was raw-device allocation). Block volumes are covered at the pool +level. + +### 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT after the scrape landed + +The bigger finding. On longhorn 1.12.1 `longhorn_volume_robustness` is **one-hot over a `state` +label** with values 0/1 — it is NOT the 0-3 enum the rules and their comment assumed: + +``` +count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) +count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 +count(longhorn_volume_robustness == 2) => (empty) +count(longhorn_volume_robustness == 3) => (empty) +``` + +So a FAULTED volume — the most serious storage state on this cluster — would have alerted nobody, +with the scrape working and the series present. The reading recorded in +`platform/longhorn/helmrelease.yaml` ("44 series … 24 healthy(1), 72 unknown(0, detached)") was this +same misreading; those zeros are the other three state labels of the same 24 healthy volumes. Both +rules are now keyed on the label, and a regression case asserts the old `== 2` form matches nothing. + +### 3. The node-level allocation alarm fired AFTER the cliff, and averaged disks together + +`LonghornNodeStorageHigh` at `> 0.85` was past Longhorn's own limit: `storage-minimal-available-percentage` +is **25**, so a disk leaves the scheduling pool at 75% used. Re-keyed to the per-disk series (the +node-level one SUMS a node's disks, so one full disk hides behind a half-empty sibling) and renamed: + +* `LonghornDiskAllocationHigh` — `(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70`, 30m +* `LonghornDiskUnschedulable` — `longhorn_disk_status{condition="schedulable"} == 0`, 15m — Longhorn's + own verdict, instead of re-deriving a scheduling formula that folds in three settings no metric + exports (reserved 20, over-provisioning 200, minimal-available 25). + +MEASURED prod usage: 0.2254 / 0.2306 / 0.2369 on prod-1/2/3; 268.9 GiB nominal scheduled per node. + +### Proof, both directions, for every rule touched + +`platform/monitoring/tests/longhorn.test.yaml` (6 cases, run by +`platform/monitoring/tests/run.sh`; the L0 register shrank by 7 and the repo is now 14/58 alerts +unit-tested). Live: all four new expressions return EMPTY on prod, while +`count(robustness{state="degraded"}) + count(robustness{state="faulted"}) + count(disk ratio) + +count(disk_status{condition="schedulable"})` returns **54** — quiet, and provably not inert. +Mutation-checked 11 ways (each turns the suite RED): both old enum expressions, the 0.85 threshold, +the node-summed metric, a threshold low enough to fire on today's healthy state, the wrong +`condition` label, three shortened/lengthened `for:` values, and two mutations of the kubelet rules. + +### OWNER DECISION 1 — is `retentionSize: 45GiB` on a 50Gi volume right? + +**Recommendation: lower it to `36GiB` (about 72% of the volume). Not changed here — retention is how +much history you keep, so it is your call.** + +The reasoning, and it is not about the alert: + +* 45GiB leaves 5 GiB of headroom on the volume, and `retentionSize` does **not** count the WAL, the + in-memory head's on-disk segments, or compaction scratch. A compaction of a large block can need + roughly the size of the blocks being merged. At 90% full, a single bad compaction is + "no space left on device", which wedges every WAL commit — scrapes still return 200, nothing is + stored, the head freezes, and every `absent()`/`== 0` platform alert fires falsely. The file's own + comment says the cap is "kept BELOW the PVC size … for WAL + head + compaction scratch"; 10% is not + below enough for that list. +* The volume is 3x replicated on mini-PCs whose data partition also carries the etcd WAL. Every GiB of + TSDB is 3 GiB of allocation across the pool and more rebuild I/O when a replica resyncs — and + etcd fsync starvation is this cluster's known way of losing a node (2026-09-07). +* It is not currently constraining anything: the TSDB is **11.2 GiB** at 30d retention on a 3-node + cluster. 36GiB is still triple today's size, so 30d would remain the binding limit in practice and + no history would actually be lost. +* If more history is wanted, grow the PVC (`allowVolumeExpansion: true`) rather than the fraction — + the knobs are `MONITORING_PROMETHEUS_PVC_SIZE` and `MONITORING_PROMETHEUS_RETENTION_SIZE` in + `clusters/prod/kustomization.yaml`, and the base comment already requires the second to stay below + the first. + +### OWNER DECISION 2 — enable discard/TRIM on the `longhorn` StorageClass? + +**Recommendation: do NOT enable it globally now. It is feasible, but it buys a metric and risks the +one subsystem this cluster keeps losing.** Not enabled here; it changes behaviour for every Longhorn +volume on the cluster. + +* Feasible. The class already sets `fsType: "ext4"`, and Longhorn passes mount options through, so + continuous discard is a StorageClass change (`mountOptions: ["discard"]`) plus a REMOUNT of every + volume to take effect — i.e. a restart of every stateful workload. Longhorn also exposes a + per-volume `Trim Filesystem` action, which is the same reclaim without touching the class. +* The setting that actually matters here is `remove-snapshots-during-filesystem-trim`, read live on + prod as **false**. With it false, a trim reclaims nothing that a snapshot is holding — so trim, in + any form, would NOT have reclaimed the 37.7 GiB. Flipping that one to true is a far larger decision + than enabling discard: it makes a routine filesystem trim DELETE SNAPSHOTS, i.e. quietly discard + recovery points. +* Also read live: `auto-cleanup-system-generated-snapshot` is already **true**, and the 37.7 GiB + system snapshot is still present — so it is most likely the replica's base snapshot, which cannot be + removed while it is the parent of the volume head. If so, no trim setting and no discard flag will + release it; only a full replica rebuild onto a fresh file would, which is the expensive operation + this cluster is trying to avoid. +* What it would buy: `actual_size` would track real usage. That is now worth nothing to alerting — + no rule reads `actual_size` any more, and the pool-level rules read disk usage, which trim improves + only for blocks no snapshot is holding. **The 37.7 GiB that started all this is held by a SNAPSHOT, + which discard cannot release** (see the two settings below). +* What it risks: `discard` makes every filesystem delete issue a synchronous block-level discard on + all three replicas, on nodes where `/data/k3s` (etcd WAL) and `/data/longhorn` share one device. The + failure mode we already have is etcd fsync starvation under Longhorn write load. Turning on + continuous discard for 24 volumes is exactly the wrong direction for that, and a mass remount is a + rebuild-storm trigger in its own right (every node reboot degrades all volumes today). +* If reclaim is wanted later, the safe order is: (1) check whether the big snapshot is a removable + snapshot or the replica's base, (2) purge what is removable from the Longhorn UI, (3) per-volume + `Trim Filesystem` during a quiet window, (4) only then consider a periodic `fstrim` timer — never + continuous `discard`, and never `remove-snapshots-during-filesystem-trim: true`, while etcd shares + the spindle. Revisit if/when `docs/plans/longhorn-ssd-tiering.md` separates etcd from Longhorn. + +### Also noticed, NOT changed (drive-by, for whoever owns it) + +`runbooks/longhorn-degraded.md` states that `snapshotHeavyTaskConcurrentLimit` was tightened 5 → 1 +after the 2026-09-07 incident. Read live on prod today: `snapshot-heavy-task-concurrent-limit` is +**5**. Either the change never landed or it was reverted; the runbook is currently telling an operator +a knob is set that is not. Out of scope for this branch. diff --git a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh index 18339af..8b3c166 100755 --- a/conformance/tests/l0-static/alert-rules-unit-tested/check.sh +++ b/conformance/tests/l0-static/alert-rules-unit-tested/check.sh @@ -168,8 +168,12 @@ UNTESTED_FILES = { # own rule nothing asserts on. This is the half that stops a new alert being slipped into an # already-"tested" file untested: anything not listed here must appear in a covering test. # The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately -# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same -# rules file by the suite convention but has no assertions of its own yet. +# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when +# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group +# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume +# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the +# redundancy meant asserting on the rules that survive). Only +# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml. UNTESTED_ALERTS = { ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"), ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"), @@ -183,10 +187,6 @@ UNTESTED_ALERTS = { ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"), ("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"), ("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"), - ("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"), - ("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"), - ("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"), - ("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"), ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"), ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"), ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"), @@ -198,13 +198,10 @@ UNTESTED_ALERTS = { ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"), ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"), ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"), - ("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"), ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"), ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"), ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"), ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"), - ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"), - ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"), ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"), } diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index a9e58d6..13bc02e 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -103,9 +103,9 @@ spec: orphanAutoDeletion: true # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ---------- - # platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical), - # LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh — - # and NONE of them could ever fire, because the chart defaults + # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13: + # LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and + # LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults # `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system. # VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn; # `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING. @@ -140,12 +140,26 @@ spec: # colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place: # count(up{job="longhorn-backend"}) => 3 (all three managers, up==1) # count({__name__=~"longhorn.*"}) => 8717 (was EMPTY) - # count(longhorn_volume_robustness) => 44 - # count_values("v", longhorn_volume_robustness) => 24 healthy(1), 72 unknown(0, detached) - # and the four alert rules went from "no series, cannot fire" to evaluating for real — one of - # them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB - # volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming - # freed blocks. Reported, NOT silenced by moving the threshold). + # count(longhorn_volume_robustness) => 96 (24 volumes x 4 `state` labels) + # count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 + # CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read + # "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot + # metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a + # `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72 + # zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached + # volumes. The consequence was that enabling this scrape did NOT un-inert + # LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample. + # Fixed in platform/monitoring/alerts.yaml and pinned by + # platform/monitoring/tests/longhorn.test.yaml. + # + # Of the rules that WERE evaluating for real, one immediately found a true positive and one + # permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the + # Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does + # not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's + # replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded): + # it measured allocation and called it capacity, the filesystem question is already answered by + # kubelet_volume_stats_*, and the allocation question now lives per disk as + # LonghornDiskAllocationHigh / LonghornDiskUnschedulable. metrics: serviceMonitor: enabled: true diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index 25aa82c..111b31a 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -147,9 +147,33 @@ spec: groups: - name: platform-health.longhorn rules: - # longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted + # ------------------------------------------------------------------------------------- + # `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13) + # + # Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment + # "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On + # longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state` + # label and its value is only ever 0 or 1: + # longhorn_volume_robustness{volume="…",state="healthy"} 1 + # longhorn_volume_robustness{volume="…",state="degraded"} 0 + # longhorn_volume_robustness{volume="…",state="faulted"} 0 + # longhorn_volume_robustness{volume="…",state="unknown"} 0 + # MEASURED on prod 2026-09-13, scrape working, 24 volumes attached: + # count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) + # count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 + # count(longhorn_volume_robustness == 2) => (empty) + # count(longhorn_volume_robustness == 3) => (empty) + # So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today + # made the series exist; it did NOT make these two rules able to fire — a faulted volume + # would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)" + # reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those + # zeros are the OTHER state labels of healthy volumes, not detached volumes. + # + # Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a + # regression case asserting the old `== 2` form matches nothing on this metric shape. + # ------------------------------------------------------------------------------------- - alert: LonghornVolumeDegraded - expr: longhorn_volume_robustness == 2 + expr: longhorn_volume_robustness{state="degraded"} == 1 for: 10m labels: severity: warning @@ -158,7 +182,7 @@ spec: description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded" - alert: LonghornVolumeFaulted - expr: longhorn_volume_robustness == 3 + expr: longhorn_volume_robustness{state="faulted"} == 1 for: 1m labels: severity: critical @@ -166,23 +190,103 @@ spec: summary: "Longhorn volume FAULTED: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted" - - alert: LonghornVolumeCapacityHigh - expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 + # ------------------------------------------------------------------------------------- + # WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND + # CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR. + # + # The deleted rule was + # (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m + # summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes + # of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume. + # MEASURED on prod 2026-09-13: + # pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn) + # longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB) + # longhorn_volume_capacity_bytes 53687091200 (50 GiB) + # ratio 0.9731 <- what the alert saw + # next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier + # the SAME filesystem, from the kubelet: + # kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used) + # + # `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS + # every snapshot still pinned. On this volume, + # longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB) + # ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly + # what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head + # is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns + # freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was + # climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With + # `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume, + # the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can + # only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator + # to ignore the storage row, which is the row that matters on this cluster. + # + # NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not + # asserted: + # * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring + # namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file; + # every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in + # platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13: + # 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and + # so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode + # Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so + # the deleted rule was not covering it either (its 0.2097 was raw-device allocation). + # Block volumes are covered at the pool level, below. + # * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and + # it is answered per DISK below, where allocation is a budget, instead of per volume + # against its own nominal size, where it is a tautology. + # ------------------------------------------------------------------------------------- + # THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET. + # + # Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to + # longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason: + # + # * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated — + # replica files plus their pinned snapshots — because that is what consumes the node's + # data partition whether or not any guest filesystem thinks it is full. "Node storage + # near capacity" invited the same misreading the deleted per-volume rule shipped; + # "disk allocation high" says what it measures. + # * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one + # full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries + # a `disk` label. Identical today (one disk per node) and correct if a disk is added. + # * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own + # storage-minimal-available-percentage is 25 on this cluster (read live from + # settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old + # threshold could therefore only fire AFTER scheduling had already been refused — and + # with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk + # that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70 + # gives lead time on the way to Longhorn's own limit. + # + # MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes: + # node capacity usage ratio scheduled (nominal) + # k3s-prod-1 445098295296 100326506496 0.2254 288836550656 + # k3s-prod-2 456002658304 105149128704 0.2306 288836550656 + # k3s-prod-3 445098295296 105464528896 0.2369 288836550656 + # ------------------------------------------------------------------------------------- + - alert: LonghornDiskAllocationHigh + expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70 for: 30m labels: severity: warning annotations: - summary: "Longhorn volume near capacity: {{ $labels.volume }}" - description: "Volume {{ $labels.volume }} is over 85% used for 30m." + summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}" + description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" - - alert: LonghornNodeStorageHigh - expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85 - for: 30m + # The terminal form of the same condition, straight from Longhorn rather than inferred from a + # threshold: the disk itself reports whether it will accept another replica. More honest than + # re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage + # 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 — + # three settings NO metric exports, so the rule would silently mis-measure if any changed. + # MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all + # three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while + # a longhorn-manager restarts. + - alert: LonghornDiskUnschedulable + expr: longhorn_disk_status{condition="schedulable"} == 0 + for: 15m labels: severity: warning annotations: - summary: "Longhorn node storage near capacity: {{ $labels.node }}" - description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m." + summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}" + description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" # ===================================================================================== diff --git a/platform/monitoring/pvc-capacity-alerts.yaml b/platform/monitoring/pvc-capacity-alerts.yaml index 3127e89..6e8fdea 100644 --- a/platform/monitoring/pvc-capacity-alerts.yaml +++ b/platform/monitoring/pvc-capacity-alerts.yaml @@ -10,12 +10,26 @@ # the SSO database with nothing watching it. # # NOTE, because it changes what "covered" means: `platform-health.longhorn`'s -# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the -# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06: -# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is -# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and -# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and -# working on both clusters. +# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two +# different reasons in sequence, and it is now GONE: +# * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified +# read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule +# group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in +# platform/longhorn/helmrelease.yaml. +# * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It +# divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head +# plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and +# called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet +# read the same filesystem 20.8% used — by design and permanently, because `retentionSize: +# 45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the +# comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now +# lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable. +# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to +# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`. +# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in +# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in +# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no +# filesystem for anyone to measure (see the inodes guard below). # # The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent # edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a diff --git a/platform/monitoring/tests/longhorn.test.yaml b/platform/monitoring/tests/longhorn.test.yaml new file mode 100644 index 0000000..d0422b2 --- /dev/null +++ b/platform/monitoring/tests/longhorn.test.yaml @@ -0,0 +1,465 @@ +# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml) +# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity" +# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file, +# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml). +# +# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via +# conformance/tests/l0-static/alert-rules-unit-tested) +# +# WHAT THIS FILE IS FOR +# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for +# the first time. Two things became visible within minutes: +# 1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the +# kubelet reported the same filesystem 20.8% used. The rule was named and documented as a +# CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can +# never clear, so it was a PERMANENTLY-FIRING alert. It is deleted. +# 2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series +# present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state` +# label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed. +# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule +# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a +# constructed bad state, with the `for:` durations asserted as real. +# +# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so +# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the +# literal bytes measured on prod 2026-09-13. Both are noted at the point of use. +# +# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so +# nothing is ever stale inside Prometheus' 5m lookback. +rule_files: + - generated.rules.yaml + +evaluation_interval: 1m + +tests: + # =========================================================================================== + # 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT. + # This is the "not permanently firing" half of the proof for every rule below, and it is the + # test that would have caught the deleted rule: the SAME fixture carries the real + # actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted + # expression evaluating to 0.9731 on a volume whose filesystem is 20% used. + # =========================================================================================== + - interval: 1m + name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet + input_series: + # --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) --- + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}' + values: '1+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}' + values: '1+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}' + values: '0+0x120' + # --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod --- + - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + values: '52243681280+0x120' + - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + values: '53687091200+0x120' + # --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) --- + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' + values: '445098295296+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' + values: '100326506496+0x120' + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}' + values: '456002658304+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}' + values: '105149128704+0x120' + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' + values: '445098295296+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' + values: '105464528896+0x120' + # --- Longhorn's own scheduling verdict: all three disks accepting replicas --- + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}' + values: '1+0x120' + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}' + values: '1+0x120' + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}' + values: '1+0x120' + # `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition` + # selector is load-bearing. Drop the selector and the rule would still be quiet on THIS + # fixture, so test 5 constructs the case where it matters. + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}' + values: '1+0x120' + # --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free --- + # Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080. + - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '53687091200+0x120' + - series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '10737418240+0x120' + - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '42949672960+0x120' + - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '3276800+0x120' + # and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod) + - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '5368709120+0x120' + - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '4993977548+0x120' + - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '327680+0x120' + alert_rule_test: + - eval_time: 2h + alertname: LonghornVolumeDegraded + exp_alerts: [] + - eval_time: 2h + alertname: LonghornVolumeFaulted + exp_alerts: [] + - eval_time: 2h + alertname: LonghornDiskAllocationHigh + exp_alerts: [] + - eval_time: 2h + alertname: LonghornDiskUnschedulable + exp_alerts: [] + # The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the + # per-volume Longhorn rule does not trade a false alarm for a blind spot on this state. + - eval_time: 2h + alertname: MonitoringVolumeFillingUp + exp_alerts: [] + - eval_time: 2h + alertname: PersistentVolumeFillingUp + exp_alerts: [] + - eval_time: 2h + alertname: PersistentVolumeAlmostFull + exp_alerts: [] + promql_expr_test: + # THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on. + - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes + eval_time: 1h + exp_samples: + - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + value: 0.9731143951416016 + # …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the + # whole argument for the deletion, and it is now an assertion rather than a paragraph. + - expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes + eval_time: 1h + exp_samples: + - labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + value: 0.2 + # THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) … + - expr: count(longhorn_volume_robustness) + eval_time: 1h + exp_samples: + - labels: '{}' + value: 8 + # … and the OLD expressions still match nothing on this metric shape. This is the regression + # guard: if someone "simplifies" the rules back to a numeric enum, these two go red. + - expr: count(longhorn_volume_robustness == 2) or vector(-1) + eval_time: 1h + exp_samples: + - labels: '{}' + value: -1 + - expr: count(longhorn_volume_robustness == 3) or vector(-1) + eval_time: 1h + exp_samples: + - labels: '{}' + value: -1 + # The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules + # are quiet because the state is healthy, not because they select nothing. + - expr: count(longhorn_volume_robustness{state="degraded"}) + eval_time: 1h + exp_samples: + - labels: '{}' + value: 2 + + # =========================================================================================== + # 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`. + # Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point. + # =========================================================================================== + - interval: 1m + name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet + input_series: + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}' + values: '1+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}' + values: '0+0x120' + alert_rule_test: + # `for: 10m` is real: nothing at 5m. + - eval_time: 5m + alertname: LonghornVolumeDegraded + exp_alerts: [] + - eval_time: 30m + alertname: LonghornVolumeDegraded + exp_alerts: + - exp_labels: + severity: warning + job: longhorn-backend + namespace: longhorn-system + node: k3s-prod-2 + pvc: keycloak-db-1 + pvc_namespace: keycloak + volume: pvc-f30d3eb7 + state: degraded + exp_annotations: + summary: "Longhorn volume degraded: pvc-f30d3eb7" + description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m." + runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded" + # Degraded must not masquerade as faulted: the critical tier stays silent. + - eval_time: 30m + alertname: LonghornVolumeFaulted + exp_alerts: [] + promql_expr_test: + # The pre-fix expression, on the state it was written for. Empty. This is the inert rule that + # reached prod, reproduced offline. + - expr: count(longhorn_volume_robustness == 2) or vector(-1) + eval_time: 30m + exp_samples: + - labels: '{}' + value: -1 + + # =========================================================================================== + # 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not. + # =========================================================================================== + - interval: 1m + name: one volume faulted - Faulted fires after 1m, Degraded stays quiet + input_series: + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}' + values: '0+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}' + values: '1+0x120' + - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}' + values: '0+0x120' + alert_rule_test: + # `for: 1m` is real: pending, not firing, at the first evaluation. + - eval_time: 0m + alertname: LonghornVolumeFaulted + exp_alerts: [] + - eval_time: 10m + alertname: LonghornVolumeFaulted + exp_alerts: + - exp_labels: + severity: critical + job: longhorn-backend + namespace: longhorn-system + node: k3s-prod-3 + pvc: nextcloud-db-1 + pvc_namespace: nextcloud-db + volume: pvc-d6621428 + state: faulted + exp_annotations: + summary: "Longhorn volume FAULTED: pvc-d6621428" + description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required." + runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted" + - eval_time: 10m + alertname: LonghornVolumeDegraded + exp_alerts: [] + + # =========================================================================================== + # 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK. + # Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the + # promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the + # node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level + # form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff. + # =========================================================================================== + - interval: 1m + name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it + input_series: + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' + values: '400000000000+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' + values: '288000000000+0x120' + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' + values: '400000000000+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' + values: '92000000000+0x120' + # Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires + # on its own, ahead of the terminal signal tested in case 5. + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}' + values: '1+0x120' + alert_rule_test: + # `for: 30m` is real: nothing at 20m. + - eval_time: 20m + alertname: LonghornDiskAllocationHigh + exp_alerts: [] + - eval_time: 45m + alertname: LonghornDiskAllocationHigh + exp_alerts: + - exp_labels: + severity: warning + job: longhorn-backend + namespace: longhorn-system + node: k3s-prod-1 + disk: default-disk-new + exp_annotations: + summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new" + description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity." + runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" + - eval_time: 45m + alertname: LonghornDiskUnschedulable + exp_alerts: [] + promql_expr_test: + # Per disk: one at 0.72, one at 0.23. + - expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + eval_time: 45m + exp_samples: + - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}' + value: 0.23 + - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}' + value: 0.72 + # Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent + # at the old 0.85. That is the blind spot the metric change closes. + - expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes) + eval_time: 45m + exp_samples: + - labels: '{node="k3s-prod-1"}' + value: 0.475 + + # =========================================================================================== + # 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires. + # The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so + # a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the + # schedulable series only by luck of which conditions happen to exist. Allocation is nowhere + # near the threshold, so this rule is proven to stand on its own signal. + # =========================================================================================== + - interval: 1m + name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone + input_series: + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}' + values: '0+0x120' + - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}' + values: '1+0x120' + - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' + values: '400000000000+0x120' + - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}' + values: '92000000000+0x120' + alert_rule_test: + # `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart. + - eval_time: 10m + alertname: LonghornDiskUnschedulable + exp_alerts: [] + - eval_time: 30m + alertname: LonghornDiskUnschedulable + exp_alerts: + - exp_labels: + severity: warning + job: longhorn-backend + namespace: longhorn-system + node: k3s-prod-3 + disk: default-disk-59b6657d + condition: schedulable + exp_annotations: + summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d" + description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node." + runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" + - eval_time: 30m + alertname: LonghornDiskAllocationHigh + exp_alerts: [] + + # =========================================================================================== + # 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule + # fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the + # allocation number cannot tell these two situations apart, and the kubelet number can. So the + # signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for + # the monitoring namespace and by the pvc-capacity rules for every other namespace. + # =========================================================================================== + - interval: 1m + name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged + input_series: + # allocation IDENTICAL to the healthy case — 0.9731, indistinguishable + - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + values: '52243681280+0x120' + - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + values: '53687091200+0x120' + # monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15% + - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '53687091200+0x120' + - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '5368709120+0x120' + - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}' + values: '3276800+0x120' + # a non-monitoring Longhorn PVC: 10% free -> warning tier + - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '5368709120+0x120' + - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '536870912+0x120' + - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}' + values: '327680+0x120' + # another one at 3% free -> critical tier as well as warning + - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' + values: '10737418240+0x120' + - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' + values: '322122547+0x120' + - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}' + values: '655360+0x120' + alert_rule_test: + # `for: 15m` on all three: nothing at 10m. + - eval_time: 10m + alertname: MonitoringVolumeFillingUp + exp_alerts: [] + - eval_time: 30m + alertname: MonitoringVolumeFillingUp + exp_alerts: + - exp_labels: + severity: warning + job: kubelet + namespace: monitoring + node: k3s-prod-1 + persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 + exp_annotations: + summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0" + description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind." + runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity" + - eval_time: 10m + alertname: PersistentVolumeFillingUp + exp_alerts: [] + # Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the + # exact-partition property pvc-capacity-alerts.yaml documents. + - eval_time: 30m + alertname: PersistentVolumeFillingUp + exp_alerts: + - exp_labels: + severity: warning + job: kubelet + namespace: gitea + node: k3s-prod-3 + persistentvolumeclaim: gitea-data + exp_annotations: + summary: "PVC filling up: gitea/gitea-data" + description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" + - exp_labels: + severity: warning + job: kubelet + namespace: keycloak + node: k3s-prod-2 + persistentvolumeclaim: keycloak-db-1 + exp_annotations: + summary: "PVC filling up: keycloak/keycloak-db-1" + description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" + # Only the 3%-free one reaches the critical tier — the 10% one must not. + - eval_time: 30m + alertname: PersistentVolumeAlmostFull + exp_alerts: + - exp_labels: + severity: critical + job: kubelet + namespace: gitea + node: k3s-prod-3 + persistentvolumeclaim: gitea-data + exp_annotations: + summary: "PVC almost full: gitea/gitea-data" + description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down." + runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity" + promql_expr_test: + # The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the + # filesystem went from 80% free to 10% free. One number moved; the other cannot. + - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes + eval_time: 30m + exp_samples: + - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}' + value: 0.9731143951416016 diff --git a/runbooks/README.md b/runbooks/README.md index 11fe78c..a088d67 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -37,12 +37,15 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`, ## Alerts you should not trust the silence of (measured 2026-09-13) -Prometheus scrapes 19 jobs on prod. There is **no** Longhorn, cert-manager, Flux-controller or -Hubble/Cilium job, so these rule groups currently evaluate against nothing and **cannot fire**: +The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no +longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the +scrape had left in place (the metric is one-hot over a `state` label, not a 0–3 enum). Take the +Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium +rule groups still evaluate against nothing and **cannot fire**: | Missing scrape job | Inert alerts | |---|---| -| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` | | cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | @@ -65,7 +68,7 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`longhorn-degraded`** — a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here. - **`longhorn-faulted`** — no usable replica. Data at risk; stop before anything destructive. -- **`longhorn-capacity`** — a volume or a node's Longhorn disk near capacity. +- **`longhorn-capacity`** — a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill see `pvc-capacity` / `monitoring-capacity`). - **`pvc-capacity`** — PVC filling up / almost full / predicted to fill. These rules work. - **`monitoring-capacity`** — Prometheus or Loki running out of disk; don't let the monitor blind itself. diff --git a/runbooks/longhorn-capacity.md b/runbooks/longhorn-capacity.md index 9258f12..698779b 100644 --- a/runbooks/longhorn-capacity.md +++ b/runbooks/longhorn-capacity.md @@ -1,47 +1,62 @@ -# longhorn-capacity — A Longhorn volume or node is near its storage capacity +# longhorn-capacity — A Longhorn disk is running out of room for replicas -**Alerts:** `LonghornVolumeCapacityHigh` (warning, 30m) and `LonghornNodeStorageHigh` (warning, 30m) +**Alerts:** `LonghornDiskAllocationHigh` (warning, 30m) and `LonghornDiskUnschedulable` (warning, 15m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` **Expressions:** -`(longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85` -`(longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85` +`(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70` +`longhorn_disk_status{condition="schedulable"} == 0` ## What this means here -Either one volume is 85% full, or one node's Longhorn disk is 85% allocated. They are different -problems with different fixes, so read the labels: `volume` means a workload is filling its own PVC; -`node` means the cluster is running out of places to put replicas. +**This is a POOL question, not a PVC question.** One of the three nodes' Longhorn data disks is +filling with replica files, or Longhorn has already stopped placing replicas on it. Neither alert +says anything about how full any single workload's filesystem is — for that, read +`/runbooks/pvc-capacity` (`PersistentVolumeFillingUp`, `PersistentVolumeAlmostFull`) or +`/runbooks/monitoring-capacity` (`MonitoringVolumeFillingUp`). Those are keyed on +`kubelet_volume_stats_*`, which is the filesystem's own accounting. -The node case is bounded by settings that are already conservative: -`storageReservedPercentageForDefaultDisk: 20`, `storageMinimalAvailablePercentage: 25`, -`storageOverProvisioningPercentage: 200`, `defaultDataLocality: disabled`, hard node anti-affinity. -Measured headroom on prod is roughly **269 GiB scheduled against an 829 GiB ceiling**, so the node -alert firing means something has changed materially, not that the platform has slowly grown. +`longhorn_disk_usage_bytes` counts bytes ALLOCATED: every replica file plus every snapshot still +pinned to it. Longhorn does not return freed blocks to a replica file without discard/TRIM, and a +system snapshot taken for a replica rebuild keeps its historical blocks alive, so this number only +goes up until something is deleted. That is the correct question *here* — allocated bytes are what +consumes the node's ~414 GiB data partition — and the wrong question per volume, which is why +`LonghornVolumeCapacityHigh` was deleted on 2026-09-13 (it read 0.973 on a Prometheus TSDB volume +whose filesystem was 20.8% used, by design, permanently). See the comment block in +`platform/monitoring/alerts.yaml` for the measurements. -The severe consequence of running out of node storage is that a degraded volume **cannot finish -rebuilding** — hard anti-affinity needs three distinct nodes with room, so one full node pins a -volume at 2 replicas indefinitely. Cross-read `/runbooks/longhorn-degraded`. -### Verify the alert can even fire (measured 2026-09-13) +**Why 0.70 and not 0.85.** `storageMinimalAvailablePercentage: 25` means Longhorn itself refuses to +schedule a new replica on a disk with under 25% available — i.e. at **75% used the disk is already +out of the scheduling pool**. An 85% threshold could only ever fire after that had happened. 0.70 is +the last point at which there is still lead time. The companion settings are +`storageReservedPercentageForDefaultDisk: 20`, `storageOverProvisioningPercentage: 200`, +`defaultDataLocality: disabled`, hard node anti-affinity. -Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, -`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, -`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, -`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, -`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. -There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this -rule depends on before you trust its silence: +Measured on prod 2026-09-13: usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3, with 268.9 GiB of nominal +replica size scheduled against (414.5 GiB − 20% reserved) × 200%. So either alert firing means +something changed materially, not that the platform slowly grew. + +The severe consequence is that a degraded volume **cannot finish rebuilding** — hard anti-affinity +needs three distinct nodes with room, so one full or unschedulable disk pins a volume at 2 replicas +indefinitely, and R-STATE-02's survive-a-node-loss promise is suspended for as long as that lasts. +Cross-read `/runbooks/longhorn-degraded`. + +### Verify the alert can fire (measured 2026-09-13) + +These alerts were INERT until 2026-09-13, when `metrics.serviceMonitor.enabled` was set in +`platform/longhorn/helmrelease.yaml`; the longhorn_* family was simply absent from the TSDB before +that. Both are now live — verified on prod the same day: ``` -infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ - sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ + prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 \ + 'count(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) + count(longhorn_disk_status{condition="schedulable"})' ``` -**These alerts are currently INERT.** `count(longhorn_volume_capacity_bytes)` and -`count(longhorn_node_storage_capacity_bytes)` both returned **0 series** on prod on 2026-09-13 — -Longhorn ships no ServiceMonitor here. Volume-fill coverage actually comes from -`kubelet_volume_stats_*` instead, i.e. `/runbooks/pvc-capacity`, which is where the working alerts -are. Use this runbook for the Longhorn-side (node allocation) half, and expect to reach it by hand. +returned **6** (3 disks, both selectors resolving), while both alert expressions returned empty. +If that count is ever empty again, the scrape has broken and this pair is silently back to inert — +check `up{job="longhorn-backend"}` and `platform/longhorn/networkpolicy-metrics.yaml`. + ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev @@ -73,14 +88,25 @@ Where this runbook gives a mutating command it is for an operator, and it says s ## First commands -Node-level allocation (the half that has no substitute alert): +Per-disk allocation and Longhorn's own scheduling verdict — the two things the alerts read: + +``` +infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ + prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 \ + 'longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes or longhorn_disk_status' +``` + +Node-level allocation, from the CRs: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get nodes.longhorn.io -o yaml | grep -E "name:|storageAvailable|storageMaximum|storageScheduled|allowScheduling"' ``` -Volume sizes versus actual usage: +Volume nominal size versus ALLOCATED size. `actualSize` is head + pinned snapshots, NOT filesystem +usage — a volume can read near 100% here with a nearly empty filesystem (see the deleted-alert note +above), so do not read this column as "full": ``` infra/devcontainer/dev.sh run bash -c \ @@ -95,7 +121,17 @@ infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace/.spec.claimRef.name,SIZE:.spec.capacity.storage,SC:.spec.storageClassName' ``` -The filesystem view, which is the alert that actually works: +Which snapshots are holding the allocation (this is the measured top cause on prod — one +`user_created=false` system snapshot on the Prometheus TSDB volume held 37.7 GiB on 2026-09-13): + +``` +infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ + prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 \ + 'topk(10, longhorn_snapshot_actual_size_bytes)' +``` + +The filesystem view — the separate question, answered by `/runbooks/pvc-capacity`: ``` infra/devcontainer/dev.sh run bash -c \ @@ -111,7 +147,15 @@ ssh k3s-prod-1 'df -h /data; du -xh --max-depth=1 /data/longhorn/replicas 2>/dev ## Likely causes, most likely first -### 1. A CNPG database is not recycling WAL (check this FIRST) +### 0. Pinned snapshots, not live data (check this FIRST for an allocation alert) +An allocation alert can fire with every filesystem in the cluster nearly empty. Run the +`longhorn_snapshot_actual_size_bytes` query above before anything else. A replica rebuild — which is +what a node reboot or a disk replacement forces, and prod-1's drive was replaced on 2026-09-12 — +leaves a `user_created=false` system snapshot behind, and it keeps its blocks until it is purged. +Prune from the Longhorn UI (Volume → Snapshots), then re-read the query; nothing else is needed if +this was the cause. + +### 1. A CNPG database is not recycling WAL (check this FIRST for a PVC-fill alert) The only cause here that grows without bound. `archive_mode=on` plus a failing `archive_command` means Postgres will not recycle unarchived WAL; measured on prod `keycloak-db` at ~6.8 MiB/h with `pg_wal` sharing the data PVC (no `walStorage:` is configured on any Cluster) — about 29 days from 93% free to @@ -144,8 +188,9 @@ answer (a ~51 GB hot tier: Prometheus, CNPG, Loki, Gitea). Operator planning wor ## When to escalate or stop -- **A node over 85% while any volume is degraded is urgent**, because the rebuild cannot complete. - Free space or restore the other node before anything else. +- **A disk over 70% while any volume is degraded is urgent**, because the rebuild cannot complete — + and at 75% Longhorn takes the disk out of the scheduling pool entirely + (`LonghornDiskUnschedulable`). Free space or restore the other node before anything else. - **Never delete files under `/data/longhorn`** or inside a Postgres `pg_wal`. Both are irreversible. - **Do not expand a PVC as a reflex** if the cause is unrecycled WAL — you are buying days and hiding the real fault. diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index 803ddb1..695a5e8 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -2,7 +2,7 @@ **Alert:** `LonghornVolumeDegraded` (warning, 10m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` -**Expression:** `longhorn_volume_robustness == 2` (0=unknown 1=healthy 2=degraded 3=faulted) +**Expression:** `longhorn_volume_robustness{state="degraded"} == 1` (one-hot over `state`, NOT a 0-3 enum) ## What this means here @@ -20,27 +20,34 @@ rebuild writes land on the same partition as the embedded etcd WAL (`/data/k3s` share one device), and that is the mechanism behind the 2026-09-07 `k3s-prod-1` NotReady outage. So the question to answer is never just "why is it degraded" but "is the rebuild about to take a node out". Cross-read `/runbooks/node-io`. -### Verify the alert can even fire (measured 2026-09-13) +### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE) -Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, -`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, -`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, -`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, -`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. -There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this -rule depends on before you trust its silence: +This rule was inert for two independent reasons, both fixed on 2026-09-13, and either one alone was +enough to make a faulted or degraded volume alert nobody: + +1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series — the longhorn_* family was + absent from the TSDB because the chart's `metrics.serviceMonitor.enabled` defaulted false. Fixed + in `platform/longhorn/helmrelease.yaml`. +2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read + `longhorn_volume_robustness == 2` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric + is one-hot over a `state` label with values 0/1 only. Measured on prod the same day: + `count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states), + `count by (state) (longhorn_volume_robustness == 1)` = `{state="healthy"} 24`, and + `count(longhorn_volume_robustness == 2)` = **empty**. Fixed by keying on the label. + +Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`, which keeps +a regression case asserting the old `== 2` form matches nothing. Check the live series before +trusting this alert's silence: ``` -infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ - sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ + prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 \ + 'count by (state) (longhorn_volume_robustness)' ``` -**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod -on 2026-09-13: the whole `platform-health.longhorn` rule group has no data, because Longhorn ships no -ServiceMonitor here. That is a separate open defect, and the practical consequence is that **you will -probably learn about a degraded volume from an application symptom or from `volumes.longhorn.io`, not -from this alert**. Do not infer "no alert, so volumes are fine". +Four states, 24 each, is healthy wiring. Empty means the scrape has broken and this alert is inert +again — do not infer "no alert, so volumes are fine". ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev diff --git a/runbooks/longhorn-faulted.md b/runbooks/longhorn-faulted.md index 7a3a5d0..3b210ad 100644 --- a/runbooks/longhorn-faulted.md +++ b/runbooks/longhorn-faulted.md @@ -2,7 +2,7 @@ **Alert:** `LonghornVolumeFaulted` (critical, 1m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.longhorn` -**Expression:** `longhorn_volume_robustness == 3` +**Expression:** `longhorn_volume_robustness{state="faulted"} == 1` (one-hot over `state`, NOT a 0-3 enum) ## What this means here @@ -22,26 +22,33 @@ Understand what is and is not recoverable before you touch anything: - If this is a **`longhorn-vm`** volume (RWX Block, `migratable: true`) the VM is down, not just the disk. If it is **`longhorn-scratch`** (single replica, CDI scratch space) losing it is expected and harmless — re-run the import. -### Verify the alert can even fire (measured 2026-09-13) +### Verify the alert can fire (measured 2026-09-13, and it was inert TWICE) -Prometheus on prod scrapes 19 jobs: `apiserver`, `cdi-prometheus-metrics`, `coredns`, -`keycloak-discovery`, `keycloak-service`, `keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, -`kube-prometheus-stack-grafana`, `kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, -`kube-state-metrics`, `kubelet`, `kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, -`node-exporter`, `oauth2-proxy`, `semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. -There is **no** Longhorn, cert-manager, Flux-controller or Hubble/Cilium job. Check the series this -rule depends on before you trust its silence: +This rule was inert for two independent reasons, both fixed on 2026-09-13. Either alone meant a +faulted volume — the most serious storage state on this cluster — alerted nobody, and announced +itself only as an application outage: + +1. **No scrape.** `count(longhorn_volume_robustness)` returned 0 series; the chart's + `metrics.serviceMonitor.enabled` defaulted false. Fixed in + `platform/longhorn/helmrelease.yaml`. +2. **Wrong metric shape.** With the series present the rule STILL could not fire. It read + `longhorn_volume_robustness == 3` on the assumption of a 0–3 enum; on longhorn 1.12.1 the metric + is one-hot over a `state` label with values 0/1 only. Measured on prod the same day: + `count(longhorn_volume_robustness)` = 96 (24 volumes × 4 states) and + `count(longhorn_volume_robustness == 3)` = **empty**. Fixed by keying on the label. + +Both directions are now pinned offline by `platform/monitoring/tests/longhorn.test.yaml`. Check the +live series before trusting this alert's silence: ``` -infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & \ - sleep 5; curl -s -G --data-urlencode "query=count by (job) (up)" localhost:19090/api/v1/query' +infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \ + prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 \ + 'count by (state) (longhorn_volume_robustness)' ``` -**This alert is currently INERT.** `count(longhorn_volume_robustness)` returned **0 series** on prod -on 2026-09-13 — Longhorn ships no ServiceMonitor here, so the whole `platform-health.longhorn` group -has no data. A faulted volume today announces itself as an application outage. That is a known open -defect; treat `volumes.longhorn.io` as the source of truth, not this alert's silence. +Four states, one entry per volume each, is healthy wiring. Empty means the scrape has broken and this +alert is inert again — treat `volumes.longhorn.io` as the source of truth, never this alert's silence. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev diff --git a/runbooks/targets-down.md b/runbooks/targets-down.md index 73643ca..1094441 100644 --- a/runbooks/targets-down.md +++ b/runbooks/targets-down.md @@ -25,7 +25,7 @@ of 2026-09-13 — every one of them means the corresponding alerts are **inert** | Missing scrape job | Alerts that cannot fire | |---|---| -| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` | +| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` | | cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` | | Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` | | Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` | @@ -129,8 +129,9 @@ An exporter that is up but publishing the wrong series is the hardest case — s monitoring one. - Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a chart value). Do not "fix" a blind spot by silencing the alert. -- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to - hand-apply monitors into the cluster — every durable change goes through git. +- The Longhorn and cert-manager scrapes were enabled on 2026-09-13 (the Longhorn row above is now + history). The Flux-controller and Hubble/Cilium monitor gaps are still open work. Note them; do not + try to hand-apply monitors into the cluster — every durable change goes through git. --- Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`