From d96165e8fb9f66691c52a5c14d4746fb9cd0d95a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 17:27:20 -0500 Subject: [PATCH] fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things found while reviewing the Longhorn alert work. Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already naming the right intent — 'headroom for WAL + head + compaction scratch'. 5 GiB is not that: retentionSize counts only persisted blocks, and a large compaction can need roughly the size of the blocks being merged. ENOSPC on /prometheus wedges every WAL commit, so scrapes keep succeeding while nothing is stored and every absent()/==0 alert fires falsely at once. Lowered to 36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains the binding limit. runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore- PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened 5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo — the work is on the unmerged fix/etcd-io-priority branch. The page is read during etcd I/O starvation on a node whose WAL shares a device with Longhorn, so it was telling an operator mid-incident that the cap they depend on is in place when it is not. Corrected, with the two mitigations that ARE real named and a command to verify any of them. Co-Authored-By: Claude Opus 5 --- clusters/prod/kustomization.yaml | 10 +++++++++- runbooks/longhorn-degraded.md | 20 ++++++++++++++++++-- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index adced93..9d56c3a 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -349,8 +349,16 @@ patches: # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the # size cap before 30d if series volume is high, keeping the disk from filling either way. - op: replace + # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for + # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL, + # not the head's in-flight segments, and not compaction scratch, and a large compaction can need + # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on + # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every + # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB + # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history, + # grow the PVC rather than the fraction of it. path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE - value: "45GiB" + value: "36GiB" - op: replace path: /data/MONITORING_LOKI_PVC_SIZE value: "50Gi" diff --git a/runbooks/longhorn-degraded.md b/runbooks/longhorn-degraded.md index 695a5e8..688d7ec 100644 --- a/runbooks/longhorn-degraded.md +++ b/runbooks/longhorn-degraded.md @@ -134,8 +134,24 @@ replicas are genuinely gone (an OS re-provision), so replenishment can actually `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is **v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1. -What was actually tightened after that incident was every *other* Longhorn I/O source: -`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1. +What was *intended* after that incident was to tighten every **other** Longhorn I/O source — +`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 -> 1. + +> **DO NOT RELY ON THAT: it was never applied.** Measured on prod 2026-09-13, +> `snapshot-heavy-task-concurrent-limit` = **5** and +> `concurrent-volume-backup-restore-per-node-limit` = **5**. Neither name appears anywhere in this +> repository, so nothing declares them and nothing would restore them. The work lives on the +> UNMERGED branch `fix/etcd-io-priority` ("cap every OTHER I/O source, and delete one setting that +> was inert"), whose tree is far behind `main` and must be cherry-picked, not merged. +> +> Only two of the four mitigations this page implies are real: +> `concurrent-replica-rebuild-per-node-limit` = **1** (its floor, as stated) and +> `node-drain-policy` = **block-if-contains-last-replica**. Verify any limit before you trust it: +> `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. +> +> This matters here specifically because the incident being described is etcd I/O starvation on a +> node whose WAL shares a device with Longhorn. If you are reading this page during that failure, +> the I/O cap you are counting on is not in place. ### 2. A node is still down Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is