fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied

Two things found while reviewing the Longhorn alert work.

Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already
naming the right intent — 'headroom for WAL + head + compaction scratch'.
5 GiB is not that: retentionSize counts only persisted blocks, and a large
compaction can need roughly the size of the blocks being merged. ENOSPC on
/prometheus wedges every WAL commit, so scrapes keep succeeding while nothing
is stored and every absent()/==0 alert fires falsely at once. Lowered to
36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains
the binding limit.

runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore-
PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened
5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo —
the work is on the unmerged fix/etcd-io-priority branch. The page is read
during etcd I/O starvation on a node whose WAL shares a device with Longhorn,
so it was telling an operator mid-incident that the cap they depend on is in
place when it is not. Corrected, with the two mitigations that ARE real named
and a command to verify any of them.

Co-Authored-By: Claude Opus 5 <[email protected]>

clusters/prod/kustomization.yaml

@@ -349,8 +349,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"

runbooks/longhorn-degraded.md

@@ -134,8 +134,24 @@ replicas are genuinely gone (an OS re-provision), so replenishment can actually
 `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
 held**; do not "fix" this by setting a knob that is already set. `replicaRebuildingBandwidthLimit` is
 **v2/SPDK-only** and does nothing here; `rebuildConcurrentSyncLimit` is already at its floor of 1.
-What was actually tightened after that incident was every *other* Longhorn I/O source:
-`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 → 1.
+What was *intended* after that incident was to tighten every **other** Longhorn I/O source —
+`concurrentVolumeBackupRestorePerNodeLimit` and `snapshotHeavyTaskConcurrentLimit` 5 -> 1.
+
+> **DO NOT RELY ON THAT: it was never applied.** Measured on prod 2026-09-13,
+> `snapshot-heavy-task-concurrent-limit` = **5** and
+> `concurrent-volume-backup-restore-per-node-limit` = **5**. Neither name appears anywhere in this
+> repository, so nothing declares them and nothing would restore them. The work lives on the
+> UNMERGED branch `fix/etcd-io-priority` ("cap every OTHER I/O source, and delete one setting that
+> was inert"), whose tree is far behind `main` and must be cherry-picked, not merged.
+>
+> Only two of the four mitigations this page implies are real:
+> `concurrent-replica-rebuild-per-node-limit` = **1** (its floor, as stated) and
+> `node-drain-policy` = **block-if-contains-last-replica**. Verify any limit before you trust it:
+> `kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`.
+>
+> This matters here specifically because the incident being described is etcd I/O starvation on a
+> node whose WAL shares a device with Longhorn. If you are reading this page during that failure,
+> the I/O cap you are counting on is not in place.
 
 ### 2. A node is still down
 Then the rebuild cannot complete and this will not clear. Go to `/runbooks/node-down` first; there is