fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied

Two things found while reviewing the Longhorn alert work.

Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already
naming the right intent — 'headroom for WAL + head + compaction scratch'.
5 GiB is not that: retentionSize counts only persisted blocks, and a large
compaction can need roughly the size of the blocks being merged. ENOSPC on
/prometheus wedges every WAL commit, so scrapes keep succeeding while nothing
is stored and every absent()/==0 alert fires falsely at once. Lowered to
36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains
the binding limit.

runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore-
PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened
5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo —
the work is on the unmerged fix/etcd-io-priority branch. The page is read
during etcd I/O starvation on a node whose WAL shares a device with Longhorn,
so it was telling an operator mid-incident that the cap they depend on is in
place when it is not. Corrected, with the two mitigations that ARE real named
and a command to verify any of them.

Co-Authored-By: Claude Opus 5 <[email protected]>

clusters/prod/kustomization.yaml

@@ -349,8 +349,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"