jakman85andClaude Opus 5
d96165e8fb
Two things found while reviewing the Longhorn alert work. Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already naming the right intent — 'headroom for WAL + head + compaction scratch'. 5 GiB is not that: retentionSize counts only persisted blocks, and a large compaction can need roughly the size of the blocks being merged. ENOSPC on /prometheus wedges every WAL commit, so scrapes keep succeeding while nothing is stored and every absent()/==0 alert fires falsely at once. Lowered to 36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains the binding limit. runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore- PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened 5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo — the work is on the unmerged fix/etcd-io-priority branch. The page is read during etcd I/O starvation on a node whose WAL shares a device with Longhorn, so it was telling an operator mid-incident that the cap they depend on is in place when it is not. Corrected, with the two mitigations that ARE real named and a command to verify any of them. Co-Authored-By: Claude Opus 5 <[email protected]>