diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 51300a2..5db94a1 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3833,3 +3833,28 @@ Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB f 16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min >10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier coinciding with velero's first full volume backup. + +**THREE OF MY mountstats FIGURES WERE WRONG — corrected 2026-09-15.** I handed an agent a baseline of +"server response 4.0–4.8 ms/op, client queue time 0 ms/op, 0 major timeouts over 24h, backlog 0". +It re-measured and found: +- **client queue time is 0.013–0.067 ms/op, max 0.333 — not zero.** I read rounded output as zero. +- **"0 over 24h" was really 0 over about ten minutes.** The node-exporter DaemonSet had just rolled to + pick up the collector: `time() - process_start_time_seconds` read 558–561 s, and + `count_over_time(...[24h])` returned **18** samples rather than ~2,880. I quoted a 24-hour baseline + from ten minutes of data. Same error class as the label-values mistake above — trusting a query + whose window was never valid. No shipped rule uses a range longer than 15m for that reason. +- **the baseline was an IDLE one.** No READ or WRITE traffic was flowing; 4.4 ms is a cached STATFS, + not a 1 MB read off a spinning array, and STATFS keepalive is the top operation on every mount. So + thresholds could not be derived as a multiple of it and were argued from the link and the array + instead. + +Two genuine discoveries from that re-measurement: +- **Mounts sharing an NFS superblock share the kernel's RPC statistics.** prod-1's + `/mnt/Storage/plex` and `/mnt/Storage/plex/downloads` report identical op rates to every digit, as + do prod-3's two nextcloud mounts. One slow export therefore raises two alerts. Left documented + rather than aggregated away, because aggregating hides which path is affected. +- **`maximum_rpc_slots` high-water: prod-1 586, prod-3 162, prod-2 2.** prod-1's slot table has grown + enormously relative to its peers — the client responds to pressure by growing the table rather than + queueing, which is also why its queue time stays near zero. This is an independent signal that + prod-1 sustains far more concurrent NFS work than the other two nodes, and it is consistent with + three of the four NFS-heavy pods being scheduled there.