docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
5c96f4f3a7
commit
3de79f81a5
1 file changed
+25
@@ -3833,3 +3833,28 @@ Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB f
|
||||
16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min
|
||||
>10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier
|
||||
coinciding with velero's first full volume backup.
|
||||
|
||||
**THREE OF MY mountstats FIGURES WERE WRONG — corrected 2026-09-15.** I handed an agent a baseline of
|
||||
"server response 4.0–4.8 ms/op, client queue time 0 ms/op, 0 major timeouts over 24h, backlog 0".
|
||||
It re-measured and found:
|
||||
- **client queue time is 0.013–0.067 ms/op, max 0.333 — not zero.** I read rounded output as zero.
|
||||
- **"0 over 24h" was really 0 over about ten minutes.** The node-exporter DaemonSet had just rolled to
|
||||
pick up the collector: `time() - process_start_time_seconds` read 558–561 s, and
|
||||
`count_over_time(...[24h])` returned **18** samples rather than ~2,880. I quoted a 24-hour baseline
|
||||
from ten minutes of data. Same error class as the label-values mistake above — trusting a query
|
||||
whose window was never valid. No shipped rule uses a range longer than 15m for that reason.
|
||||
- **the baseline was an IDLE one.** No READ or WRITE traffic was flowing; 4.4 ms is a cached STATFS,
|
||||
not a 1 MB read off a spinning array, and STATFS keepalive is the top operation on every mount. So
|
||||
thresholds could not be derived as a multiple of it and were argued from the link and the array
|
||||
instead.
|
||||
|
||||
Two genuine discoveries from that re-measurement:
|
||||
- **Mounts sharing an NFS superblock share the kernel's RPC statistics.** prod-1's
|
||||
`/mnt/Storage/plex` and `/mnt/Storage/plex/downloads` report identical op rates to every digit, as
|
||||
do prod-3's two nextcloud mounts. One slow export therefore raises two alerts. Left documented
|
||||
rather than aggregated away, because aggregating hides which path is affected.
|
||||
- **`maximum_rpc_slots` high-water: prod-1 586, prod-3 162, prod-2 2.** prod-1's slot table has grown
|
||||
enormously relative to its peers — the client responds to pressure by growing the table rather than
|
||||
queueing, which is also why its queue time stays near zero. This is an independent signal that
|
||||
prod-1 sustains far more concurrent NFS work than the other two nodes, and it is consistent with
|
||||
three of the four NFS-heavy pods being scheduled there.
|
||||
Reference in new issue
Block a user