docs: correct three mountstats figures I supplied from bad measurements
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major
timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read
rounded output as zero); the 24h claim was really ten minutes, because the
DaemonSet had just rolled and the range query returned 18 samples rather than
~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real
media I/O and thresholds could not be multiples of it.

The 24h error is the same class as the label-values mistake recorded above:
trusting a query whose window was never valid.

Records two real discoveries: mounts sharing an NFS superblock share kernel RPC
statistics, so prod-1's two plex mounts are one signal reported twice; and
maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an
independent indication that prod-1 carries far more concurrent NFS work.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 23:22:04 -05:00
1 parent 5c96f4f3a7
commit 3de79f81a5
1 file changed
+25
+25
View File
@@ -3833,3 +3833,28 @@ Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB f
16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min
>10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier
coinciding with velero's first full volume backup.
**THREE OF MY mountstats FIGURES WERE WRONG — corrected 2026-09-15.** I handed an agent a baseline of
"server response 4.0–4.8 ms/op, client queue time 0 ms/op, 0 major timeouts over 24h, backlog 0".
It re-measured and found:
- **client queue time is 0.013–0.067 ms/op, max 0.333 — not zero.** I read rounded output as zero.
- **"0 over 24h" was really 0 over about ten minutes.** The node-exporter DaemonSet had just rolled to
pick up the collector: `time() - process_start_time_seconds` read 558–561 s, and
`count_over_time(...[24h])` returned **18** samples rather than ~2,880. I quoted a 24-hour baseline
from ten minutes of data. Same error class as the label-values mistake above — trusting a query
whose window was never valid. No shipped rule uses a range longer than 15m for that reason.
- **the baseline was an IDLE one.** No READ or WRITE traffic was flowing; 4.4 ms is a cached STATFS,
not a 1 MB read off a spinning array, and STATFS keepalive is the top operation on every mount. So
thresholds could not be derived as a multiple of it and were argued from the link and the array
instead.
Two genuine discoveries from that re-measurement:
- **Mounts sharing an NFS superblock share the kernel's RPC statistics.** prod-1's
`/mnt/Storage/plex` and `/mnt/Storage/plex/downloads` report identical op rates to every digit, as
do prod-3's two nextcloud mounts. One slow export therefore raises two alerts. Left documented
rather than aggregated away, because aggregating hides which path is affected.
- **`maximum_rpc_slots` high-water: prod-1 586, prod-3 162, prod-2 2.** prod-1's slot table has grown
enormously relative to its peers — the client responds to pressure by growing the table rather than
queueing, which is also why its queue time stays near zero. This is an independent signal that
prod-1 sustains far more concurrent NFS work than the other two nodes, and it is consistent with
three of the four NFS-heavy pods being scheduled there.