diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index eee4568..c1f55a0 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3765,3 +3765,51 @@ some earlier path. The cluster is not compromised by their presence: it is Linux, nothing invokes them, and they were never executed. The exposure is any **Windows** client that browses the share and opens one. Flagged to the operator three times; awaiting consent to delete rather than removing their data unilaterally. + +--- + +## 2026-09-14 — DECIDED (owner): leave the /downloads files; don't move the NFS workloads; instrument instead + +Three decisions, all the owner's, closing questions I had raised repeatedly. + +**1. The `.exe` / `.zipx` files stay.** I had confirmed them as malware by magic bytes — two Windows +PE binaries sharing one MD5, padded to ~926 MB and named as two different shows — and recommended +deletion. The owner declined, with reasons that make the risk assessment different from mine: +- the **Proxmox-hosted version of this stack still has operational control of these files**, so + deleting them here just makes sonarr/radarr re-download them; +- **nothing touches `/downloads` except the arr suite**; +- **there are no Windows machines anywhere** on the network. +The exposure I was worried about was a Windows client opening one. With no Windows hosts and no +consumer other than the arrs, the files are inert payloads on a Linux-only NFS share. Recorded so a +future reader does not "discover" them and delete them, which would trigger a re-download. + +**2. No anti-affinity for the NFS-heavy media pods.** I found `sonarr`, `qbittorrent` and `jackett` +all on k3s-prod-1 and proposed spreading them. Declined — "no reason to move the heavy nfs workload". +So the concentration is accepted, not overlooked. + +**3. The real question is WHY the I/O pressure happens, and the answer is to instrument it.** Owner's +framing: the pressure appears seemingly at random, so add the alerting that supplies CONTEXT for the +next occurrence rather than chasing this one. + +### The finding that makes it tractable: NFS is entirely unmonitored +`/api/v1/label/__name__/values` filtered for `node_nfs*` / `node_mountstats*` / `nfs*` returns an +**empty list**. node-exporter's `nfs` and `mountstats` collectors are off by default and nothing here +enables them. So a cluster whose whole media library and the legacy Nextcloud dataset live on four NFS +mounts against `192.168.2.102` has **zero NFS observability** — while +`node_pressure_io_stalled_seconds_total` counts NFS stalls as I/O stalls. + +That is why tonight could not be explained from metrics: 20.98% stall with local devices at 0.09–0.21 +utilisation and 1.4–4.8 ms latency. The stall was real and the local disks were idle, and there was no +third signal to consult. Enabling those collectors is the prerequisite for any rule that hopes to say +*which* I/O stalled. + +**A hypothesis the instrumentation should be built to test**, following from decision 1: if the +Proxmox stack still drives these files, the I/O source is **outside this cluster**, which would explain +the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. The +in-cluster signal cannot see the other consumer, but it CAN say "the stall is on NFS and our own NFS +traffic does not account for it", which points straight at it. + +Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB fsync, 21.4 MB/s at +16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min +>10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier +coinciding with velero's first full volume backup.