From 7369495b67cdc5f4c5499834f79e1eb834e3010f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 22:07:15 -0500 Subject: [PATCH] =?UTF-8?q?docs(pm):=20owner=20decisions=20=E2=80=94=20fil?= =?UTF-8?q?es=20stay,=20workloads=20stay,=20instrument=20NFS=20instead?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index eee4568..c1f55a0 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3765,3 +3765,51 @@ some earlier path. The cluster is not compromised by their presence: it is Linux, nothing invokes them, and they were never executed. The exposure is any **Windows** client that browses the share and opens one. Flagged to the operator three times; awaiting consent to delete rather than removing their data unilaterally. + +--- + +## 2026-09-14 — DECIDED (owner): leave the /downloads files; don't move the NFS workloads; instrument instead + +Three decisions, all the owner's, closing questions I had raised repeatedly. + +**1. The `.exe` / `.zipx` files stay.** I had confirmed them as malware by magic bytes — two Windows +PE binaries sharing one MD5, padded to ~926 MB and named as two different shows — and recommended +deletion. The owner declined, with reasons that make the risk assessment different from mine: +- the **Proxmox-hosted version of this stack still has operational control of these files**, so + deleting them here just makes sonarr/radarr re-download them; +- **nothing touches `/downloads` except the arr suite**; +- **there are no Windows machines anywhere** on the network. +The exposure I was worried about was a Windows client opening one. With no Windows hosts and no +consumer other than the arrs, the files are inert payloads on a Linux-only NFS share. Recorded so a +future reader does not "discover" them and delete them, which would trigger a re-download. + +**2. No anti-affinity for the NFS-heavy media pods.** I found `sonarr`, `qbittorrent` and `jackett` +all on k3s-prod-1 and proposed spreading them. Declined — "no reason to move the heavy nfs workload". +So the concentration is accepted, not overlooked. + +**3. The real question is WHY the I/O pressure happens, and the answer is to instrument it.** Owner's +framing: the pressure appears seemingly at random, so add the alerting that supplies CONTEXT for the +next occurrence rather than chasing this one. + +### The finding that makes it tractable: NFS is entirely unmonitored +`/api/v1/label/__name__/values` filtered for `node_nfs*` / `node_mountstats*` / `nfs*` returns an +**empty list**. node-exporter's `nfs` and `mountstats` collectors are off by default and nothing here +enables them. So a cluster whose whole media library and the legacy Nextcloud dataset live on four NFS +mounts against `192.168.2.102` has **zero NFS observability** — while +`node_pressure_io_stalled_seconds_total` counts NFS stalls as I/O stalls. + +That is why tonight could not be explained from metrics: 20.98% stall with local devices at 0.09–0.21 +utilisation and 1.4–4.8 ms latency. The stall was real and the local disks were idle, and there was no +third signal to consult. Enabling those collectors is the prerequisite for any rule that hopes to say +*which* I/O stalled. + +**A hypothesis the instrumentation should be built to test**, following from decision 1: if the +Proxmox stack still drives these files, the I/O source is **outside this cluster**, which would explain +the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. The +in-cluster signal cannot see the other consumer, but it CAN say "the stall is on NFS and our own NFS +traffic does not account for it", which points straight at it. + +Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB fsync, 21.4 MB/s at +16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min +>10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier +coinciding with velero's first full volume backup.