docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead

Records three owner decisions and the finding that makes the third tractable.

The /downloads malware stays: the Proxmox-hosted stack still has operational
control of those files so deletion just triggers a re-download, nothing but the
arr suite touches the directory, and there are no Windows machines anywhere —
which removes the exposure I was concerned about. Written down so nobody
'discovers' and deletes them later.

Anti-affinity for the co-located NFS pods declined; the concentration is
accepted rather than overlooked.

And the enabling finding: there are NO NFS metrics in the TSDB at all —
node-exporter's nfs and mountstats collectors are off by default. A cluster
with four NFS mounts carrying the media library and legacy Nextcloud data has
zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is
precisely why a 20.98% stall alongside idle local disks could not be explained.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3765,3 +3765,51 @@ some earlier path.
 The cluster is not compromised by their presence: it is Linux, nothing invokes them, and they were
 never executed. The exposure is any **Windows** client that browses the share and opens one. Flagged
 to the operator three times; awaiting consent to delete rather than removing their data unilaterally.
+
+---
+
+## 2026-09-14 — DECIDED (owner): leave the /downloads files; don't move the NFS workloads; instrument instead
+
+Three decisions, all the owner's, closing questions I had raised repeatedly.
+
+**1. The `.exe` / `.zipx` files stay.** I had confirmed them as malware by magic bytes — two Windows
+PE binaries sharing one MD5, padded to ~926 MB and named as two different shows — and recommended
+deletion. The owner declined, with reasons that make the risk assessment different from mine:
+- the **Proxmox-hosted version of this stack still has operational control of these files**, so
+  deleting them here just makes sonarr/radarr re-download them;
+- **nothing touches `/downloads` except the arr suite**;
+- **there are no Windows machines anywhere** on the network.
+The exposure I was worried about was a Windows client opening one. With no Windows hosts and no
+consumer other than the arrs, the files are inert payloads on a Linux-only NFS share. Recorded so a
+future reader does not "discover" them and delete them, which would trigger a re-download.
+
+**2. No anti-affinity for the NFS-heavy media pods.** I found `sonarr`, `qbittorrent` and `jackett`
+all on k3s-prod-1 and proposed spreading them. Declined — "no reason to move the heavy nfs workload".
+So the concentration is accepted, not overlooked.
+
+**3. The real question is WHY the I/O pressure happens, and the answer is to instrument it.** Owner's
+framing: the pressure appears seemingly at random, so add the alerting that supplies CONTEXT for the
+next occurrence rather than chasing this one.
+
+### The finding that makes it tractable: NFS is entirely unmonitored
+`/api/v1/label/__name__/values` filtered for `node_nfs*` / `node_mountstats*` / `nfs*` returns an
+**empty list**. node-exporter's `nfs` and `mountstats` collectors are off by default and nothing here
+enables them. So a cluster whose whole media library and the legacy Nextcloud dataset live on four NFS
+mounts against `192.168.2.102` has **zero NFS observability** — while
+`node_pressure_io_stalled_seconds_total` counts NFS stalls as I/O stalls.
+
+That is why tonight could not be explained from metrics: 20.98% stall with local devices at 0.09–0.21
+utilisation and 1.4–4.8 ms latency. The stall was real and the local disks were idle, and there was no
+third signal to consult. Enabling those collectors is the prerequisite for any rule that hopes to say
+*which* I/O stalled.
+
+**A hypothesis the instrumentation should be built to test**, following from decision 1: if the
+Proxmox stack still drives these files, the I/O source is **outside this cluster**, which would explain
+the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. The
+in-cluster signal cannot see the other consumer, but it CAN say "the stall is on NFS and our own NFS
+traffic does not account for it", which points straight at it.
+
+Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB fsync, 21.4 MB/s at
+16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min
+>10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier
+coinciding with velero's first full volume backup.