docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads

NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.

Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.

Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.

Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3715,3 +3715,53 @@ all three verified against public DNS. Requires the operator's sudo; the file wa
    and this incident is what that costs.
 
 Until 1 or 2 lands, README's Phase 2 "split-horizon CoreDNS" should be read as **in-cluster only**.
+
+---
+
+## 2026-09-14 — prod-1 I/O pressure critical: NOT a failing disk. Plus: confirmed malware in /downloads
+
+**The alert was real and the alerting worked.** `NodeIOPressureCritical` fired on k3s-prod-1 — the
+first time this class of alert has fired on prod, and only because the scrape gap was closed on
+2026-09-13. Peak I/O PSI **20.98%** (threshold 10%), sustained roughly 02:40–02:45Z, while prod-2/3
+stayed under 1%. It recovered unattended: PSI 0.94%, node Ready, `EtcdIsVoter: True`.
+
+**It is NOT the 2026-09-07 failing-drive repeat, and the evidence says so:**
+
+| measurement (during the spike) | value | reading |
+|---|---|---|
+| writes on prod-1 | sda 3.5 MB/s, sdi 2.9 MB/s | trivial volume |
+| reads | ~0 | — |
+| device utilisation | sdi 0.21, sda 0.09 | nowhere near saturated |
+| avg write latency | 1.4 – 4.8 ms | healthy; a dying drive shows hundreds of ms |
+| NFS RPC retransmissions | 0 | server not timing out |
+| NFS write throughput (measured after) | 21.4 MB/s | healthy |
+| `kube_node_status_condition{Ready}` min over 2h | 1 | KSM never observed NotReady; any flap was sub-scrape |
+| Longhorn volumes degraded | 0 of 24 | not out long enough to unseat a replica |
+
+3.5 MB/s of writes producing a 21% stall, with healthy device latency, is not a hardware problem. It
+is a transient stall of blocking I/O — and `node_pressure_io_stalled_seconds_total` counts *any*
+blocking I/O, NFS included, which is why local disk metrics look innocent.
+
+**The structural finding — media pods are concentrated.** `sonarr`, `qbittorrent` AND `jackett` all
+run on **k3s-prod-1**, while radarr is on prod-3 and plex on prod-2. Three of the four NFS-heavy
+workloads share one node, and it is the node with the I/O history whose drive was already replaced
+once. This is the same concentration class as the CNPG primaries fixed on 2026-09-13 (all three
+primaries had landed on one node), and it has the same remedy shape: spread them deliberately rather
+than letting the scheduler stack them. NOT fixed — it wants pod anti-affinity in the catalog/renderer
+so any NFS-mounting service repels its peers, which is a renderer contract change, not a hand patch.
+
+### CONFIRMED MALWARE in the bulk share — `/media/downloads`
+Identified by magic bytes, read-only, nothing executed:
+
+    Slow Horses S06E01 … NTb.exe      4D 5A ("MZ")  Windows PE executable   926,131,215 bytes
+    South Park S29E01 … NTb.exe       4D 5A ("MZ")  Windows PE executable   926,131,215 bytes
+    It's Always Sunny … DTS5.zipx     50 4B ("PK")  ZIP archive           1,102,185,621 bytes
+
+**Both `.exe` files have the same MD5, `054ada1e379c100e323367fe7555fb41`** — one binary, padded to
+~926 MB so it passes for a 1080p episode, planted under two different show names. Present since
+2026-09-07 / 09-11, i.e. they predate the qBittorrent `/downloads` mount landing, so they arrived by
+some earlier path.
+
+The cluster is not compromised by their presence: it is Linux, nothing invokes them, and they were
+never executed. The exposure is any **Windows** client that browses the share and opens one. Flagged
+to the operator three times; awaiting consent to delete rather than removing their data unilaterally.