From dafa9cd2412ad2d05f793b5aae245068903946c3 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 21:55:24 -0500 Subject: [PATCH] docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered unattended) — the first time this alert has fired on prod, and only because the scrape gap was closed yesterday. Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions, 21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed NotReady. A 21% stall on trivial volume with healthy latency is transient blocking I/O, not hardware — and PSI counts NFS too. Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the node with the I/O history. Same concentration class as the CNPG primaries. Separately, the /downloads files are confirmed malware: two PE executables ('MZ') sharing one MD5, padded to ~926 MB and named as two different shows, plus a ZIP masquerading as video. Not executed, cluster not compromised; the risk is a Windows client opening them. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 2f49adc..eee4568 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3715,3 +3715,53 @@ all three verified against public DNS. Requires the operator's sudo; the file wa and this incident is what that costs. Until 1 or 2 lands, README's Phase 2 "split-horizon CoreDNS" should be read as **in-cluster only**. + +--- + +## 2026-09-14 — prod-1 I/O pressure critical: NOT a failing disk. Plus: confirmed malware in /downloads + +**The alert was real and the alerting worked.** `NodeIOPressureCritical` fired on k3s-prod-1 — the +first time this class of alert has fired on prod, and only because the scrape gap was closed on +2026-09-13. Peak I/O PSI **20.98%** (threshold 10%), sustained roughly 02:40–02:45Z, while prod-2/3 +stayed under 1%. It recovered unattended: PSI 0.94%, node Ready, `EtcdIsVoter: True`. + +**It is NOT the 2026-09-07 failing-drive repeat, and the evidence says so:** + +| measurement (during the spike) | value | reading | +|---|---|---| +| writes on prod-1 | sda 3.5 MB/s, sdi 2.9 MB/s | trivial volume | +| reads | ~0 | — | +| device utilisation | sdi 0.21, sda 0.09 | nowhere near saturated | +| avg write latency | 1.4 – 4.8 ms | healthy; a dying drive shows hundreds of ms | +| NFS RPC retransmissions | 0 | server not timing out | +| NFS write throughput (measured after) | 21.4 MB/s | healthy | +| `kube_node_status_condition{Ready}` min over 2h | 1 | KSM never observed NotReady; any flap was sub-scrape | +| Longhorn volumes degraded | 0 of 24 | not out long enough to unseat a replica | + +3.5 MB/s of writes producing a 21% stall, with healthy device latency, is not a hardware problem. It +is a transient stall of blocking I/O — and `node_pressure_io_stalled_seconds_total` counts *any* +blocking I/O, NFS included, which is why local disk metrics look innocent. + +**The structural finding — media pods are concentrated.** `sonarr`, `qbittorrent` AND `jackett` all +run on **k3s-prod-1**, while radarr is on prod-3 and plex on prod-2. Three of the four NFS-heavy +workloads share one node, and it is the node with the I/O history whose drive was already replaced +once. This is the same concentration class as the CNPG primaries fixed on 2026-09-13 (all three +primaries had landed on one node), and it has the same remedy shape: spread them deliberately rather +than letting the scheduler stack them. NOT fixed — it wants pod anti-affinity in the catalog/renderer +so any NFS-mounting service repels its peers, which is a renderer contract change, not a hand patch. + +### CONFIRMED MALWARE in the bulk share — `/media/downloads` +Identified by magic bytes, read-only, nothing executed: + + Slow Horses S06E01 … NTb.exe 4D 5A ("MZ") Windows PE executable 926,131,215 bytes + South Park S29E01 … NTb.exe 4D 5A ("MZ") Windows PE executable 926,131,215 bytes + It's Always Sunny … DTS5.zipx 50 4B ("PK") ZIP archive 1,102,185,621 bytes + +**Both `.exe` files have the same MD5, `054ada1e379c100e323367fe7555fb41`** — one binary, padded to +~926 MB so it passes for a 1080p episode, planted under two different show names. Present since +2026-09-07 / 09-11, i.e. they predate the qBittorrent `/downloads` mount landing, so they arrived by +some earlier path. + +The cluster is not compromised by their presence: it is Linux, nothing invokes them, and they were +never executed. The exposure is any **Windows** client that browses the share and opens one. Flagged +to the operator three times; awaiting consent to delete rather than removing their data unilaterally.