From 2f38b8156bdbe1ae7f180dfa8e81f2a724da1761 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 23:57:56 -0500 Subject: [PATCH] fix(arr): wire qBittorrent as a download client + Remote Path Mappings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit qBittorrent reports finished releases at /downloads/; sonarr and radarr see the same inodes at /media/downloads/, so Completed Download Handling could never resolve what the client handed it. Configured, on prod, via each app's own HTTP API: * download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080 (it did not exist in EITHER arr: both /api/v3/downloadclient returned []) * Remote Path Mapping /downloads/ -> /media/downloads/ for that host Both built from each app's /downloadclient/schema, matched on identity before writing, and re-running creates nothing. downloadclient/testall reports isValid=true in both. Proven with a negative control: Sonarr's queue showed outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it present, for the same completed torrent. This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a rebuild from git alone does not restore it. Recorded as a wart, with path agreement (repointing SavePath to /media/downloads) recommended as the durable fix and put to the operator rather than taken unilaterally. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 181 +++++++++++++++++++++++++++++++++++++++ catalog/qbittorrent.yaml | 28 ++++-- 2 files changed, 203 insertions(+), 6 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 9c745b2..76f07ce 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2688,3 +2688,184 @@ removing the readiness gate, or shortening `for:` each makes a test *fail*, so t - **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`. `cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description` carries the full actionable procedure so the missing page is not load-bearing. + +## Remote Path Mappings: the arrs can finally find what qBittorrent finished — 2026-09-12 + +The previous session gave qBittorrent a writable download destination (`/downloads` = +`192.168.2.102:/mnt/Storage/plex/downloads`, `subPath: downloads` of the export sonarr/radarr mount at +`/media`). That closed "nowhere to write". It did **not** close "the two sides disagree about the name +of the same inode": qBittorrent calls a finished release `/downloads/`, sonarr/radarr call the +same bytes `/media/downloads/`. Completed Download Handling looks up the path the client +reports, so the download finished and then nothing imported. + +### What was actually broken was bigger than the path + +Before touching path mapping I checked the prerequisite the brief asked for — and the arrs had **no +download client at all**: + +``` +sonarr GET /api/v3/downloadclient -> [] +radarr GET /api/v3/downloadclient -> [] +``` + +So "Completed Download Handling fails" was, strictly, "Completed Download Handling was never wired +up". Reachability itself was fine, and the `dependencies: []` worry was unfounded in the direction +that matters: the catalog's `qbittorrent` entry is default-deny **egress**, but sonarr and radarr each +declare `- to: qbittorrent / port 8080`, and the renderer emits the reciprocal ingress allows +(`qbittorrent-allow-sonarr-8080`, `qbittorrent-allow-radarr-8080` in `apps/wave-0/qbittorrent.yaml`). +Measured from inside each pod: + +| from | DNS | HTTP | +|---|---|---| +| sonarr | `qbittorrent.qbittorrent.svc.cluster.local` -> 10.43.121.152 | `403 Forbidden`, connect 0.0012 s | +| radarr | same | `403 Forbidden`, connect 0.0036 s | + +`403` is the *good* answer here: TCP connected and qBittorrent's own WebUI answered, demanding auth. +Also note `Service/qbittorrent` port 8080 -> `targetPort 8080`, i.e. straight to the app container; +only 4180 goes through oauth2-proxy. The arrs reach the API, not the SSO proxy. + +### What was configured (in each arr's own database, via its HTTP API) + +Both, identically, built from each app's own `/api/v3/downloadclient/schema` rather than a hand-typed +body: + +- **Download client** `qBittorrent` (`implementation: QBittorrent`), host + `qbittorrent.qbittorrent.svc.cluster.local`, port 8080, no TLS, user `admin`, category + `tv-sonarr` / `radarr`, `enable: true`, `removeCompletedDownloads: true`. +- **Remote Path Mapping** host `qbittorrent.qbittorrent.svc.cluster.local`, + remote `/downloads/` -> local `/media/downloads/`. + +**Idempotent by construction, and proven so by re-running.** The script reads the existing collections +first and matches on identity, not on count: a client is a duplicate if +`implementation == "QBittorrent"` *and* its `host` field equals the target host; a mapping is a +duplicate if `host` matches *and* `remotePath` matches after trailing-slash normalisation. First run +created (`HTTP 201` x4). Second run printed `IDEMPOTENT SKIP` for all four and created nothing. + +One honest wrinkle in the second run: `POST /downloadclient/test` returned `400` with +`{"propertyName":"Name","errorMessage":"Should be unique"}` — that is name-collision validation +against the client already saved, **not** a connectivity failure. The right way to test a persisted +client is `POST /api/v3/downloadclient/testall`, which tests by id: + +``` +sonarr -> [{"id":1,"isValid":true,"validationFailures":[]}] +radarr -> [{"id":1,"isValid":true,"validationFailures":[]}] +``` + +### Proof of the chain, with a negative control + +An 8 MiB payload was written **at qBittorrent's own path** as +`/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv` (+ a movie-shaped twin), owned +`1000:3000`. Same inode from both sides, which is the whole point: + +``` +qbittorrent /downloads/.../Test.Show...mkv dev=2097261 inode=36144 size=8388608 +sonarr /media/downloads/.../Test.Show...mkv dev=2097261 inode=36144 size=8388608 +radarr /media/downloads/.../Test.Movie...mkv dev=2097361 inode=34783 size=8388608 +``` + +(The differing `dev` for radarr is a separate NFS mount instance on a different node; the inode is the +identity that matters.) And neither arr can see the literal remote path — the defect, stated: +`ls -d /downloads/` -> `No such file or directory` in both. + +A **completed torrent was then made real**, because a mapping is only exercised when the client +*reports* a path. A minimal trackerless, `private: 1` .torrent describing that exact 8 MiB file was +generated and added with `savepath=/downloads`, `category=tv-sonarr`, `skip_checking=true`, +`stopped=true` — private + no trackers means qBittorrent performed **no announce, no DHT, no PEX**, so +nothing left the cluster. qBittorrent then reported it as genuinely finished: + +``` +state=stoppedUP progress=1 completed=8388608 +content_path=/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv +``` + +Then `RefreshMonitoredDownloads`, with and without the mapping. **This is the load-bearing evidence:** + +| Sonarr's own `GET /api/v3/queue` | `outputPath` | +|---|---| +| mapping **deleted** | `/downloads/Test.Show...CLAUDETEST/...mkv` — a path that does not exist for Sonarr | +| mapping **present** | `/media/downloads/Test.Show...CLAUDETEST/...mkv` — the real file, inode 36144 | + +Same torrent, same client, one row of app state between "unresolvable" and "resolved". A +`GET /api/v3/manualimport?folder=/media/downloads/` on each arr independently parsed the file +through the local side of the mapping — `quality: WEBDL-1080p`, `releaseGroup: CLAUDETEST` — so the +scan resolves where it previously could not. + +**What was NOT proven: an actual import.** Both manual-import scans end in a permanent rejection — +`Unknown Series` (sonarr) / `Unknown Movie` (radarr) — and the queue row says `Series title mismatch`. +That is not a path failure; it is an **empty library**: `GET /api/v3/series` -> `0`, +`GET /api/v3/movie` -> `0`. No file was hardlinked into `/media/tv` as part of this work, and nothing +should be until the library is populated. Hardlinkability across `/media/downloads -> /media/tv` was +already proven live last session (same device, link count 2, no EXDEV). + +All test state removed and verified: torrent deleted from qBittorrent with `deleteFiles=false` then +the payloads `rm -rf`'d; `ls | grep -c CLAUDETEST` -> `0` from qBittorrent, sonarr **and** radarr; +`torrents/info` -> `[]`; sonarr queue `totalRecords: 0`. The two download clients and two mappings +survive, re-verified after cleanup. + +### THE WART: this is app state, not git + +**A rebuild from git alone does NOT restore any of it.** The download clients and the Remote Path +Mappings live in `sonarr.db` / `radarr.db` on each app's Longhorn config PVC. `catalog/*.yaml` and +`apps/wave-1/*.yaml` describe the *pods, volumes and network policy* — nothing in this repo describes +"sonarr has a download client" or "here is how to translate its paths". Restore the cluster from git +with empty PVCs and the import chain is silently back to broken, with no gate failing. The same is +true of the symmetric option: qBittorrent's `SavePath` lives in +`/config/qBittorrent/qBittorrent.conf` on **its** config PVC, which is why the disagreement exists at +all — two apps, two private databases, two sources of truth, and git is neither. + +Two ways out, and they are not equal: + +1. **Path agreement** — mount the `downloads` subPath at `/media/downloads` in the qbittorrent pod + instead of `/downloads`, and repoint `Session\DefaultSavePath` / `Downloads\SavePath` to match. + Then every component names the same inode the same way and **the mappings become unnecessary**. + Half of it is declarative: `mountPath` is a line in `catalog/qbittorrent.yaml`, i.e. in git and + gate-covered. The other half is still a write to qBittorrent's config PVC — but a *one-time* write + whose result git then enforces the shape of, rather than two rows per arr that git never mentions. +2. **A reconciler** — a Job/CronJob that PUTs the desired clients + mappings into each arr's API from + a committed manifest (the `*arr-config`/Recyclarr pattern). Honest and general, restores on + rebuild, works for indexers and root folders too — but it is new machinery, needs each arr's API + key as a secret, and adds a component to own. + +**Recommendation: (1) path agreement, and I have not done it.** It deletes the failure mode instead of +compensating for it, and it moves the load-bearing part into git where this repo's gate can see it. +But it changes a running download client's save path — the exact class of change the deferral in +`catalog/qbittorrent.yaml` shows is easy to get wrong — and it is a design decision, not a bug fix, so +it is **put to the operator, not taken unilaterally**. The mappings asked for are in place and working +today either way; if (1) is approved they simply get deleted afterwards. + +### Blocker that outlives this change: qBittorrent has no durable WebUI credential + +`qBittorrent.conf` contains no `WebUI\Username` and no `WebUI\Password_PBKDF2`, so v5.1.2 falls back +to user `admin` plus a **temporary password regenerated on every start**, announced only on stdout +(`The WebUI administrator password was not set. A temporary password is provided for this session:`). +That is what the passing connection tests above authenticate with — read out of the pod's log, piped +through stdin, never written to disk or printed. + +**So the download clients will break the next time the qbittorrent pod restarts.** Nothing about the +mappings is affected; only the stored password goes stale, and both arrs will report the client +unavailable until it is updated. Fixing it properly means either a real password (SOPS-encrypted, +materialised into `qBittorrent.conf` — the app takes no password env var) or +`WebUI\AuthSubnetWhitelistEnabled=true` + `AuthSubnetWhitelist=10.42.0.0/16`, which the Cilium +default-deny already narrows to exactly sonarr and radarr. Both are writes to qBittorrent's config +PVC and both change the operator's own login story, so **neither was done**. Flagged for the operator. + +### Layout consistency checked while in there (reported, not restructured) + +- **Sonarr root folder** `/media/tv`, `accessible: true`, 7.6 TB free — same mount as + `/media/downloads`, so hardlinked imports stay possible. But **0 series in the database** against + ~100 unmapped folders on disk; the library exists on the export and Sonarr does not know about it. +- **Radarr has NO root folder at all** (`GET /api/v3/rootfolder` -> `[]`) and 0 movies, while + `/media/movies` exists on the export. Radarr cannot import anything until a root folder is added. + Not added here — where the movie library lives is the operator's call. +- **qBittorrent categories** were empty; `tv-sonarr` and `radarr` now exist (the arrs create their own + category on test/grab), both with `savePath: ""`, so both inherit the global `save_path=/downloads` + and the single `/downloads/ -> /media/downloads/` mapping covers both. `auto_tmm_enabled=false` and + `use_category_paths_in_manual_mode=false`, so categories will not start inventing subdirectories. If + a per-category save path is ever set, the mapping set must be revisited. +- **`temp_path=/downloads/incomplete` with `temp_path_enabled=false`** — incomplete data lands in + `/downloads` alongside finished data. `incomplete/` does exist on the export. +- **Pre-existing content in the downloads area, untouched:** a `complete/` and `incomplete/` tree, a + **dangling symlink `downloads -> /shared/downloads`** (a legacy non-k3s path that resolves nowhere + in these pods), and three ~1 GB files named `...S18E05....zipx` / `...S06E01....exe` / + `...S29E01....exe`. Media files with `.exe`/`.zipx` extensions are worth an operator's eye; nothing + here read or moved them. diff --git a/catalog/qbittorrent.yaml b/catalog/qbittorrent.yaml index d9d3889..fc0393e 100644 --- a/catalog/qbittorrent.yaml +++ b/catalog/qbittorrent.yaml @@ -100,16 +100,32 @@ storage: # — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with # non-cluster consumers, which should never be a surprise. # - # PATH AGREEMENT WITH THE ARRS IS *NOT* CLOSED BY THIS CHANGE — see the report/DECISIONS-LOG. This - # pod calls the finished file /downloads/; sonarr/radarr call the same bytes + # PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path + # Mappings". This pod calls the finished file /downloads/; sonarr/radarr call the same bytes # /media/downloads/. Same inode, different absolute path, so Completed Download Handling needs - # ONE Remote Path Mapping per arr (remote host `qbittorrent`, remote path /downloads/, local path - # /media/downloads/). That lives in the arrs' own databases (app state on their config PVCs), not in - # git, so it is an operator step and is called out rather than silently assumed. The alternative — + # ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote + # path /downloads/, local path /media/downloads/). + # + # THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent + # download client itself — which did not exist in EITHER arr until then (both + # `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up + # at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed + # reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it + # present — same torrent, same client. + # + # BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild + # from git alone does NOT restore them and no gate here notices their absence. The alternative — # repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead — # buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was # not chosen because it changes a running download client's save path, which the deferral above - # already shows is easy to get wrong. + # already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the + # mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator + # decision, not a technical blocker. + # + # RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username / + # WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every + # start. The arrs' saved download-client password therefore goes stale on every pod restart. See the + # same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally. - name: downloads class: bulk export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export