Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

merge: the arrs had no download client at all, and now agree on paths

Browse Source 
Investigating the path disagreement found the larger defect: sonarr and
radarr each returned [] from /api/v3/downloadclient — Completed Download
Handling had never been wired up, not merely broken. Both now have
qBittorrent configured plus a remote path mapping /downloads/ ->
/media/downloads/, proven with a real completed torrent and a negative
control: same torrent, same client, mapping absent gave an unresolvable
/downloads/... and mapping present resolved the real file at inode 36144.

This state lives in each arr's database, NOT in git — a rebuild from git
does not restore it. Recorded in DECISIONS-LOG with the recommendation to
close it properly via path agreement instead.
This commit is contained in:
jakman85 committed 2026-09-13 00:01:55 -05:00
2 parents 4865734c5f 2f38b8156b
commit 55b90849d1
2 files changed
+203 -6
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DECISIONS-LOG.md
+181
View File
@@ -2688,3 +2688,184 @@ removing the readiness gate, or shortening `for:` each makes a test *fail*, so t
- **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`.
`cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description`
carries the full actionable procedure so the missing page is not load-bearing.
## Remote Path Mappings: the arrs can finally find what qBittorrent finished — 2026-09-12
The previous session gave qBittorrent a writable download destination (`/downloads` =
`192.168.2.102:/mnt/Storage/plex/downloads`, `subPath: downloads` of the export sonarr/radarr mount at
`/media`). That closed "nowhere to write". It did **not** close "the two sides disagree about the name
of the same inode": qBittorrent calls a finished release `/downloads/<name>`, sonarr/radarr call the
same bytes `/media/downloads/<name>`. Completed Download Handling looks up the path the client
reports, so the download finished and then nothing imported.
### What was actually broken was bigger than the path
Before touching path mapping I checked the prerequisite the brief asked for — and the arrs had **no
download client at all**:
```
sonarr GET /api/v3/downloadclient -> []
radarr GET /api/v3/downloadclient -> []
```
So "Completed Download Handling fails" was, strictly, "Completed Download Handling was never wired
up". Reachability itself was fine, and the `dependencies: []` worry was unfounded in the direction
that matters: the catalog's `qbittorrent` entry is default-deny **egress**, but sonarr and radarr each
declare `- to: qbittorrent / port 8080`, and the renderer emits the reciprocal ingress allows
(`qbittorrent-allow-sonarr-8080`, `qbittorrent-allow-radarr-8080` in `apps/wave-0/qbittorrent.yaml`).
Measured from inside each pod:
| from | DNS | HTTP |
|---|---|---|
| sonarr | `qbittorrent.qbittorrent.svc.cluster.local` -> 10.43.121.152 | `403 Forbidden`, connect 0.0012 s |
| radarr | same | `403 Forbidden`, connect 0.0036 s |
`403` is the *good* answer here: TCP connected and qBittorrent's own WebUI answered, demanding auth.
Also note `Service/qbittorrent` port 8080 -> `targetPort 8080`, i.e. straight to the app container;
only 4180 goes through oauth2-proxy. The arrs reach the API, not the SSO proxy.
### What was configured (in each arr's own database, via its HTTP API)
Both, identically, built from each app's own `/api/v3/downloadclient/schema` rather than a hand-typed
body:
- **Download client** `qBittorrent` (`implementation: QBittorrent`), host
`qbittorrent.qbittorrent.svc.cluster.local`, port 8080, no TLS, user `admin`, category
`tv-sonarr` / `radarr`, `enable: true`, `removeCompletedDownloads: true`.
- **Remote Path Mapping** host `qbittorrent.qbittorrent.svc.cluster.local`,
remote `/downloads/` -> local `/media/downloads/`.
**Idempotent by construction, and proven so by re-running.** The script reads the existing collections
first and matches on identity, not on count: a client is a duplicate if
`implementation == "QBittorrent"` *and* its `host` field equals the target host; a mapping is a
duplicate if `host` matches *and* `remotePath` matches after trailing-slash normalisation. First run
created (`HTTP 201` x4). Second run printed `IDEMPOTENT SKIP` for all four and created nothing.
One honest wrinkle in the second run: `POST /downloadclient/test` returned `400` with
`{"propertyName":"Name","errorMessage":"Should be unique"}` — that is name-collision validation
against the client already saved, **not** a connectivity failure. The right way to test a persisted
client is `POST /api/v3/downloadclient/testall`, which tests by id:
```
sonarr -> [{"id":1,"isValid":true,"validationFailures":[]}]
radarr -> [{"id":1,"isValid":true,"validationFailures":[]}]
```
### Proof of the chain, with a negative control
An 8 MiB payload was written **at qBittorrent's own path** as
`/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv` (+ a movie-shaped twin), owned
`1000:3000`. Same inode from both sides, which is the whole point:
```
qbittorrent /downloads/.../Test.Show...mkv dev=2097261 inode=36144 size=8388608
sonarr /media/downloads/.../Test.Show...mkv dev=2097261 inode=36144 size=8388608
radarr /media/downloads/.../Test.Movie...mkv dev=2097361 inode=34783 size=8388608
```
(The differing `dev` for radarr is a separate NFS mount instance on a different node; the inode is the
identity that matters.) And neither arr can see the literal remote path — the defect, stated:
`ls -d /downloads/` -> `No such file or directory` in both.
A **completed torrent was then made real**, because a mapping is only exercised when the client
*reports* a path. A minimal trackerless, `private: 1` .torrent describing that exact 8 MiB file was
generated and added with `savepath=/downloads`, `category=tv-sonarr`, `skip_checking=true`,
`stopped=true` — private + no trackers means qBittorrent performed **no announce, no DHT, no PEX**, so
nothing left the cluster. qBittorrent then reported it as genuinely finished:
```
state=stoppedUP progress=1 completed=8388608
content_path=/downloads/Test.Show.S01E01.1080p.WEB-DL.x264-CLAUDETEST/...mkv
```
Then `RefreshMonitoredDownloads`, with and without the mapping. **This is the load-bearing evidence:**
| Sonarr's own `GET /api/v3/queue` | `outputPath` |
|---|---|
| mapping **deleted** | `/downloads/Test.Show...CLAUDETEST/...mkv` — a path that does not exist for Sonarr |
| mapping **present** | `/media/downloads/Test.Show...CLAUDETEST/...mkv` — the real file, inode 36144 |
Same torrent, same client, one row of app state between "unresolvable" and "resolved". A
`GET /api/v3/manualimport?folder=/media/downloads/<release>` on each arr independently parsed the file
through the local side of the mapping — `quality: WEBDL-1080p`, `releaseGroup: CLAUDETEST` — so the
scan resolves where it previously could not.
**What was NOT proven: an actual import.** Both manual-import scans end in a permanent rejection —
`Unknown Series` (sonarr) / `Unknown Movie` (radarr) — and the queue row says `Series title mismatch`.
That is not a path failure; it is an **empty library**: `GET /api/v3/series` -> `0`,
`GET /api/v3/movie` -> `0`. No file was hardlinked into `/media/tv` as part of this work, and nothing
should be until the library is populated. Hardlinkability across `/media/downloads -> /media/tv` was
already proven live last session (same device, link count 2, no EXDEV).
All test state removed and verified: torrent deleted from qBittorrent with `deleteFiles=false` then
the payloads `rm -rf`'d; `ls | grep -c CLAUDETEST` -> `0` from qBittorrent, sonarr **and** radarr;
`torrents/info` -> `[]`; sonarr queue `totalRecords: 0`. The two download clients and two mappings
survive, re-verified after cleanup.
### THE WART: this is app state, not git
**A rebuild from git alone does NOT restore any of it.** The download clients and the Remote Path
Mappings live in `sonarr.db` / `radarr.db` on each app's Longhorn config PVC. `catalog/*.yaml` and
`apps/wave-1/*.yaml` describe the *pods, volumes and network policy* — nothing in this repo describes
"sonarr has a download client" or "here is how to translate its paths". Restore the cluster from git
with empty PVCs and the import chain is silently back to broken, with no gate failing. The same is
true of the symmetric option: qBittorrent's `SavePath` lives in
`/config/qBittorrent/qBittorrent.conf` on **its** config PVC, which is why the disagreement exists at
all — two apps, two private databases, two sources of truth, and git is neither.
Two ways out, and they are not equal:
1. **Path agreement** — mount the `downloads` subPath at `/media/downloads` in the qbittorrent pod
instead of `/downloads`, and repoint `Session\DefaultSavePath` / `Downloads\SavePath` to match.
Then every component names the same inode the same way and **the mappings become unnecessary**.
Half of it is declarative: `mountPath` is a line in `catalog/qbittorrent.yaml`, i.e. in git and
gate-covered. The other half is still a write to qBittorrent's config PVC — but a *one-time* write
whose result git then enforces the shape of, rather than two rows per arr that git never mentions.
2. **A reconciler** — a Job/CronJob that PUTs the desired clients + mappings into each arr's API from
a committed manifest (the `*arr-config`/Recyclarr pattern). Honest and general, restores on
rebuild, works for indexers and root folders too — but it is new machinery, needs each arr's API
key as a secret, and adds a component to own.
**Recommendation: (1) path agreement, and I have not done it.** It deletes the failure mode instead of
compensating for it, and it moves the load-bearing part into git where this repo's gate can see it.
But it changes a running download client's save path — the exact class of change the deferral in
`catalog/qbittorrent.yaml` shows is easy to get wrong — and it is a design decision, not a bug fix, so
it is **put to the operator, not taken unilaterally**. The mappings asked for are in place and working
today either way; if (1) is approved they simply get deleted afterwards.
### Blocker that outlives this change: qBittorrent has no durable WebUI credential
`qBittorrent.conf` contains no `WebUI\Username` and no `WebUI\Password_PBKDF2`, so v5.1.2 falls back
to user `admin` plus a **temporary password regenerated on every start**, announced only on stdout
(`The WebUI administrator password was not set. A temporary password is provided for this session:`).
That is what the passing connection tests above authenticate with — read out of the pod's log, piped
through stdin, never written to disk or printed.
**So the download clients will break the next time the qbittorrent pod restarts.** Nothing about the
mappings is affected; only the stored password goes stale, and both arrs will report the client
unavailable until it is updated. Fixing it properly means either a real password (SOPS-encrypted,
materialised into `qBittorrent.conf` — the app takes no password env var) or
`WebUI\AuthSubnetWhitelistEnabled=true` + `AuthSubnetWhitelist=10.42.0.0/16`, which the Cilium
default-deny already narrows to exactly sonarr and radarr. Both are writes to qBittorrent's config
PVC and both change the operator's own login story, so **neither was done**. Flagged for the operator.
### Layout consistency checked while in there (reported, not restructured)
- **Sonarr root folder** `/media/tv`, `accessible: true`, 7.6 TB free — same mount as
`/media/downloads`, so hardlinked imports stay possible. But **0 series in the database** against
~100 unmapped folders on disk; the library exists on the export and Sonarr does not know about it.
- **Radarr has NO root folder at all** (`GET /api/v3/rootfolder` -> `[]`) and 0 movies, while
`/media/movies` exists on the export. Radarr cannot import anything until a root folder is added.
Not added here — where the movie library lives is the operator's call.
- **qBittorrent categories** were empty; `tv-sonarr` and `radarr` now exist (the arrs create their own
category on test/grab), both with `savePath: ""`, so both inherit the global `save_path=/downloads`
and the single `/downloads/ -> /media/downloads/` mapping covers both. `auto_tmm_enabled=false` and
`use_category_paths_in_manual_mode=false`, so categories will not start inventing subdirectories. If
a per-category save path is ever set, the mapping set must be revisited.
- **`temp_path=/downloads/incomplete` with `temp_path_enabled=false`** — incomplete data lands in
`/downloads` alongside finished data. `incomplete/` does exist on the export.
- **Pre-existing content in the downloads area, untouched:** a `complete/` and `incomplete/` tree, a
**dangling symlink `downloads -> /shared/downloads`** (a legacy non-k3s path that resolves nowhere
in these pods), and three ~1 GB files named `...S18E05....zipx` / `...S06E01....exe` /
`...S29E01....exe`. Media files with `.exe`/`.zipx` extensions are worth an operator's eye; nothing
here read or moved them.
catalog/qbittorrent.yaml
+22 -6
View File
@@ -100,16 +100,32 @@ storage:
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
# non-cluster consumers, which should never be a surprise.
#
# PATH AGREEMENT WITH THE ARRS IS *NOT* CLOSED BY THIS CHANGE — see the report/DECISIONS-LOG. This
# pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
# ONE Remote Path Mapping per arr (remote host `qbittorrent`, remote path /downloads/, local path
# /media/downloads/). That lives in the arrs' own databases (app state on their config PVCs), not in
# git, so it is an operator step and is called out rather than silently assumed. The alternative —
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
# path /downloads/, local path /media/downloads/).
#
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
# download client itself — which did not exist in EITHER arr until then (both
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
# present — same torrent, same client.
#
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
# not chosen because it changes a running download client's save path, which the deferral above
# already shows is easy to get wrong.
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
# decision, not a technical blocker.
#
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
- name: downloads
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export