fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid

Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.

Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.

Two real bugs found in the WIP, both the postBuild-substitution trap:

  * PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
    (envsubst collapses it) and the renderer accepts it, but run any other way
    bash reads the doubled dollar as its PID, so every curl got an unparseable
    URL and the hook logged "the WebUI rejected the durable credential" three
    times having never asked it anything — see the marker log at 05:35Z. Now a
    literal PORT=8080; the script contains no braced expansion at all and
    behaves identically however it is run.
  * One shared wait budget covered both the s6 wait and the API wait, so the
    first consumed all 90s and attempts 2-3 fired instantly. Separate counters
    now; "the API never answered" is reported as a distinct fault from "the
    password was rejected"; and a rejection restarts qbittorrent-nox so a retry
    can actually change its outcome.

apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.

Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.

NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o
 
 Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits"
 only because their agents had not committed yet, and deleting those would have destroyed live work.
+
+## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`)
+
+**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor
+`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a
+fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences,
+and **both had already fired by the time this was finished** — this was not a hypothesis:
+
+- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ,
+  before any change: `POST /api/v3/downloadclient/testall` returned
+  `isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]`
+  on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a
+  failed client, not a failed import.
+- No human had a stable login either; every restart meant reading the pod log.
+
+**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32
+alphanumeric chars) is the single source of truth, consumed as env by the app container. A
+`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000
+iterations, 16-byte random salt, 64-byte digest, stored as
+`WebUI\Password_PBKDF2="@ByteArray(<b64 salt>:<b64 digest>)"` — asserts both keys into the conf on
+**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is
+never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken
+and makes auth depend on network policy staying correct for ever).
+
+**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential:
+`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does
+not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod
+to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from
+git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the
+credential into it before the WebUI is serving. The only input is the SOPS file in git.
+
+**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`,
+the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state
+in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger
+differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential
+once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after
+every restart in which the app serves a random password and both arrs are broken. A CronJob would also
+have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or
+`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose
+own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or
+API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it
+would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform
+the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has
+`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across
+every service's golden to buy something strictly weaker.
+
+**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook
+computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the
+cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes
+it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became
+`<pid>{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI
+rejected the durable credential" three times having never asked it anything**. That misdiagnosis is
+preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal
+`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically
+however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and
+"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have
+separate counters, "the API never answered" is reported as a distinct fault from "the password was
+rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome.
+
+**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the
+arrs at the correct password FAILED at first: the arr's own save-time validation returned
+`Authentication Failure`, and from inside the sonarr pod qBittorrent answered
+`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries
+against the stale password had banned both arr pod IPs. The credential was right the whole time. The
+ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed
+(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating
+failures). The hook spends at most 3 logins per start for exactly this reason.
+
+**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and
+Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every
+password field. Comparing against that makes every run look like a change — and it also means the
+earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The
+convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it
+says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing.
+
+**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted
+and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the
+WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still
+reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential
+rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization`
+produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write
+path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the
+helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed
+(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the
+digest it produced verifies under an independent PBKDF2 implementation with the git password and
+fails with a wrong one.
+
+**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod.
+The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live
+object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's
+permission classifier. The durability shown above is therefore the *credential's* durability across
+two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually
+invoking it is verified only on merge. **First thing to check after merging:** that
+`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential`
+line on the next pod start.
+
+**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's
+download-client rows — including this password — live in their own databases on their config PVCs.
+Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against
+both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records.
+
+**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as
+`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml`
+did not list it**, so Flux would never have applied it and the adoption the last handoff expected
+would silently not have happened. It is listed now (verified: the decrypted SOPS values are
+byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password).
+Note also that the live Secret carries the plaintext in a
+`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by
+hand; Flux's server-side apply on merge does not write that annotation.
+
+**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree
+cannot work here because the container mounts only the worktree while its `.git` points outside).
+spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden
+(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched
+requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left
+skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the
+only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a
+scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC
+bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of
+the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are
+about: a stage that reports green while verifying nothing.