Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid

Browse Source 
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.

Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.

Two real bugs found in the WIP, both the postBuild-substitution trap:

  * PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
    (envsubst collapses it) and the renderer accepts it, but run any other way
    bash reads the doubled dollar as its PID, so every curl got an unparseable
    URL and the hook logged "the WebUI rejected the durable credential" three
    times having never asked it anything — see the marker log at 05:35Z. Now a
    literal PORT=8080; the script contains no braced expansion at all and
    behaves identically however it is run.
  * One shared wait budget covered both the s6 wait and the API wait, so the
    first consumed all 90s and attempts 2-3 fired instantly. Separate counters
    now; "the API never answered" is reported as a distinct fault from "the
    password was rejected"; and a rejection restarts qbittorrent-nox so a retry
    can actually change its outcome.

apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.

Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.

NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 16:31:58 -05:00
1 parent 204f0ec28e
commit e4bd76070e
4 files changed
+195 -15
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

catalog/qbittorrent.yaml
+70 -13
View File
@@ -87,6 +87,19 @@ lifecycle:
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
# keeps in its own state), and the schema documents this hook for exactly that use.
#
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
# credential — because the WebUI is not running yet. That assertion is the most valuable line
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
# mean a schema version bump plus a renderer change — a blast radius across every service's
# golden — to buy something strictly weaker than what the hook already does.
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
#
# DURABILITY, the whole point — all three paths end with the SAME credential:
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
@@ -107,15 +120,33 @@ lifecycle:
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
#
# NOTE the `$${...}` escapes below — load-bearing. apps-wave-0 declares
# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at
# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst
# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the
# incident that taught this, and render.py's assert_substitution_safe, which enforces it).
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
# CLUSTER — but only there:
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
# however it is run.
set -u
CONF=/config/qBittorrent/qBittorrent.conf
SVC=/run/service/svc-qbittorrent
PORT=$${WEBUI_PORT:-8080}
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
# env var for a lookup to read anyway.
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
# logged "the WebUI rejected the durable
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
PORT=8080
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
@@ -237,8 +268,12 @@ lifecycle:
# have read the old, credential-less conf and be serving a random temporary password.
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
# Nothing to do when the service has not started yet - it will read the new file.
while [ "$waited" -lt 90 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
waited=$((waited + 1)); sleep 1
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
# 2026-09-13 failure spent the entire verification budget before the first login and
# then fired attempts 2 and 3 with no wait left at all.
sv=0
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
sv=$((sv + 1)); sleep 1
done
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "qbittorrent-nox was already running - restarting it to load the credential"
@@ -251,14 +286,26 @@ lifecycle:
exit 0
fi
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all
# (shared ~90s budget across attempts), then spend exactly ONE login per attempt -
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
while [ "$waited" -lt 90 ]; do
#
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
# conflating them is what made this hook lie once already (see the PORT comment). An
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
# its remaining login attempts to produce two more identical wrong diagnoses.
api_ok=0
while [ "$waited" -lt 120 ]; do
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
[ -n "$code" ] && [ "$code" != "000" ] && break
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
waited=$((waited + 1)); sleep 1
done
if [ "$api_ok" != 1 ]; then
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
log "never submitted. Do not rotate the credential on the strength of this line." >&2
exit 0
fi
body=$(curl -sS -m 10 \
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
@@ -267,7 +314,17 @@ lifecycle:
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
exit 0
fi
log "WebUI rejected the durable credential (login said '$body') - reasserting" >&2
log "WebUI rejected the durable credential (login said '$body')" >&2
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
# make the process RE-READ the file, so restart the supervised service and let the next
# attempt re-verify. Without this the retry loop could never change its own outcome.
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
sleep 3
fi
done
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2