Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid

Browse Source 
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.

Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.

Two real bugs found in the WIP, both the postBuild-substitution trap:

  * PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
    (envsubst collapses it) and the renderer accepts it, but run any other way
    bash reads the doubled dollar as its PID, so every curl got an unparseable
    URL and the hook logged "the WebUI rejected the durable credential" three
    times having never asked it anything — see the marker log at 05:35Z. Now a
    literal PORT=8080; the script contains no braced expansion at all and
    behaves identically however it is run.
  * One shared wait budget covered both the s6 wait and the API wait, so the
    first consumed all 90s and attempts 2-3 fired instantly. Separate counters
    now; "the API never answered" is reported as a distinct fault from "the
    password was rejected"; and a rejection restarts qbittorrent-nox so a retry
    can actually change its outcome.

apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.

Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.

NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 16:31:58 -05:00
1 parent 204f0ec28e
commit e4bd76070e
4 files changed
+195 -15
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DECISIONS-LOG.md
+121
View File
@@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o
Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits"
only because their agents had not committed yet, and deleting those would have destroyed live work.
## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`)
**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor
`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a
fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences,
and **both had already fired by the time this was finished** — this was not a hypothesis:
- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ,
before any change: `POST /api/v3/downloadclient/testall` returned
`isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]`
on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a
failed client, not a failed import.
- No human had a stable login either; every restart meant reading the pod log.
**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32
alphanumeric chars) is the single source of truth, consumed as env by the app container. A
`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000
iterations, 16-byte random salt, 64-byte digest, stored as
`WebUI\Password_PBKDF2="@ByteArray(<b64 salt>:<b64 digest>)"` — asserts both keys into the conf on
**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is
never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken
and makes auth depend on network policy staying correct for ever).
**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential:
`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does
not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod
to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from
git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the
credential into it before the WebUI is serving. The only input is the SOPS file in git.
**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`,
the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state
in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger
differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential
once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after
every restart in which the app serves a random password and both arrs are broken. A CronJob would also
have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or
`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose
own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or
API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it
would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform
the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has
`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across
every service's golden to buy something strictly weaker.
**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook
computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the
cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes
it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became
`<pid>{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI
rejected the durable credential" three times having never asked it anything**. That misdiagnosis is
preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal
`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically
however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and
"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have
separate counters, "the API never answered" is reported as a distinct fault from "the password was
rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome.
**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the
arrs at the correct password FAILED at first: the arr's own save-time validation returned
`Authentication Failure`, and from inside the sonarr pod qBittorrent answered
`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries
against the stale password had banned both arr pod IPs. The credential was right the whole time. The
ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed
(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating
failures). The hook spends at most 3 logins per start for exactly this reason.
**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and
Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every
password field. Comparing against that makes every run look like a change — and it also means the
earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The
convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it
says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing.
**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted
and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the
WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still
reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential
rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization`
produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write
path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the
helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed
(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the
digest it produced verifies under an independent PBKDF2 implementation with the git password and
fails with a wrong one.
**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod.
The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live
object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's
permission classifier. The durability shown above is therefore the *credential's* durability across
two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually
invoking it is verified only on merge. **First thing to check after merging:** that
`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential`
line on the next pod start.
**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's
download-client rows — including this password — live in their own databases on their config PVCs.
Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against
both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records.
**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as
`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml`
did not list it**, so Flux would never have applied it and the adoption the last handoff expected
would silently not have happened. It is listed now (verified: the decrypted SOPS values are
byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password).
Note also that the live Secret carries the plaintext in a
`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by
hand; Flux's server-side apply on merge does not write that annotation.
**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree
cannot work here because the container mounts only the worktree while its `.git` points outside).
spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden
(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched
requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left
skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the
only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a
scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC
bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of
the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are
about: a stage that reports green while verifying nothing.
apps/wave-0/kustomization.yaml
+1
View File
@@ -17,6 +17,7 @@ resources:
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
- qbittorrent.yaml
- semaphore-db-app.sops.yaml
apps/wave-0/qbittorrent.yaml
+3 -2
View File
@@ -152,8 +152,9 @@ spec:
command:
- /bin/bash
- -c
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# NOTE the `$${...}` escapes below \u2014 load-bearing. apps-wave-0 declares\n# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at\n# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst\n# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the\n# incident that taught this, and render.py's assert_substitution_safe, which enforces it).\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\nPORT=$${WEBUI_PORT:-8080}\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\", \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY\
\ + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n while [ \"$waited\" -lt 90 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all\n # (shared ~90s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n while [ \"$waited\" -lt 90 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n [ -n \"$code\" ] && [ \"$code\" != \"000\" ] && break\n waited=$((waited + 1)); sleep 1\n done\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body') - reasserting\" >&2\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
catalog/qbittorrent.yaml
+70 -13
View File
@@ -87,6 +87,19 @@ lifecycle:
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
# keeps in its own state), and the schema documents this hook for exactly that use.
#
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
# credential — because the WebUI is not running yet. That assertion is the most valuable line
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
# mean a schema version bump plus a renderer change — a blast radius across every service's
# golden — to buy something strictly weaker than what the hook already does.
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
#
# DURABILITY, the whole point — all three paths end with the SAME credential:
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
@@ -107,15 +120,33 @@ lifecycle:
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
#
# NOTE the `$${...}` escapes below — load-bearing. apps-wave-0 declares
# postBuild.substituteFrom cluster-vars, so Flux runs envsubst over this whole manifest at
# RECONCILE time, and a SINGLE-dollar brace expansion is not a cluster-var, so envsubst
# would blank it. Bare `$VAR` and `$(cmd)` are left alone (see catalog/gitea.yaml for the
# incident that taught this, and render.py's assert_substitution_safe, which enforces it).
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
# CLUSTER — but only there:
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
# however it is run.
set -u
CONF=/config/qBittorrent/qBittorrent.conf
SVC=/run/service/svc-qbittorrent
PORT=$${WEBUI_PORT:-8080}
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
# env var for a lookup to read anyway.
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
# logged "the WebUI rejected the durable
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
PORT=8080
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
@@ -237,8 +268,12 @@ lifecycle:
# have read the old, credential-less conf and be serving a random temporary password.
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
# Nothing to do when the service has not started yet - it will read the new file.
while [ "$waited" -lt 90 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
waited=$((waited + 1)); sleep 1
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
# 2026-09-13 failure spent the entire verification budget before the first login and
# then fired attempts 2 and 3 with no wait left at all.
sv=0
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
sv=$((sv + 1)); sleep 1
done
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "qbittorrent-nox was already running - restarting it to load the credential"
@@ -251,14 +286,26 @@ lifecycle:
exit 0
fi
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at all
# (shared ~90s budget across attempts), then spend exactly ONE login per attempt -
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
while [ "$waited" -lt 90 ]; do
#
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
# conflating them is what made this hook lie once already (see the PORT comment). An
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
# its remaining login attempts to produce two more identical wrong diagnoses.
api_ok=0
while [ "$waited" -lt 120 ]; do
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
[ -n "$code" ] && [ "$code" != "000" ] && break
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
waited=$((waited + 1)); sleep 1
done
if [ "$api_ok" != 1 ]; then
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
log "never submitted. Do not rotate the credential on the strength of this line." >&2
exit 0
fi
body=$(curl -sS -m 10 \
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
@@ -267,7 +314,17 @@ lifecycle:
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
exit 0
fi
log "WebUI rejected the durable credential (login said '$body') - reasserting" >&2
log "WebUI rejected the durable credential (login said '$body')" >&2
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
# make the process RE-READ the file, so restart the supervised service and let the next
# attempt re-verify. Without this the retry loop could never change its own outcome.
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
sleep 3
fi
done
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2