merge: qBittorrent has a durable credential, and the arrs work again

qBittorrent 5.x minted a new temporary admin password on every process
start, so the Sonarr/Radarr download clients broke on the next restart — and
had already broken: both reported isValid: false, Authentication Failure.

A SOPS secret is now the single source of truth and a postStart hook derives
qBittorrent's stored PBKDF2 form on every container start, then logs in to
prove it. The hook rather than a CronJob because the failure IS a container
start: qbittorrent-nox reads the credential once at start from an RWO PVC the
app holds, so a 15-minute cron leaves a window after every restart where both
arrs are broken. An initContainer cannot do the ground-truth login.

Proven: testall flipped false -> true on both arrs, and survived a real pod
deletion and recreation with the conf still holding both keys.

Two bugs found in the inherited work, both the envsubst trap again: a
doubled-dollar port expansion that bash read as its own PID, which made the
hook report 'the WebUI rejected the credential' three times without ever
asking it anything; and one shared wait budget that let the first attempt
burn all 90s. Also caught that the SOPS file was never listed in
apps/wave-0/kustomization.yaml, so Flux would never have applied it — the
live Secret was not going to be adopted at all.
This commit is contained in:
jakman85 committed 2026-09-13 16:34:35 -05:00
commit cc648ba02c
5 files changed
+490

No files matched your search

+121
View File
@@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o
Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits"
only because their agents had not committed yet, and deleting those would have destroyed live work.
## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`)
**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor
`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a
fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences,
and **both had already fired by the time this was finished** — this was not a hypothesis:
- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ,
before any change: `POST /api/v3/downloadclient/testall` returned
`isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]`
on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a
failed client, not a failed import.
- No human had a stable login either; every restart meant reading the pod log.
**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32
alphanumeric chars) is the single source of truth, consumed as env by the app container. A
`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000
iterations, 16-byte random salt, 64-byte digest, stored as
`WebUI\Password_PBKDF2="@ByteArray(<b64 salt>:<b64 digest>)"` — asserts both keys into the conf on
**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is
never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken
and makes auth depend on network policy staying correct for ever).
**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential:
`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does
not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod
to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from
git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the
credential into it before the WebUI is serving. The only input is the SOPS file in git.
**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`,
the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state
in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger
differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential
once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after
every restart in which the app serves a random password and both arrs are broken. A CronJob would also
have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or
`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose
own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or
API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it
would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform
the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has
`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across
every service's golden to buy something strictly weaker.
**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook
computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the
cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes
it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became
`<pid>{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI
rejected the durable credential" three times having never asked it anything**. That misdiagnosis is
preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal
`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically
however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and
"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have
separate counters, "the API never answered" is reported as a distinct fault from "the password was
rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome.
**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the
arrs at the correct password FAILED at first: the arr's own save-time validation returned
`Authentication Failure`, and from inside the sonarr pod qBittorrent answered
`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries
against the stale password had banned both arr pod IPs. The credential was right the whole time. The
ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed
(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating
failures). The hook spends at most 3 logins per start for exactly this reason.
**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and
Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every
password field. Comparing against that makes every run look like a change — and it also means the
earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The
convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it
says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing.
**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted
and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the
WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still
reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential
rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization`
produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write
path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the
helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed
(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the
digest it produced verifies under an independent PBKDF2 implementation with the git password and
fails with a wrong one.
**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod.
The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live
object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's
permission classifier. The durability shown above is therefore the *credential's* durability across
two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually
invoking it is verified only on merge. **First thing to check after merging:** that
`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential`
line on the next pod start.
**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's
download-client rows — including this password — live in their own databases on their config PVCs.
Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against
both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records.
**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as
`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml`
did not list it**, so Flux would never have applied it and the adoption the last handoff expected
would silently not have happened. It is listed now (verified: the decrypted SOPS values are
byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password).
Note also that the live Secret carries the plaintext in a
`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by
hand; Flux's server-side apply on merge does not write that annotation.
**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree
cannot work here because the container mounts only the worktree while its `.git` points outside).
spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden
(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched
requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left
skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the
only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a
scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC
bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of
the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are
about: a stage that reports green while verifying nothing.
+1
View File
@@ -17,6 +17,7 @@ resources:
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
- qbittorrent.yaml
- semaphore-db-app.sops.yaml
@@ -0,0 +1,60 @@
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
#
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
# read the pod log after every restart.
#
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
# credential, because the hook re-asserts it on every start rather than once.
#
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
# entries store in their own databases (app state on their config PVCs, NOT in git — see
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
#
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
metadata:
name: qbittorrent-bootstrap
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
type: Opaque
stringData:
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-13T05:23:23Z"
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+19
View File
@@ -130,12 +130,31 @@ spec:
value: '1000'
- name: PGID
value: '3000'
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
volumeMounts:
- name: config
mountPath: /config
- name: downloads
mountPath: /downloads
subPath: downloads
lifecycle:
postStart:
exec:
command:
- /bin/bash
- -c
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
+289
View File
@@ -44,6 +44,295 @@ env:
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
# readable to the EXISTING non-k3s consumers of the same dataset, and land
# group-owned the same way the arrs' own writes do
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
# tested valid against the then-current temporary password; both would have started failing at
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
# report a failed client, not a failed import).
# * The human operator had no stable login: every restart meant reading the pod log.
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
lifecycle:
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
#
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
# the container start rather than a clock, because the failure this closes IS a container start:
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
# restore or rebuild in which the app serves a random password and both arrs are broken. The
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
# In-container, before the service starts, none of that exists.
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
# privilege than the CronJob alternative (R-SEC/least-privilege).
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
# keeps in its own state), and the schema documents this hook for exactly that use.
#
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
# credential — because the WebUI is not running yet. That assertion is the most valuable line
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
# mean a schema version bump plus a renderer change — a blast radius across every service's
# golden — to buy something strictly weaker than what the hook already does.
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
#
# DURABILITY, the whole point — all three paths end with the SAME credential:
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
# hook runs and re-asserts over whatever the snapshot contained.
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
# credential into it before the service is serving. Nothing outside git.
#
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
# password is simply now known, fixed, and reproducible from git.
postStart:
exec:
command:
- /bin/bash
- -c
- |
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
#
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
# CLUSTER — but only there:
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
# however it is run.
set -u
CONF=/config/qBittorrent/qBittorrent.conf
SVC=/run/service/svc-qbittorrent
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
# env var for a lookup to read anyway.
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
# logged "the WebUI rejected the durable
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
PORT=8080
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
# credential from git was asserted and accepted on this start.
log() {
echo "[qbt-webui-cred] $*"
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
}
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
# racing to create one it would then not overwrite.
n=0
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
waited=0
attempt=0
while [ "$attempt" -lt 3 ]; do
attempt=$((attempt + 1))
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
# 10 = rewritten, anything else = could not write.
python3 - "$CONF" <<'PY'
import base64, hashlib, hmac, os, stat, sys
conf = sys.argv[1]
user = os.environ.get("QBT_WEBUI_USERNAME", "")
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
if not user or not pw:
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
raise SystemExit(1)
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
# iterations, 16-byte random salt, 64-byte digest, stored as
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
# Get any one of them wrong and the app simply rejects every login.
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
def matches(raw):
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
v = raw.strip().strip('"')
if v.startswith("@ByteArray(") and v.endswith(")"):
v = v[len("@ByteArray("):-1]
try:
salt_b64, digest_b64 = v.split(":", 1)
salt = base64.b64decode(salt_b64, validate=True)
digest = base64.b64decode(digest_b64, validate=True)
except Exception:
return False
if not salt or not digest:
return False
return hmac.compare_digest(
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
try:
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
except FileNotFoundError:
lines = []
def value_of(key):
for i, line in enumerate(lines):
if line.startswith(key + "="):
return i, line.split("=", 1)[1]
return -1, None
ui, uval = value_of(UKEY)
pi, pval = value_of(PKEY)
if uval == user and pval is not None and matches(pval):
raise SystemExit(0) # converged - leave the file (and its salt) alone
salt = os.urandom(SALT_LEN)
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
if SECTION in out:
at = out.index(SECTION) + 1
else:
if out and out[-1].strip():
out.append("")
out.append(SECTION)
at = len(out)
out[at:at] = desired
try:
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
tmp = conf + ".webui-cred.new"
with open(tmp, "w", encoding="utf-8") as fh:
fh.write("\n".join(out) + "\n")
fh.flush()
os.fsync(fh.fileno())
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
# cost the app the ability to save its own settings, so carry the ownership over.
try:
st = os.stat(conf)
os.chown(tmp, st.st_uid, st.st_gid)
os.chmod(tmp, stat.S_IMODE(st.st_mode))
except FileNotFoundError:
os.chown(tmp, int(os.environ.get("PUID", "1000")),
int(os.environ.get("PGID", "1000")))
os.chmod(tmp, 0o664)
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
except OSError as exc:
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
raise SystemExit(1)
raise SystemExit(10)
PY
rc=$?
if [ "$rc" = 0 ]; then
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
elif [ "$rc" = 10 ]; then
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
# have read the old, credential-less conf and be serving a random temporary password.
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
# Nothing to do when the service has not started yet - it will read the new file.
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
# 2026-09-13 failure spent the entire verification budget before the first login and
# then fired attempts 2 and 3 with no wait left at all.
sv=0
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
sv=$((sv + 1)); sleep 1
done
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "qbittorrent-nox was already running - restarting it to load the credential"
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
sleep 3
fi
else
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
exit 0
fi
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
#
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
# conflating them is what made this hook lie once already (see the PORT comment). An
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
# its remaining login attempts to produce two more identical wrong diagnoses.
api_ok=0
while [ "$waited" -lt 120 ]; do
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
waited=$((waited + 1)); sleep 1
done
if [ "$api_ok" != 1 ]; then
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
log "never submitted. Do not rotate the credential on the strength of this line." >&2
exit 0
fi
body=$(curl -sS -m 10 \
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
if [ "$body" = "Ok." ]; then
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
exit 0
fi
log "WebUI rejected the durable credential (login said '$body')" >&2
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
# make the process RE-READ the file, so restart the supervised service and let the next
# attempt re-verify. Without this the retry loop could never change its own outcome.
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
sleep 3
fi
done
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
# app, and the arrs' own download-client health check surfaces a broken credential.
exit 0
podSecurityContext:
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem