diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cdf348b..2a12ab6 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3042,3 +3042,124 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits" only because their agents had not committed yet, and deleting those would have destroyed live work. + +## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`) + +**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor +`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a +fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences, +and **both had already fired by the time this was finished** — this was not a hypothesis: + +- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ, + before any change: `POST /api/v3/downloadclient/testall` returned + `isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]` + on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a + failed client, not a failed import. +- No human had a stable login either; every restart meant reading the pod log. + +**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32 +alphanumeric chars) is the single source of truth, consumed as env by the app container. A +`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000 +iterations, 16-byte random salt, 64-byte digest, stored as +`WebUI\Password_PBKDF2="@ByteArray(:)"` — asserts both keys into the conf on +**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is +never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken +and makes auth depend on network policy staying correct for ever). + +**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential: +`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does +not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod +to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from +git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the +credential into it before the WebUI is serving. The only input is the SOPS file in git. + +**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`, +the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state +in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger +differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential +once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after +every restart in which the app serves a random password and both arrs are broken. A CronJob would also +have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or +`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose +own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or +API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it +would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform +the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has +`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across +every service's golden to buy something strictly weaker. + +**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook +computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the +cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes +it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became +`{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI +rejected the durable credential" three times having never asked it anything**. That misdiagnosis is +preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal +`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically +however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and +"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have +separate counters, "the API never answered" is reported as a distinct fault from "the password was +rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome. + +**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the +arrs at the correct password FAILED at first: the arr's own save-time validation returned +`Authentication Failure`, and from inside the sonarr pod qBittorrent answered +`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries +against the stale password had banned both arr pod IPs. The credential was right the whole time. The +ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed +(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating +failures). The hook spends at most 3 logins per start for exactly this reason. + +**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and +Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every +password field. Comparing against that makes every run look like a change — and it also means the +earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The +convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it +says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing. + +**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted +and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the +WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still +reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential +rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization` +produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write +path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the +helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed +(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the +digest it produced verifies under an independent PBKDF2 implementation with the git password and +fails with a wrong one. + +**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod. +The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live +object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's +permission classifier. The durability shown above is therefore the *credential's* durability across +two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually +invoking it is verified only on merge. **First thing to check after merging:** that +`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential` +line on the next pod start. + +**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's +download-client rows — including this password — live in their own databases on their config PVCs. +Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against +both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records. + +**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as +`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml` +did not list it**, so Flux would never have applied it and the adoption the last handoff expected +would silently not have happened. It is listed now (verified: the decrypted SOPS values are +byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password). +Note also that the live Secret carries the plaintext in a +`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by +hand; Flux's server-side apply on merge does not write that annotation. + +**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree +cannot work here because the container mounts only the worktree while its `.git` points outside). +spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden +(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched +requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left +skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the +only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a +scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC +bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of +the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are +about: a stage that reports green while verifying nothing. diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 4899214..aa3b049 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -17,6 +17,7 @@ resources: - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml - plex.yaml +- qbittorrent-bootstrap-secret.sops.yaml - qbittorrent-oauth2-proxy-credentials.sops.yaml - qbittorrent.yaml - semaphore-db-app.sops.yaml diff --git a/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml b/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml new file mode 100644 index 0000000..126ce16 --- /dev/null +++ b/apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml @@ -0,0 +1,60 @@ +# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux. +# +# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13): +# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR +# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW +# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary +# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's +# saved download-client credentials stop working at the next qbittorrent pod restart, silently +# breaking Completed Download Handling; (b) a human operator has no stable login at all and must +# read the pod log after every restart. +# +# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env +# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as +# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores +# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then +# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore +# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME +# credential, because the hook re-asserts it on every start rather than once. +# +# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is +# the name its own temporary-password path uses, so keeping it avoids inventing a second identity. +# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client +# entries store in their own databases (app state on their config PVCs, NOT in git — see +# DECISIONS-LOG), so rotating it here means re-pointing both arrs. +# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON +# payloads, and a shell/URL-special character in any of those is a bug waiting to happen. +# +# NEVER put a value in a comment: SOPS does NOT encrypt comments. +# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml +apiVersion: v1 +kind: Secret +metadata: + name: qbittorrent-bootstrap + namespace: qbittorrent + labels: + app.kubernetes.io/name: qbittorrent +type: Opaque +stringData: + webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str] + webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2 + TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3 + UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv + VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv + FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-13T05:23:23Z" + mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index 25ede18..da9e4d1 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -130,12 +130,31 @@ spec: value: '1000' - name: PGID value: '3000' + - name: QBT_WEBUI_USERNAME + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-username + - name: QBT_WEBUI_PASSWORD + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-password volumeMounts: - name: config mountPath: /config - name: downloads mountPath: /downloads subPath: downloads + lifecycle: + postStart: + exec: + command: + - /bin/bash + - -c + - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(:)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\ + , \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\ + \ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 imagePullPolicy: IfNotPresent diff --git a/catalog/qbittorrent.yaml b/catalog/qbittorrent.yaml index fc0393e..840561f 100644 --- a/catalog/qbittorrent.yaml +++ b/catalog/qbittorrent.yaml @@ -44,6 +44,295 @@ env: value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay # readable to the EXISTING non-k3s consumers of the same dataset, and land # group-owned the same way the arrs' own writes do + # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) ------------------- + # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username` + # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a + # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary + # password is provided for this session: ..."). Two real consequences, not hypotheticals: + # * Sonarr and Radarr store a download-client username/password in their OWN databases. Both + # tested valid against the then-current temporary password; both would have started failing at + # the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs + # report a failed client, not a failed import). + # * The human operator had no stable login: every restart meant reading the pod log. + # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) — + # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave. + - name: QBT_WEBUI_USERNAME + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-username + - name: QBT_WEBUI_PASSWORD + valueFrom: + secretKeyRef: + name: qbittorrent-bootstrap + key: webui-password +lifecycle: + # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start. + # + # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach + # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml, + # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is + # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is + # the container start rather than a clock, because the failure this closes IS a container start: + # * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config + # PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC + # restore or rebuild in which the app serves a random password and both arrs are broken. The + # hook has no such window: it runs before/with the very start that would otherwise regenerate. + # * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn + # PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and + # then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit. + # In-container, before the service starts, none of that exists. + # * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less + # privilege than the CronJob alternative (R-SEC/least-privilege). + # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app + # keeps in its own state), and the schema documents this hook for exactly that use. + # + # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly + # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout + # would reach `kubectl logs -c ` instead of the kubelet. Two reasons it loses here: + # * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the + # credential — because the WebUI is not running yet. That assertion is the most valuable line + # in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and + # writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong. + # * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would + # mean a schema version bump plus a renderer change — a blast radius across every service's + # golden — to buy something strictly weaker than what the hook already does. + # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by + # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade. + # + # DURABILITY, the whole point — all three paths end with the SAME credential: + # pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING. + # config-PVC restore from an older backup -> restore requires the pod to be recreated, so the + # hook runs and re-asserts over whatever the snapshot contained. + # rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the + # credential into it before the service is serving. Nothing outside git. + # + # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth + # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth + # depend on network policy staying correct forever. The WebUI keeps demanding a password; that + # password is simply now known, fixed, and reproducible from git. + postStart: + exec: + command: + - /bin/bash + - -c + - | + # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the + # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all. + # + # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE. + # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over + # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar; + # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE + # CLUSTER — but only there: + # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl + # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during + # development) keeps `$$`, which bash expands to its own PID. That is not a hypothetical; + # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)` + # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY + # however it is run. + set -u + CONF=/config/qBittorrent/qBittorrent.conf + SVC=/run/service/svc-qbittorrent + # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this + # ServiceDefinition — which is the single place the renderer takes the containerPort, the + # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT + # env var for a lookup to read anyway. + # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and + # that cost real time: run without envsubst, bash read the doubled dollar as its own PID, + # so PORT became "{WEBUI_PORT:-8080}" — the brace never expanded, the digits never + # matched. Every curl below then got an unparseable URL, returned nothing, and the hook + # logged "the WebUI rejected the durable + # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this + # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP, + # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled. + PORT=8080 + # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook + # that only echoes is effectively silent (gitea's hits the same wall and writes a marker + # file for it). Every outcome therefore also lands, timestamped, on the persistent config + # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the + # credential from git was asserted and accepted on this start. + log() { + echo "[qbt-webui-cred] $*" + printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \ + >> /config/qbittorrent-webui-cred.log 2>/dev/null || true + } + + # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC + # before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of + # racing to create one it would then not overwrite. + n=0 + while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done + + waited=0 + attempt=0 + while [ "$attempt" -lt 3 ]; do + attempt=$((attempt + 1)) + + # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt), + # 10 = rewritten, anything else = could not write. + python3 - "$CONF" <<'PY' + import base64, hashlib, hmac, os, stat, sys + + conf = sys.argv[1] + user = os.environ.get("QBT_WEBUI_USERNAME", "") + pw = os.environ.get("QBT_WEBUI_PASSWORD", "") + if not user or not pw: + sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n") + raise SystemExit(1) + + # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000 + # iterations, 16-byte random salt, 64-byte digest, stored as + # WebUI\Password_PBKDF2="@ByteArray(:)" + # Get any one of them wrong and the app simply rejects every login. + ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16 + UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]" + + def matches(raw): + """True iff the stored digest is one OUR password produces (salt is read from it).""" + v = raw.strip().strip('"') + if v.startswith("@ByteArray(") and v.endswith(")"): + v = v[len("@ByteArray("):-1] + try: + salt_b64, digest_b64 = v.split(":", 1) + salt = base64.b64decode(salt_b64, validate=True) + digest = base64.b64decode(digest_b64, validate=True) + except Exception: + return False + if not salt or not digest: + return False + return hmac.compare_digest( + hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest) + + try: + lines = open(conf, encoding="utf-8", errors="replace").read().splitlines() + except FileNotFoundError: + lines = [] + + def value_of(key): + for i, line in enumerate(lines): + if line.startswith(key + "="): + return i, line.split("=", 1)[1] + return -1, None + + ui, uval = value_of(UKEY) + pi, pval = value_of(PKEY) + if uval == user and pval is not None and matches(pval): + raise SystemExit(0) # converged - leave the file (and its salt) alone + + salt = os.urandom(SALT_LEN) + digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN) + stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode() + desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"'] + + out = [line for i, line in enumerate(lines) if i not in (ui, pi)] + if SECTION in out: + at = out.index(SECTION) + 1 + else: + if out and out[-1].strip(): + out.append("") + out.append(SECTION) + at = len(out) + out[at:at] = desired + + try: + os.makedirs(os.path.dirname(conf) or ".", exist_ok=True) + tmp = conf + ".webui-cred.new" + with open(tmp, "w", encoding="utf-8") as fh: + fh.write("\n".join(out) + "\n") + fh.flush() + os.fsync(fh.fileno()) + # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would + # cost the app the ability to save its own settings, so carry the ownership over. + try: + st = os.stat(conf) + os.chown(tmp, st.st_uid, st.st_gid) + os.chmod(tmp, stat.S_IMODE(st.st_mode)) + except FileNotFoundError: + os.chown(tmp, int(os.environ.get("PUID", "1000")), + int(os.environ.get("PGID", "1000"))) + os.chmod(tmp, 0o664) + os.replace(tmp, conf) # atomic: the app never reads a half-written conf + except OSError as exc: + sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc)) + raise SystemExit(1) + raise SystemExit(10) + PY + rc=$? + + if [ "$rc" = 0 ]; then + log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)" + elif [ "$rc" = 10 ]; then + log "asserted WebUI Username + Password_PBKDF2 into $CONF" + # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY + # have read the old, credential-less conf and be serving a random temporary password. + # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written. + # Nothing to do when the service has not started yet - it will read the new file. + # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the + # 2026-09-13 failure spent the entire verification budget before the first login and + # then fired attempts 2 and 3 with no wait left at all. + sv=0 + while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do + sv=$((sv + 1)); sleep 1 + done + if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then + log "qbittorrent-nox was already running - restarting it to load the credential" + s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" + sleep 3 + fi + else + log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2 + log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2 + exit 0 + fi + + # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL + # (shared ~120s budget across attempts), then spend exactly ONE login per attempt - + # at most 3 rejected logins, which stays under qBittorrent's default ban threshold. + # + # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and + # conflating them is what made this hook lie once already (see the PORT comment). An + # unreachable WebUI is reported as unreachable, and the hook stops rather than spending + # its remaining login attempts to produce two more identical wrong diagnoses. + api_ok=0 + while [ "$waited" -lt 120 ]; do + code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null) + if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi + waited=$((waited + 1)); sleep 1 + done + if [ "$api_ok" != 1 ]; then + log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2 + log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2 + log "never submitted. Do not rotate the credential on the strength of this line." >&2 + exit 0 + fi + body=$(curl -sS -m 10 \ + --data-urlencode "username=$QBT_WEBUI_USERNAME" \ + --data-urlencode "password=$QBT_WEBUI_PASSWORD" \ + "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null) + if [ "$body" = "Ok." ]; then + log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'" + exit 0 + fi + log "WebUI rejected the durable credential (login said '$body')" >&2 + # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean + # the running qbittorrent-nox is serving something it read from an EARLIER conf, so + # rewriting the same bytes on the next attempt would change nothing. The remedy is to + # make the process RE-READ the file, so restart the supervised service and let the next + # attempt re-verify. Without this the retry loop could never change its own outcome. + if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then + log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2 + s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2 + sleep 3 + fi + done + + log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2 + # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container + # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage. + # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the + # app, and the arrs' own download-client health check surfaces a broken credential. + exit 0 podSecurityContext: # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem